[#] Sat Feb 02 2019 11:16:19 EST from IGnatius T Foobar @ Uncensored
Why not. Let her defend her own honor if she doesn't want to be swimming in mayo.
[#] Fri Feb 08 2019 21:25:41 EST from LoanShark @ Uncensored
while (true) System.err.println("node.js sucks, and you're a moron for even considering using it.")
Worst language runtime in the world. You'll never get a meaningful stacktrace for anything, because everything is a short-duration callback run from the event loop. Meanwhile, the language designers have piled enough syntactic sugar on top of callbacks to make them look like linear flow-control, basically reifying threads (albiet cooperative threads.) If the new calling convention is "everything is an async function", you just trashed the alleged performance benefits of async IO.
[#] Mon Feb 11 2019 11:53:42 EST from Ragnar Danneskjold @ Uncensored
Does copy data to a data warehouse via a cron job make sense? Shouldn't that happen INSIDE the database instead?
[#] Mon Feb 11 2019 12:27:19 EST from LoanShark @ Uncensored
2019-02-11 11:53 from Ragnar Danneskjold
Does copy data to a data warehouse via a cron job make sense?
Shouldn't that happen INSIDE the database instead?
We did a lot of this a couple of jobs ago; maybe it wans't exactly a cron, the ETL tool might have managed the scheduling. But either way, it's not an uncommon pattern.
We had it set up with a little bit of both. Inside the database there were audit tables (which track insert/update/deletes for each associated table) and these were populated by triggers. Then the ETL job comes along periodically and pulls the new audit rows.
[#] Mon Feb 11 2019 13:43:04 EST from wizard of aahz @ Uncensored
Ragnar - Our backups happen via transaction log shipping. and then being reapplied to backup databases, but that's not really what you want, so you'll need to have something outside the database as well to pull I would imagine.
[#] Mon Feb 11 2019 14:05:40 EST from Ragnar Danneskjold @ Uncensored
Data warehouse is really for reporting purposes over time, not for the real time work done. Not backup. Although that's something we need to consider as well.....
[#] Mon Feb 11 2019 14:39:12 EST from wizard of aahz @ Uncensored
Correct. Usually when grabbing for data warehouse you're doing some sort of aggregation before you carry it over. WEll in the old days that's how we would do it. Now we might bring over the details and aggregate on the other machine.
[#] Fri Feb 15 2019 15:01:11 EST from LoanShark @ Uncensored
Riddle me this. Why the eff does mongodb need 1gb of heap cache to serve a database that has about 10 rows?
I get that the heap limit is a tunable with the new storage backend since 3.2, but something seems wrong here.
[#] Fri Feb 15 2019 15:30:44 EST from LoanShark @ Uncensored
Correct. Usually when grabbing for data warehouse you're doing some
sort of aggregation before you carry it over. WEll in the old days
that's how we would do it. Now we might bring over the details and
aggregate on the other machine.
Yes. Production OLTP should be assumed to be too overloaded (even if it isn't) to be bothered doing the cube rollups. Unless perhaps they're fairly simple and can be done in triggers on every insert--but that's only the case for the simplest of data warehouses.
Read replicas (when set up with popular toolsets such as RDS) tend to be read-only. So, it doesn't happen there either.
There *are* some alternatives, I remember seeing a closed-source product that would parse your mysql binlogs and help handle pushing all that data to your warehouse. It's not something I've ever tried, but it's out there.
Sorry I haven't been around much lately. I have to redesign our product at work, and it has me very distracted.
It's good, though. Our original design isn't cutting it, but now that we know how folks want to work with the product, we can build something better.
Besides, some of the designed involved efforts from someone who doesn't really know how to engineer this kind of thing, so reworking things from another perspective should give us more flexibility for all kinds of peculiar ways that someone might want to work with this.
But... it's making me think about stuff like 'Do we want to use a nosql database, or stick to a relational database with a kind of object-orientedness grafted onto it', or 'these guys communicate to each other using HTTPS, but are mechanical (all API-driven), so authentication should be handled through http signatures instead of the usual authentication schemes, since we can share keys', and so on.
It's liberating, but a lot of work.
Go to page: ...