Please wait...
Well yes, there is that. It pains me to see a bunch of paper tigers insisting that you need half a dozen servers to provide email to a group of 100 people, because that's what they were told was "best practice." And then it has regular outages and they all blame each other.
This is not an exaggeration; I speak from direct experience. The environment in question ran Citadel for nearly seven years without a single outage. So you have a system built by hobbyists in our spare time, with quality that greatly exceeds an expensive system built by thousands of highly paid developers and QA people.
So I'm going to come to a different conclusion. Quality happens when someone is thinking about quality. It happens when delivering something that works properly is held as a higher priority than delivering something that has an ever-growing feature set. I believe that this can happen in the cathedral or in the bazaar.
Wearing my pointy-haired toupee for the moment... part of my responsibilities are tending toward team lead these days. So I think: quality happens if you lecture the junior devs until their ears bleed. ( I try to be a bit nicer about this than it sounds. ) That's more cathedral than bazaar though.
Imagine that... me, responsible.
As long as they're legitimate competition, I'm okay.
If their supposed to be on the same team, I need to find a smaller team.
Meh... I haven't had too many dealings with those types, because I don't get hired to work in environments like that. Possibly because I'm viewed as a threat.
It's nice to be important, but it's more important to be nice.
The more monitoring you have, the more trouble you find. If you don’t have monitoring, trouble finds you.
Splunk. Has worked very well for us.
Pricey, but when you don't have time to *design* something, you just throw a full-text search engine at your app logs.
I think we're starting to talk about a more open alternative?
theres something like logstash, which I again instantly forgot about since its also done in some bizare language running in the java interperter.
logstash is one of the alternatives we were talking about, yes. I doubt we did a full evaluation yet.
One of the most powerful things we do with Splunk is the "transaction" filter. I don't see any direct replacement with logstash... this is statically configured and seems to have limitations, but it's somewhere in the ballpark: http://logstash.net/docs/1.4.2/filters/multiline
not quite close enough, transaction is an ad-hoc query: http://docs.splunk.com/Documentation/Splunk/6.2.1/SearchReference/Transaction
probably one would want something like that:
this is the logstash alternative:
https://www.graylog.org/overview
(the credativ guys work with it)
Another tool /me wouldn't use... but may be interesting ;-)
I like very much this one:
(have a look at the crazy videos ;-)
It uses a pimped collectd as some of the data sources.
OK, for one thing, I hadn't correctly understood how logstash fits together with the rest of its ecosystem. Logstash is like splunkforwarder, I guess- it's a piece of low-level plumbing.
The querying all happens in elasticsearch: http://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-queries.html