The Workplace

http://blogs.msdn.com/b/jmeier/archive/2010/10/21/40-hour-work-week-at-microsoft.aspx

fancy to read something like that out of the fingers of a microsoft employe...

I currently know all of the points he enlists in the 60-80 hours week.

Yesterday, the customer's staff plugged a USB drive into the server we maintain for them, and infected it with two viruses.  It took nearly 8 hours of two sets of staff's time to take care of that, so we could rule out viruses as the cause of the symptoms of a problem we were trying to address.

We asked that they stop doing this... that they use some other approach instead.

Today, someone did it again.

They're trying to copy these caption files to a folder where we'll turn them into our own file format and pull them into our file server, so we can play them back when they're supposed to be played back for broadcast.

But the left hand doesn't know what the right hand over there is doing, so they put the files in the wrong folder, and wondered why things weren't working.

The insanity gets even nuttier, the deeper you look into this, though.  Honestly, if I hadn't met some of these guys, I'd assume that this company has hired patients from a mental institution to take care of their broadcasting needs.

You guys are geeky... you might appreciate the details involved in this particular version of hell.

A playlist has a media identifier that describes the tape used to play the media (or, perhaps in these modern times, an MPEG2 file).  We use this media identifier to find the appropriate caption file for playback later.

Based on the media ID, we run it through a hash that generates a number from 00 to ff (hex), which we then use as a subfolder, so all these files won't be in one folder (it gets pretty nasty if all the files are in the one folder).  However, we also have this 'alias' mechanism... sometimes, the media ID has some extra characters that describe the segment of the program, and the playlist will give us that instead of the actual media ID.  We use an aliasing mechanism to help us find the real media ID (the user has to manually enter these aliases so we know what file to pull).  Well, since it's all just files, we have these 0-length files whose names describe the alias... which are themselves hashed into the various sub-folders.

When you start to get a lot of files in these folders, it can take a long time to find all the aliases for a particular media ID.

Well, we've been wanting to get away from this hackish mechanism for a while, and we've elected to store it all in database instead.

We use a web interface to the database to push and pull the files.  It's actually a pretty good use for HTTP, I think... a protocol precisely designed for this kind of work... why not use it?

(You'll note that I'm glossing over which db and web engine we're using.  You probably already know.  I feel dirty.)

It is surprising difficult to migrate over to this other mechanism.  I have to change all the client machines that fetch these files so they can use a web interface... but the code wasn't well-designed for this kind of change, so (given the time constraints) I had to add hacks upon hacks to get this to work.  And... well, it worked in testing, but not in their environment.  So it was something of a disaster.  I've since asked that we use a 'backup' box they have so I can test our client application before we go live, to avoid this kind of nonsense.

Add to this that I needed to migrate these files into the database.. and my boss wanted me to delete the files as I ran.  I thought that was insane, but he's the boss.  I at least had the wisdom to make a backup of the original set of files (something that gave him a disgusted look, but he agreed), which we have since needed.  I'm so very thankful I insisted upon that... it saved our bacon, since the client didn't handle the web requests entirely properly.  Man, this was bad, and unfortunately our fault.

But there's blame to share.

Now, they didn't like having to register hundreds (or more) of these caption files, one at a time, using our registration tool, so I created this thing that let them drop the files into a folder, and we'd convert it for them.  It's easy to share the folder out so another computer can have access to it... so they can use another computer to drop the files in, so they don't have to actually touch the server.  And our media registration tool is a client to a server, so they could install it on that other computer, too, to ensure the files are going in okay.  Even so, they could also examine the 'error' folder for any problems that might have happened after dropping the files.  Sooo many ways they can protect themselves.

But, like a moth to flame, they plug USB drives into the server, the one machine that makes these files available to the workstations that play back the caption files, causing the machine to get infected with a virus, and wreaking absolute hell on the server.

So, we had this really big discussion about everything yesterday.  We explained our points of concern.  We answered their questions.  We took care of whatever problems arose, and generally worked through a lot of problems caused by a lack of management on their part.  But, most especially, we got them to (after months of this) finally install a new box through which their folks may communicate with our server, making it so only a very few people ever touch this server in any way (me, my boss, and this guy out there who is responsible for handling how they register these media files).

Last night, they had several problems with files not playing out.

I come in this morning, and find this 'bolt.exe' sucking up 50% of the CPU on the server (it's two cores, so basically it's consuming a core), and the only bolt.exe files I can find are in TEMP folders.

Naturally, I thought; "oh, great, another virus."  But according to one of their technicians, it's likely the cleanup tool they used to remove the previous virus... their engineer probably forgot to remove it, and it's trashing the disk (making it harder for me to serve files).

Kind of hilarious, in an oh-fuck sort of way.

It's been a week of this insanity.  It's the technical equivalent to a sitcom.  Only geeks can get it, though.  Enjoy the show.  I'll be here all week.

The primary server blue-screened.

Glorious!

Sun Nov 07 2010 22:26:35 EST from Ford II @ Uncensored

It's been a week of this insanity.  It's the technical equivalent to
a sitcom.  Only geeks can get it, though.  Enjoy the show.  I'll
be here all week.

I'm still laughing. I'm envisioning this group of geek scrambling over a production system and then just when things are almost working, some boob walks by the server and mindlessly plugs a usb drive into it immediately infecting it.
The whole image just keeps making me laugh.
I'm sorry, but it is funny.

That is *exactly* what happened.

And it's hilarious.  It really is.  You can almost hear the embarrassment in the engineer's voice out there.  The sign of resignation.

thats why I call'em winTendo.

"Did you also tell your customers that you are no longer using the server as a virus container?"

"*embarassed laugh* Not in those words, but yeah.  We told them that."

(that was my boss speaking to our customer, not me... I don't think I know the guy well enough to say anything like that).

Though the powers that be are already aware of my plans, I'm giving official notice today.  I'm going to start my new job 16 Feb.

Since I have to burn or sell about 30 days of leave between now and then, I started by requesting about 10 days on the back end, and then every Wednesday off, starting in December.  Wednesdays are our staff meetings.

I've got mixed feelings about it.  I feel a little like I'm abandoning ship, but the boss has made it clear that he has no problem with burning out mid-grade NCOs after two years of the job; there are plenty more where I came from.  Nice.  It will be good to go back to a job with the exact same pay and benefits, but doing my actual MOS and leaving it at work every day (vice waking up at 0200 thinking about things I didn't get done).

In semi-related news, I found out last night that one of my teammates from Afghanistan in '06-'07 died of cancer last month.  He was about 25 or so.  In traditional terms (phone calls, visits, Christmas cards) we weren't close, but if any of my team found themselves with an awkward corpse, I have a shovel and reliable transportation.  We tend to forget that just because the talibs couldn't get us, that doesn't mean we're immortal.