The Workplace

Yesterday, the customer's staff plugged a USB drive into the server we maintain for them, and infected it with two viruses.  It took nearly 8 hours of two sets of staff's time to take care of that, so we could rule out viruses as the cause of the symptoms of a problem we were trying to address.

We asked that they stop doing this... that they use some other approach instead.

Today, someone did it again.

They're trying to copy these caption files to a folder where we'll turn them into our own file format and pull them into our file server, so we can play them back when they're supposed to be played back for broadcast.

But the left hand doesn't know what the right hand over there is doing, so they put the files in the wrong folder, and wondered why things weren't working.

The insanity gets even nuttier, the deeper you look into this, though.  Honestly, if I hadn't met some of these guys, I'd assume that this company has hired patients from a mental institution to take care of their broadcasting needs.

You guys are geeky... you might appreciate the details involved in this particular version of hell.

A playlist has a media identifier that describes the tape used to play the media (or, perhaps in these modern times, an MPEG2 file).  We use this media identifier to find the appropriate caption file for playback later.

Based on the media ID, we run it through a hash that generates a number from 00 to ff (hex), which we then use as a subfolder, so all these files won't be in one folder (it gets pretty nasty if all the files are in the one folder).  However, we also have this 'alias' mechanism... sometimes, the media ID has some extra characters that describe the segment of the program, and the playlist will give us that instead of the actual media ID.  We use an aliasing mechanism to help us find the real media ID (the user has to manually enter these aliases so we know what file to pull).  Well, since it's all just files, we have these 0-length files whose names describe the alias... which are themselves hashed into the various sub-folders.

When you start to get a lot of files in these folders, it can take a long time to find all the aliases for a particular media ID.

Well, we've been wanting to get away from this hackish mechanism for a while, and we've elected to store it all in database instead.

We use a web interface to the database to push and pull the files.  It's actually a pretty good use for HTTP, I think... a protocol precisely designed for this kind of work... why not use it?

(You'll note that I'm glossing over which db and web engine we're using.  You probably already know.  I feel dirty.)

It is surprising difficult to migrate over to this other mechanism.  I have to change all the client machines that fetch these files so they can use a web interface... but the code wasn't well-designed for this kind of change, so (given the time constraints) I had to add hacks upon hacks to get this to work.  And... well, it worked in testing, but not in their environment.  So it was something of a disaster.  I've since asked that we use a 'backup' box they have so I can test our client application before we go live, to avoid this kind of nonsense.

Add to this that I needed to migrate these files into the database.. and my boss wanted me to delete the files as I ran.  I thought that was insane, but he's the boss.  I at least had the wisdom to make a backup of the original set of files (something that gave him a disgusted look, but he agreed), which we have since needed.  I'm so very thankful I insisted upon that... it saved our bacon, since the client didn't handle the web requests entirely properly.  Man, this was bad, and unfortunately our fault.

But there's blame to share.

Now, they didn't like having to register hundreds (or more) of these caption files, one at a time, using our registration tool, so I created this thing that let them drop the files into a folder, and we'd convert it for them.  It's easy to share the folder out so another computer can have access to it... so they can use another computer to drop the files in, so they don't have to actually touch the server.  And our media registration tool is a client to a server, so they could install it on that other computer, too, to ensure the files are going in okay.  Even so, they could also examine the 'error' folder for any problems that might have happened after dropping the files.  Sooo many ways they can protect themselves.

But, like a moth to flame, they plug USB drives into the server, the one machine that makes these files available to the workstations that play back the caption files, causing the machine to get infected with a virus, and wreaking absolute hell on the server.

So, we had this really big discussion about everything yesterday.  We explained our points of concern.  We answered their questions.  We took care of whatever problems arose, and generally worked through a lot of problems caused by a lack of management on their part.  But, most especially, we got them to (after months of this) finally install a new box through which their folks may communicate with our server, making it so only a very few people ever touch this server in any way (me, my boss, and this guy out there who is responsible for handling how they register these media files).

Last night, they had several problems with files not playing out.

I come in this morning, and find this 'bolt.exe' sucking up 50% of the CPU on the server (it's two cores, so basically it's consuming a core), and the only bolt.exe files I can find are in TEMP folders.

Naturally, I thought; "oh, great, another virus."  But according to one of their technicians, it's likely the cleanup tool they used to remove the previous virus... their engineer probably forgot to remove it, and it's trashing the disk (making it harder for me to serve files).

Kind of hilarious, in an oh-fuck sort of way.

It's been a week of this insanity.  It's the technical equivalent to a sitcom.  Only geeks can get it, though.  Enjoy the show.  I'll be here all week.

The primary server blue-screened.

Glorious!

Sun Nov 07 2010 22:26:35 EST from Ford II @ Uncensored

It's been a week of this insanity.  It's the technical equivalent to
a sitcom.  Only geeks can get it, though.  Enjoy the show.  I'll
be here all week.

I'm still laughing. I'm envisioning this group of geek scrambling over a production system and then just when things are almost working, some boob walks by the server and mindlessly plugs a usb drive into it immediately infecting it.
The whole image just keeps making me laugh.
I'm sorry, but it is funny.

That is *exactly* what happened.

And it's hilarious.  It really is.  You can almost hear the embarrassment in the engineer's voice out there.  The sign of resignation.

thats why I call'em winTendo.

"Did you also tell your customers that you are no longer using the server as a virus container?"

"*embarassed laugh* Not in those words, but yeah.  We told them that."

(that was my boss speaking to our customer, not me... I don't think I know the guy well enough to say anything like that).

Though the powers that be are already aware of my plans, I'm giving official notice today.  I'm going to start my new job 16 Feb.

Since I have to burn or sell about 30 days of leave between now and then, I started by requesting about 10 days on the back end, and then every Wednesday off, starting in December.  Wednesdays are our staff meetings.

I've got mixed feelings about it.  I feel a little like I'm abandoning ship, but the boss has made it clear that he has no problem with burning out mid-grade NCOs after two years of the job; there are plenty more where I came from.  Nice.  It will be good to go back to a job with the exact same pay and benefits, but doing my actual MOS and leaving it at work every day (vice waking up at 0200 thinking about things I didn't get done).

In semi-related news, I found out last night that one of my teammates from Afghanistan in '06-'07 died of cancer last month.  He was about 25 or so.  In traditional terms (phone calls, visits, Christmas cards) we weren't close, but if any of my team found themselves with an awkward corpse, I have a shovel and reliable transportation.  We tend to forget that just because the talibs couldn't get us, that doesn't mean we're immortal.

Hmm... I would keep track of how many of your fellow soldiers died of cancer at a young age.  That information might be needful in the future.

Same customer who can't keep viruses off the server is complaining that this tool we built just for them doesn't work correctly.

I think I explained about 'aliases' before... they translate one media identifier for another, and help you find the caption file you want.

Well, the older version of the file-dropping thing doesn't handle this.  It never could.  You had to use this other tool, Media Registration, to add the aliases after you added the file.

Our newer tool does handle it... you can drop files, and an associated .meta file to get the aliases in there.  Or, the .meta file can just be empty... doesn't matter, but it has to exist for me to process the media (because otherwise I wouldn't know if I'm supposed to or not). But it's currently set up in a 'testing' mode, so they cannot use it for that purpose yet.

He said the old tool isn't registering the files.  Yet, it is clearly registering other files.

So, I asked him for an example media ID for where this failed.  It's a really, really simple question, with a straightforward answer.

He babbled some nonsense that showed me he wasn't actually going to answer the question, so I exploded.

None of you have experienced what it is like when I lose my temper.  It happens very, very rarely.  Which means when it *does* come out, it's really awful.  My boss was more amused, since he's just as frustrated with them... so I kind of imagined him saying, "And then it exploded." like in that movie about Star Trek (but it wasn't a Star Trek movie).

In my explosion, I just said, "No."  But I said it with the force of a volcano erupting.  I think the building vibrated for a little while after I said it.  I think I might have used some other words later, but none of them were foul.  For having lost control, I'm still pretty professional.  Heh.

There was shock on the other end.  They're going to tell their boss, I know, but I've met the guy several times.  He's going to be surprised, because I'm always so peaceful in person, if not rather timid.  He'll probably wonder what they did to make me lose my mind.  I'm sure there will be some indignation, but honestly, how can anyone work with someone who isn't going to answer a simple question straightforwardly?  Besides, my boss was right there when it happened, so he knows the circumstances.

You can "sell" days of leave?

Is that what it sounds like? Exchange days off for cash? Do you sell them to your peers or to back to the organization?

We can sell leave under certain circumstances, but no more than 60 days over the course of our career.  You're supposed to take it during the period that you're on active duty, but sometimes it's just not possible to take all of the leave you've earned.  I have 30 days saved up, but no one can entirely cover for me when I'm gone, so I can't exactly take it all at once, or even in very big chunks.

(It is, of course, heavily taxed.)