Networking

I have what might be a sort of fun issue for you guys to ponder.  I'm not really asking for advice, although I'm curious as to what you might think about the situation.

A customer installed one of our boxes in their facility.  The box stopped working properly a couple of days ago.  They sent the box to us, and it's working without any issues at all here in our networking environment.

When it was out there, if I used a remote login service to connect to the desktop of the box immediately after the box booted up, I could log into it.  Otherwise, I couldn't access the box at all.

When I was finally able to get into the box, I noticed our services couldn't be restarted.  Upon start, they would generate an error message indicating that the box itself was out of networking resources.

Have you ever seen that before?  Personally, I've never observed that problem on a box.  I guess I've always been in networked environments that were properly configured and designed.

None of our other boxes out there have this kind of problem, but admittedly, this box is kind of special, in that we have to have a single port made available to the outside world.  Not a big deal, really... just port-forward to our box, and everything is peachy-keen.  Nobody else has a problem doing that... but I suspect these guys did something "different"... since he was concerned that port-forwarding was a security issue.  He felt more secure just exposing the entire box to the outside world and dropping our internal firewall.

(Note: The box doesn't appear to be infested, in case you're wondering... virus scans find nothing on it).

At the moment, the box is safely in our own facility.

But someone out there would need to investigate it, not me.

I think your assessment, though, is spot on.  Someone is hammering the box from outside their network, and they aren't handling it properly.  Likely, the problem will go away if they just put a firewall in place and forward the port we want.

Heh... so we can repair it from the damage of improperly securing the network?

The plot thickens.

Yesterday through today, the spare box we sent them worked well.  The box that they returned to us showed absolutely no problems, so we shipped it back without changing anything on it (except to ensure nobody could modify the executables).

They swapped the two boxes out, and now the box they returned to us is dead on their network again.

One difference between the two boxes: the spare we sent didn't have an a/v capture card, so it didn't make sense to install all the bits that allow it to stream a/v content to stenographers... which is the one port that needs to be exposed to the internet.

I'm starting to wonder if the network engineer was forwarding the port in some bizarre way that floods the box.

I also wonder if he opted to use a MAC address to forward stuff... such that the other box wouldn't have been a problem anyway (because it didn't have the right MAC address).

In the end, I told them that the box worked flawlessly in our facility, and that the problem must therefore be with their network in some way.  When they said the other box worked fine, I told them I couldn't explain why (which, technically, I can't... I have no idea how they have their network set up).  I then told him that I didn't particularly care what their network policy was, or how they have their network environment set up... I am only responsible for ensuring our equipment works properly, and it does.

We'll see if they ever figure it out.  In the meantime, their customer is going berserk.  I wonder if they'll lose the account.  I wonder if someone out there will get fired for this.  I wonder who that person will be (the network engineer who screwed up their network and is protected by the CEO, or his boss, who can't fire him).

Well, after weeks of this kind of nonsense, it gets harder not to point out the nature of our working relationship.  They've been abusing the hell out of our technical support contract, and we don't appreciate it.