A quick search suggests to SET so_linger with a small/zero timeout
just before calling close(). This will have the server deliver a RST
TCP packet, which bassically equals to sending a message into the void

This makes sense. There's that whole "half-closed" thing we used to deal with on the web side, if I remember correctly.

I think part of the problem may be that these sessions are stuck in the middle of a transaction, which is to say that the server is still either sending or expecting data, as opposed to sitting idle waiting for the next command.
Citadel Server binds a session to a thread while a command is in progress, and unbinds it when the command completes. Both the idle timer and the cancelled session reaper skip right past sessions that are bound to a thread.

If this is what's happening, then I need to totally re-evaluate how that works. The winning move might be to ignore the session state completely, look only for idle time, and instead of terminating them by setting them to be reaped, just perform that so_linger/close operation on the socket. This would theoretically cause the connection to reset and then it would get cleaned up by itself just as if the other end terminated the correct way.

I've added an extra parameter to the session listing code in the server.
Once this update makes it to the production system, the next time someone does this to the system I can look at the session listing and confirm that this is what's happening.

 I have seen in the past one slow drive on  your bus can drag down
the others.   I suppose it depends on how they wire the bus on
whatever board you get, but i have seen it.

I can see how that might happen. On the other hand, my desktop is set up that way (SATA SSD main drive, HDD for backups) and there doesn't appear to be any impact. I wonder if it has anything to do with the fact that SATA still emulates the old primary/secondary setup of IDE, and ST-506 before that, and I have the drives on different controllers. Basically I have them as drives 1 and 3 instead of as drives 1 and 2.

If that is the case then I'll be sure to put the SSD's on 1 and 2, and the HDD's on 3 and 4.

Sounds like it was most of AT&T and some of Verizon and T-Mobile. That makes sense because carriers sometimes buy roaming capacity from each other.

Hey, sometimes it happens. I had the pleasure of visiting the headquarters of CenturyLink/Level3 just a short while after their nationwide outage (and got a sweetheart deal on some nationwide 100 Gbps capacity because they were still trying to recover from it at the time). Just for the asking, they showed me all of the different kinds of network management equipment they used, and pointed out which vendor's equipment had the cascading failure and why.

Anyone who works in technology knows that shit happens. As much fun as it is to immerse oneself in conspiracy theories -- indeed, I test them that way -- sometimes a network outage is just a network outage.

That having been said, I sincerely hope the big three cloud providers have catastrophic and permanent outages that they never recover from. They're all too big.