That is an awesome post. If you are thinking about writing distributed systems code you need to understand this behavior of TCP/IP completely. Back when a 100 nodes was a "big" network these sorts of problems were so rare that you could be forgiven for not taking them into account, but these days with virtualization you can have a 500 node network all on the same freakin' host! And the law of large numbers says "rare things happen often, when you have a lot of things."
Fun anecdote, at Blekko we had people who tried to scrape the search engine by fetching all 300 pages of results. They would do that with some script or code and it would be clear they weren't human because they would ask for each page right after the other. We sent them to a process that Greg wrote on a machine that did most of the TCP handshake and then went away. As a result the scrapers script would hang forever. We saw output that suggested some of these things sat their for months waiting for results that would never come.
> If you are thinking about writing distributed systems code you need to understand this behavior of TCP/IP completely.
Not necessarily. If you have a good RPC layer, it will abstract away most of these details. For example, gRPC automatically does periodic health checking so you never see a connection hang forever. (But you do see some CPU and network cost associated with open channels!) gRPC (or at least gRPC's predecessor, Stubby) has various other quirks which are more critical to understand if you build your system on top of it. I expect the same is true of other RPC layers. Some examples below:
* a boolean controlling "initial reachability" of a channel before a TCP connection is established (when load-balancing between several servers, you generally want it to be false so your RPC goes to a known-working server)
* situations in which existing channels are working but establishing new ones does not
* the notion of "fail-fast" when a channel is unreachable
* lame-ducking when a server is planning to shut down
* deadlines being automatically adjusted for expected round-trip transit time
* load-balanced channels: various algorithms, subsetting
* quirky but extremely useful built-in debugging webpages
It's valuable to understand all the layers underneath when abstractions leak, but the abstractions work most of the time. I'd say it's more essentially to understand the abstraction you're using than ones two or three layers down. People build (or at least contribute to) wildly successful distributed systems without a complete understanding of TCP or IP or Ethernet every day.
Hmm, I realize you can't say but when I was there 'Stubby hangs forever ..." was in a painfully large number of bug reports. So gRPC has fixed all that has it? Great to know.
In my experience an abstraction is only as strong as an organization's ability to hold its invariant assumptions, well invariant. And what I took away from that experience was that knowing how an abstraction was implemented allowed me to see those invariance violations way before my peers were starting to ask, "well maybe this library isn't working like I expect it to."
> Hmm, I realize you can't say but when I was there 'Stubby hangs forever ..." was in a painfully large number of bug reports. So gRPC has fixed all that has it? Great to know.
Huh. I don't see bug reports like that. "This RPC with no deadline hangs forever" definitely but I wouldn't call that a Stubby problem. I'd call it a buggy server (one that leaks its reply callback, has a deadlock in some thread pool it's using for request handling, etc.) and a client that isn't properly using deadline propagation.
Application-level keepalives can be handy in other regards, eg. you can assert that your application's event loop isn't blocked on an infinite loop, deadlock, or blocking call in user-level code, and you can pass operational statistics back to your (presumably proprietary) monitoring system.
I found in practice that is usually not enough and every distributed system I designed or worked on ended up with heartbeats of some point. It could OS peculiarities, or could be inability to tweak the times of keepalives.
Sometimes the heartbeats are sent by the server, sometimes by the client, it depends. But they always end up in the application layer protocol.
I feel like putting heartbeats themselves into the application level is a layering violation. They go in the session or presentation layer. WebSockets does it right, with its own heartbeat frame type that gets handled in userland but outside of your app's code.
On Linux you can set tcp_keepalive_intvl and tcp_keepalive_probed to make this much shorter, but it's global to all sockets, so app keepalives are better for finer control, among other things mentioned.
> Back when a 100 nodes was a "big" network these sorts of problems were so rare that you could be forgiven for not taking them into account
Not really. Writing reliable network programs always had to take this into account. Take the scenario (not mentioned in TFA) where a firewall somewhere in the network path suddenly starts blocking traffic in one direction. Or handle a patch cable being pulled from a switch (or from your computer which has a different result). These scenarios always were real and resulted in comparable connection error states.
Do agree it's a good post though, explains it rather nicely.
So there's a way to hang the connection that won't be caught by a `timeout` setting in most HTTP clients? Or the bot writers were just too lazy to add timeouts in their bots?
He had no way to know if they were acutally still waiting months later. Just because the peer has not sent a RST doesn't mean remote end still has a process/machine still in existence.
This practice is a type of tarpitting, and is fairly common and achievable in several ways. I've seen it implemented with several-second delays between bytes, but you have to weigh such active techniques against having the connection and resources set up.
Or you can re-write enough of the TCP stack in user land to do all of the handshaking and then go away. No connections, no resources wasted, infinite numbers of clients waiting forever and ever in their read() call.
Idle but half-dead TCP connections will eventually time out. It can take hours, though, on some systems.[1] Early thinking was that if you have a Telnet session open, TCP itself should not time out. The Telnet server might have a timeout, but it wasn't TCP's job to decide how long you get for lunch.
Today most servers have shorter timeouts, mostly as a defense against denial of service attacks. But it's often the HTTP server, not the TCP level, that times out first.
I find that usually, long-lived idle TCP connections get killed by stateful firewalls, of which there are often several along any given path through the Internet.
e.g. my home router has a connection timeout of 24 hours.
There's quite a number of stateful firewalls that just silently drop the connection with out sending a RST though, meaning TCP connections are idling forever, if it does not employ any tcp or application level keep-alives or timeouts.
There's quite a number of stateful firewalls that just silently drop the connection with out sending a RST though, meaning TCP connections are idling forever, if it does not employ any tcp or application level keep-alives.
> The TCP user timeout controls how long transmitted data may remain
> unacknowledged before a connection is forcefully closed.
As I understand it, this only applies if there is data outstanding. In the puzzler, there was no data outstanding. You're right that if there had been, the side with data outstanding would eventually notice the problem and terminate the connection. The default timeout on most systems I've seen is 5-8 minutes.
By contrast, the previous article you linked was about KeepAlive, which will always eventually detect this condition, but by default usually not for at least two hours.
I'm battling this problem at the movement in a server I wrote (although I'm using a C++ framework above the socket interface which adds to the complexity). The problem only occurs on linux so wasn't detected for a while, and only with one client (running an embedded RTOS). My server ends up stuck in CLOSE_WAIT and therefore wastes a responder thread, and eventually this reaches the limit and the server stops responding completely. It's a really difficult one to debug as it takes about 3 minutes to cause the problem to occur. It's easy enough to see what is going on at the TCP level, but it gets more complex to try and resolve this, as the various software layers add further abstractions above this. It's one thing to see the TCP messages, but another to try and understand this at a higher code level. The CLOSE_WAIT state does appear to timeout but not for a very long time, too long in this case.
If you're stuck in CLOSE_WAIT, it's a bug in your software: You've received a fin and need to close the socket if you're done with it.
The socket should be marked ready for reading, but when you try to read you'll get zero bytes back: Something in your framework may not realize that -- truss/strace the process and I'd guess you'll see a 0 byte read followed by not closing it; alternatively you may not be polling the socket for read availability?
Some things would change if you intended for the socket to be half closed, but I don't think you do?
Depends if TCP keepalives are enabled, but if the connection goes through a NAT gateway, that will certainly have a tracking state timeout. Usually it is on the order of at least hours though, sometimes days.
SO_RCVTIMEO and SO_SNDTIMEO are for setting a timeout on blocking socket operations. They don't tell you anything about whether the other end is still there or not.
Setting those options isn't much different from setting a timeout in your poll() call.
There are some other fun edge cases - for example:
- If you have a port-forward to a machine that is switched off then you can get ICMP network unreachable or ICMP host unreachable as the response to the a SYN in the initial handshake.
This can also happen at any point in the connection. Other ICMP messages can also occur like this (eg. admin prohibited).
It's always worth remembering that the TCP connection is sitting on an underlying network stack that can also signal errors outside of the TCP protocol itself.
I never liked dtruss much for complicated debugging because the output shows way too many things in hex. OTOH, output from strace on Linux tends to be much nicer. e.g.
Yeah same here. When I want to trace my code, boot up a VM just because am more familiar with strace and gdb rather than dtruss and lldb. I can see dtruss having more and fancier features, I but am just too lazy to learn two tools of the same type.
Other than using VMs and 'surprise' power-cycling them, what other ways are there to test these sorts of behaviours? Would an entry like `iptables -I forward -s ... -d ... -j DENY` on a host the 2 endpoints route over suffice to suddenly stop all traffic (or a pair of rules to cover a->b and b->a packets)?
Are there any tools for 'and suddenly, the network disappears/goes to 99% packetloss/10s latency' etc that you can use to test existing software? I'm imagining some sort of iptables/tc wrappers that can take defined scenarios (or just chaos-monkey it) and apply them to traffic, possibly allowing assertions as to how the application should behave for various situations.
I wrote a tool for basically that exact use case. It causes deterministic network faults (on a per-flow rather than per-interface basis), and takes trace files from both sides of the fault.
I used to mimimc the power off scenario using a wired connection and a router. The client machine ethernet connects to an up-linked router, and to mimic the server's poweroff, we just pull the uplink cable out from the router. As far as the client can tell, the network interface is still up (because the router is still alive). If you pull the client side cable from the router, the client's network stack detects the lost network interface and the socket closes immediately.
When a process exits abnormally or not and if there's unread data in the receive buffer then the kernel will send a RST, otherwise it would send a FIN.
"When I type the message and hit enter, kodos gets woken up, reads the message from stdin, and sends it over the socket." <- From Puzzler #1, this is a typo: it should say "Kang", not "Kodos". Kang is sending a message to Kodos, and this message was typed into Kang's console (waking up Kang) which sent the message to Kodos.
Puzzler 1 is when a machine goes down and stays down, and you hang or retransmit until a timeout. Puzzler 2 is when a machine goes down and then comes back up, and you get reset.
One thing missing from this article is TCP heartbeats, which prevents the first two conditions from going on indefinitely.
However since the minimum allowed heartbeat is (usually) two hours host reboot/shutdown is usually detected via a heartbeat at the application layer instead, which sends some no-op traffic every n seconds or minutes.
What he means is that the contrast between the text and background makes the article hard to read. If they would up the font-weight to 400 from 300 it would give a much better contrast.
Fun anecdote, at Blekko we had people who tried to scrape the search engine by fetching all 300 pages of results. They would do that with some script or code and it would be clear they weren't human because they would ask for each page right after the other. We sent them to a process that Greg wrote on a machine that did most of the TCP handshake and then went away. As a result the scrapers script would hang forever. We saw output that suggested some of these things sat their for months waiting for results that would never come.