It's necessary to share file handles between processes using TCP over UNIX Domain Sockets. I'm sure some other mechanism could be devised for that though.
Minor nit: While Unix Domain Sockets (AF_UNIX) can send open file descriptors between processes, and it uses the msg_control field of sendmsg's struct msghdr ("ancillary data"), that's not the same as TCP's OOB data and in fact TCP is not involved at all in unix sockets, even when used as SOCK_STREAM. sendmsg() even has a MSG_OOB flag that is used to trigger TCP's OOB mechanism that (as far as I can tell) doesn't use the msg_control field at all, as the same option is available for plain send() and sendto(); the OOB data should be sent through the normal msg_iov field.
Some places where msg_control is used for TCP/IP sockets (at least in Linux) include kernel timestamping of messages. This is "out of band" data, but it is out of band to the kernel, not the network peer.
Weird. I guess I built a mental model that felt like it made sense. Turns out computers are crazier than common sense. Thanks for the detailed reply :)
