TCP does what it's designed for: Reliable, ordered, stream connection with fixed end-points. UDP is the other extreme of this permutation. Wonder why no one explores the spectrum in between?
All of those properties are binary things -- it's either reliable or not reliable, ordered or not ordered, etc. The notion of mostly ordered or mostly reliable we sort of get "for free" with UDP. So then your question becomes: what about some properties and not others? There are ways around the fixedness of endpoints so I'll just look at reliability and ordered-ness.
Unreliable but ordered: use UDP, enumerate your packets, and if you get a packet out of order, discard it. No cheaper than UDP, and I can't really see a potential benefit over it.
Reliable but unordered: for this to make sense you have to impose a timeout (i.e. if you're willing to wait forever, UDP is reliable in that you can never be sure you won't ever receive that packet). So now you have ACKs and NACKs and you essentially have TCP minus congestion control and where you don't bother to re-order packets based on seq #. I can't really see the benefit of this either.
That said, there are many non-tcp-non-udp protocols out there. I just wouldn't say that the protocol-space is a spectrum with TCP on one side and UDP on the other -- there are many, many dimensions to look at.
