a) packets are arriving from a fast link and don't fit onto a slow link. Dropping packets signals TCP or your UDP protocol to adjust to the slower link speed, exactly the desired behaviour.
b) the router buffer is too small to handle uneven bursts of packets even when the destination link has spare capacity. This is a misbehaviour and will cause needless packet loss.
Then there's the "too much buffering" failure mode
c) the router buffer is too big, so that there is constantly a long queue of packets in the buffer at the upper limit of the link speed, causing excess delay. This is also called "buffer bloat" and is a sin often committed by consumer ADSL modems and such.
So, an UDP protocol with retransmission capability could possibly overcome b) at the cost of behaving counterproductively in the presence of real congestion.
The remaining problems would be to estimate the actual link
bandwidth, and figuring out how to retransmit the large portion of lost packets while retaining good app layer latency.
But that's a rare problem to have in the network because the failure mode is so obvious (TCP stops working properly even though your network is idle). c) is much more common.
But, a) is a buffer issue too. Of course no buffer is big enough if the stream is continuous. But the backoff works exactly because it doesn't overwhelm the router buffer, leaving it time between bursts to empty into the slower network.
When TCP 'backs off' that's failure to a streaming protocol - now we get to spend seconds retransmitting and performing 'slow start' probing to determine an ideal rate. Which blows the latency of time-critical packets to the point its useless (as covered elsewhere in this thread).
And this is almost always the upstream link - your local home or business network is probably gigabit. Your ISP link is very likely slower than that.
TCP would really suck if it went to multi-second coma on every bandwidth adjustment!
Buffer getting "overwhelmed" is exactly what causes TCP to adjust.
Fortunately TCP doesn't go to slow start immediately when it sees congestion in the network.
It first tries slowing down. There's a "congestion avoidance" phase as long as not too many packets are lost, "slow start" happens only when congestion avoidance gives up.
Excess buffering can infact mess up congestion avoidance in addition to causing needless additional latency, since it in effect postpones the timely signaling of link capacity. See the bits about congestion control at https://www.bufferbloat.net/projects/bloat/wiki/TechnicalInt...
After this probing initially happens, TCP will remember the link bandwidth, window size etc even after periods of inactivity.
This is also how UDP based protocols are expected to behave (specced by IETF/IAB).
All cool if you have one connection and don't share it. But real apps on real computers have multiple apps running - while you're trying to communicate, email fires up and scans a dozen servers; your progeny upstairs clicks on a video; ROKU decides to update.
Bandwidth and window size are a fiction over time. TCP believes they will be stable (for many seconds); its tuned for old server-to-server connections, not the extremely dynamic network conditions in the average home/business today.
So you end up in slow start and congestion avoidance, and only after packets are dropped and resent pointlessly adding to congestion (as you point out).
I've spent years now examining traces of TCP catastrophes. It's because TCP is NOT dynamic enough that folks like my company are driven to finding alternatives.
