DisplayPort and DVI both support HDCP. This wasn't the purpose behind HDMI, though support for it was no doubt a requirement for adoption. It was designed to be a single cable for carrying video and audio between playback devices, receivers, and displays.
For this purpose, it succeeded and did a much better job at it than alternatives. HDMI still makes far more sense for use in a home theater environment than DisplayPort thanks to features like ARC.
I think the better question is why SDI video connections aren't available on any consumer devices.
While HDMI is nice for consumers because it carries audio/data, SDI cables are cheap (just a single coax cable!) and easy to route (T-splitters are a thing!).
I think cost might be the main factor there. SDI is serial instead of parallel like all other consumer digital video cables. Hardware to serialize and deserialize bits this fast on a single conductor is expensive. HDMI 1.0 had a max transmission rate of 4.95 Gbit/s in 2002. Today, HDMI 2.1 goes up to 48 Gbit/s.
HDMI is great for a home theatre set up where there's an obvious central unit, but the ecosystem has gotten worse if your speakers don't take in HDMI, at least at the very cheap end of the spectrum I buy on.
My current TV will only put out an inferior "headphone" mix over the 3.5mm connection, and the SPDIF connection is co-axial on the tv, but optical on the speaker. Having to run a powered converter box just to get audio from my tv to a speaker feels like such a step backwards.
It sounds like your speaker system predates HDMI's adoption, or was never intended for use in a home theater system. Even the lowest budget soundbars will include an HDMI port and support ARC. I am surprised your TV has coax SPIDF but not toslink, as it was the gold standard for home theater audio before HDMI came around.
It sounds like you just got bad luck with your TV having a bad 3.5mm jack and not supporting toslink, as most I've seen will at the very least support the latter, and often have a separate "line out" port for the former.
3.5mm is an analogue signal that can only output stereo. Its quality will be limited by the device it comes out of.
SPDIF (both optical and coaxial) support sending multiple audio channels in their original encoding, to support surround sound. The receiving device needs to support decoding of the audio, which is why there is often an option to force the SPDIF output to use PCM stereo instead of surround sound.
If you have a cheap TV and a good hifi, you'll want to use SPDIF or HDMI so that your audio isn't ruined by poor quality of the audio chipset in the TV.
My assumption was just that it was something about how the EQ is mixed for that jack, because it is labelled specifically as headphones. The other replies about surround sound are true, as well, but I don't think should apply to my tv -> stereo soundbar set-up.
But the difference is definitely noticeably, even to my relatively forgiving ear.
For this purpose, it succeeded and did a much better job at it than alternatives. HDMI still makes far more sense for use in a home theater environment than DisplayPort thanks to features like ARC.