Based on reading this it seems like you haven't worked on a large application where there are lots of modules and libraries developed by independent teams all coming together to form a final product (shipping with CD or not).
I work on a platform team developing many different libraries used by many different teams throughout the company. If we didn't leverage semver (or some versioning scheme that at least differentiates between breaking and non-breaking changes) I don't know how we would do it. We either a) wouldn't be able to release 'patch' updates to a particular library without consumers getting it for free/automatically without changing anything or b) wouldn't be able to release breaking changes without it automatically breaking consumers builds.
Semver may not be useful for the final build or end product that you end up shipping. But it is a very useful tool for all the parts (dependencies) that make up that final product.
I work on a platform team developing many different libraries used by many different teams throughout the company. If we didn't leverage semver (or some versioning scheme that at least differentiates between breaking and non-breaking changes) I don't know how we would do it. We either a) wouldn't be able to release 'patch' updates to a particular library without consumers getting it for free/automatically without changing anything or b) wouldn't be able to release breaking changes without it automatically breaking consumers builds.
Semver may not be useful for the final build or end product that you end up shipping. But it is a very useful tool for all the parts (dependencies) that make up that final product.