It’s crazy sometimes how big of a difference it is. One recent example - I had to build a custom Docker image of some OSS project. Not even a huge one - only what I would call small-mid size. Just clone the repo and run the makefile, super simple. It took 35 minutes to build on my 2020 Mac Mini (Intel) and would have been probably half that if I had the most recent machine.
Why would I build on a local machine vs running the build on a server in a datacenter? Per your own arguments, server grade hardware is going to compile much faster than any local workstation.
Ah, good old "compiling" [0]. When a worker needs a $4000 machine to actually do his work then it's unavoidable. The slow machine? $2000 out to be enough™ for everyone else.