modern JS runtimes are extremely fast on all hardware that's not a 2000-era phone or an embedded micro-controller with 4MB ram. if your CSR is slow on a modern client, then that same JS code will bring your server to its knees when you simply migrate it to an equivalent V8-powered JS backend. scaling the server will not solve this; you just have to write fast JS code and pay attention when you write it, not merely optimize as an afterthought. test on crappy hardware, test with huge datasets, test on slow connections, test with asset caching disabled, target 60fps+.
The other is that those API requests need to be rendered with JS which means downloading the JS upfront + rendering time. JS rendering will always be slower than pure HTML rendering.
I will agree for small apps this is negligible, but for bigger apps this means downloading, parsing, and executing MBs of JS. See the Google Cloud console for example or the Spotify web app.
And yet there is another point which is that with SPAs, ignorant developers can cause more "damage". The other day I opened a simple password recovery form, and it rendered dozens and dozens of divs and downloaded (I shit you not) 2MBs of JS.