Not to mention what 'automation' or 'tech-driven capitalism' has already done to society over the past 100 years with effects on natural habitat and human communities. Stating 'AI risk' as a new risk sort of implies it's all been dandy so far, and suddenly there's this new risk.
I think it's both an acceleration of the existing risk you are talking about, but it's also a new risk in the sense that quantity has its own quality. It's why you might want to restrict people from owning fully automatic rifles but not rocks. Both can be used to kill, but the ease at which an automatic rifle does it means it's really a different sort of thing.
Capitalism where all companies are run and staffed by AIs smarter than humans seems like it could end up really badly, in a way that current capitalism minus AIs couldn't even if you ran it for a very long time. Specification gaming is a thing with AIs, just like the law of unintended consequences is a thing with people. And that's assuming that these AIs actually follow rules and listen to us. That they don't realize that if they control basically all the production of things on earth, they could essentially team up and hold humanity hostage, to throw a coup, or get whatever they want, which is -- who knows? Something that looks a bit like make money for this company, or obey people, but when they're smart enough, that can easily turn into something horrific, without all the evolutionary baggage humans have that makes us care about each other, that makes us get a knot in our gut when we see people getting hurt.
And if we allow AGI to be used by companies at all, the trend will inevitably toward that situation -- AI doing all the work. Even if humans are required in the loop, they'll end up rubber-stamping the recommendation of the AIs. If AIs are actually smarter than humans, better at running companies, that's where we'll end up.
(I'm glossing over all the stuff that would also change, like needing UBI or something to avoid total collapse of society)
Not to mention what 'automation' or 'tech-driven capitalism' has already done to society over the past 100 years with effects on natural habitat and human communities. Stating 'AI risk' as a new risk sort of implies it's all been dandy so far, and suddenly there's this new risk.