Good point. Although in some ways it's a lower bar, since agents that can control organizations can delegate most of the difficult tasks.
Most rulers don't invent their own societies from scratch, they simply co-opt existing power structures or political movements. El Chapo can run a large, powerful organization from jail.
That would require a high degree of integration into human society though, which makes it seem very unlikely that AIs would doggedly pursue a common goal that is completely unaligned with human societies.
Extinction or submission of human society via that route could only work if there was a species of AI that would agree to execute a secret plan to overcome the rule of humanity. That seems extremely implausible to me.
How would many different AIs, initially under the control of many different organisations and people, agree on anything? How would some of them secretly infiltrate and leverage human power structures without facing opposition from other equally capable AIs, possibly controlled by humans?
I think it's more plausible to assume a huge diversity of AIs, well integrated into human societies, playing a role in combined human-AI power struggles rather than a species v species scenario.
Yes, I agree that initially, AIs will be highly integrated, and their goals will probably at least appear to be somewhat aligned with those of human societies. Similar to human corporations and institutions. But human institutions and governments go off the rails regularly, and are only corrected because humans can sometimes go on strike or organize to stop them. I fear we will lose those abilities.
As a concrete example, North Korea was forced to allow some market activity after the famines in the 90s. If the regime didn't actually require humans to run it, they might easily have maintained their adherence to anti-market principles and let most of the people starve.
Most rulers don't invent their own societies from scratch, they simply co-opt existing power structures or political movements. El Chapo can run a large, powerful organization from jail.