The AGI does not need to control the Pentagon. It might for example invent novel technologies that you and I cannot even imagine because we are much less capable at inventing new technologies than the AGI will be.
If members of an alien civilization much more advanced than our own suddenly appeared in orbit around Earth, would that not be dangerous?
That I understand, but my thinking automatically goes to control and restrictions. A superintelligent AGI that's as restricted as chatgpt in terms of what it can actually control seems... not so existentially dangerous? In that scenario, that actual problem would rise by not-so-smart humans listening to that AGI blindly, no?
This said, ChatGPT with a plugin is potentially dangerous. We are lucky enough that it gets caught in loops pretty easily and stops when using tools.
But lets imagine a super intelligence that can only communicate over the internet, what can it accomplish? Are there places you can sign up for banking accounts without a physical presence? If so, then it could open accounts and move money. Money is the ultimate form of instrumental convergence
Once you have money you can buy things, like more time on AWS. Or machine parts from some factory in China, or assembly in Mexico. None of these require an in person presence and yet real items exist because of the digital actions. At the end of the day money is all you need, and being super intelligent seems like a good path on figuring out ways to get it.
Oh, lets go even deeper in to things that aren't real and control. Are religions real? If a so called super intelligence can make up its own religion and get believers it becomes the 'follow AGI blindly' scenario you give. It gives me no more comfort that someone could wrap themselves in a bomb vest for a digital voice than it does when they do so for a human one.
I'm confused by your "as restricted as chatgpt". Hasn't chatgpt been given unrestricted access to the web? Surely if chatgpt were sufficiently capable, it would be able to use that access to hack web sites and thereby gain control over money. Then it can use the web to buy things with that money just like you and I can.
It is a lot harder to prevent the sysadmins who have temporary control over you from noticing that you just hacked some web sites and stole some money, but a sufficiently capable AI would know about the sysadmins and would take pains to hide these activities from them.
It doesn't even need to hack or do anything illegal. It can make money trading stocks, predicting or modelling far better than market participants, use money to influence the real world, bribe politicians etc. Like with few billions, i bet you can create a political party, influence campagins and end up with real human followers and lobbying power.
It doesn't even need to run on its own, if there was a trading bot that said, in order for it to make me money, I need to give it internet access and run it 24/7, rent gpu clusters and s3 buckets etc, I'd do it. this is the most probable scenario imo, AI creating beneficial scenarios for subset of people so that we comply with its requests. very little convincing is necessary
The aliens wouldn't be dangerous. There is no motivation for them to be. There is an opportunity cost for them to come here. To arbitrarily harm the human population is not a good justification to incur that cost.
The human would probably become dangerous. We get kind of stupid when we're scared. But AGI would be of our own creation. Why do we need to fear our own child?
If members of an alien civilization much more advanced than our own suddenly appeared in orbit around Earth, would that not be dangerous?