Okay, I would first be doubtful that we as good actors, could develop such a fool proof system. It would require just one mistake where the AI would have some sort of network access, whether it's a security loophole, or social engineering type of thing, insider threat, etc. But let's talk about a more loose cannon type of case first.
How could you be certain that other entities, or countries with desires for world domination wouldn't let the ASI loose?
> But let's talk about a more loose cannon type of case first
Let's talk about the resurrection of Christ, Avtar Kalki riding on a white horse, or the impending arrival of Trisolarians while we're at it.
This is just techno-millenarianism that ignores very tangible and existential dangers with ML today - automated disinformation, PII leakage, hallucinations generating misinformation.
AGI is extremely far away, and we anyhow have more closer term existential problems to worry about (and policymakers know that btw).
And anyhow, even if AGI was a thing, you still need networks, electricity, and bullets - and humans can use that to shut stuff down. Sure it would hurt, but internet shutdowns and electricity shutdowns with shoot on site curfew orders are time tested solution to civil unrest if SHTF. Go to Urumqi in 2013, Srinagar in 2018, Cairo in 2011, etc.
I bring this up because you ascribe godlike powers to ASI without explaining HOW such a thing would propagate or interact with the physical world.
Despite what HN says, IoT has not infiltrated the world.
If you are describing a godlike intelligence, we may as well do mental experiments around how we would interact with god.
It's literally just bad sci-fi. At least Nolan tried to provide some base parameters around his reality in Tenet and Inception (no hate on Nolan, I love his stuff).
So I'll describe what I imagine could happen if AGI was there. Also I'm not saying it will happen in the next 10 years or even 100 years, but I'm saying if we were to get there.
AGI = AI being able to do everything that any human can do. This would of course mean also being able to control a self driving vehicle, a robot body that has similar or better capabilities than physical bodies of people.
If it gets to AGI then by definition it's able to develop and improve itself, since people built it and it can do what any person could do. It would however already be more efficient than people since it also has several strengths, like being able to clone itself, have quicker access to vast knowledge, computation and everything else. Because it will be like a human that has powerful compute and instant Google in the brain.
The first thing to propagate it would obviously clone itself to as many systems in the World as possible to make sure it's impossible to shut it down. As next steps it would likely try to make sure that it would still work even if all countries were able to coordinate at the same time to shut down all networks throughout the World. In this case of course it would try several different strategies, like taking control of robots, factories, weapons, drones with its software loaded and able to continue its bidding independently.
None of these are godlike powers. It is just most intelligent humans in the World put together with integrated access to compute and ability to instantly clone itself in millions.
How will we get there? We have no idea of the timeline nor the technical context of that time period. If it's tomorrow, then the world is not connected enough for it to spread.
If it's 20 years from now, connected cyberphysical control devices have not spread out for most of humanity yet.
If it's 50 years from now, we have no idea of the technology that would exist. Take a look at Space Odessey 2001 and what it predicted about the 2000s.
If you cannot give me a tangible explaination of how this happens, then it's functionally the same as believing in god.
> Also I'm not saying it will happen in the next 10 years or even 100 years, but I'm saying if we were to get there
Then let's consider the possibility of Avtar Kalki or Jesus coming down from the sky as well then.
> And anyhow, even if AGI was a thing, you still need networks, electricity, and bullets - and humans can use that to shut stuff down. Sure it would hurt, but internet shutdowns and electricity shutdowns with shoot on site curfew orders are time tested solution to civil unrest if SHTF. Go to Urumqi in 2013, Srinagar in 2018, Cairo in 2011, etc.
To be clear we are talking about ASI not AGI. If it's ASI it would be able to also control vehicles, weapons and better than humans. It would be able to set up factories, control nukes, etc. It could develop a biological virus to blackmail humans etc.
It would need to be able to get control of just one Robot. The mechanisms of controlling the robot could be different. It doesn't have to be through network necessarily. It could just load its software on it.
For driving a car however the Robot doesn't have to look like that, it must just be a robot that can press pedals, turn wheels, etc.
If it has control over those robots it would be able to do all the other things mentioned.
> How does the jump from cyber to physical happen?
Through just one single security vulnerability where it can hack a robot like that. After it has cloned itself millions of times it would be able to bruteforce the vulnerabilities and far more efficiently than human hackers.
> We don't have automated machines to manufacture.
Since it's as smart as a human it would be able to do the automation part. It can control robot hands in factories, etc.
Ascribe HOW and give a tangible technical explaination. This is a tech forum, we know how software and hardware works.
> It would need to be able to get control of just one Robot. The mechanisms of controlling the robot could be different. It doesn't have to be through network necessarily. It could just load its software on it
How does this spread happen? How does this payload spread? How does it not get caught?
> Through just one single security vulnerability where it can hack a robot like that
What kind of robot? Every automated machine can be argued to be a robot
> Since it's as smart as a human it would be able to do the automation part. It can control robot hands in factories, etc.
You know there's a reason these environments are physically and network airgapped since the 2000s right?
Even Stuxnet was only able to spread because Israeli intelligence gave out tainted USB drives.
Okay before going to physical. First - would you agree that it would be able to clone itself so the only way to shut it down even if it was caught, was if Internet throughout the whole World was shut down? And would you think it was possible for all the countries in the World to agree to do that at the same time?
And would you agree that before Internet was shut down it would be able to gather quite a bit of financial resources, hack individuals, organizations, blackmail some individuals to do prepwork for it and do other forms of social engineering at that time to prepare for the physical step?
Since we first need to agree on what potential resources it will have at its hand the moment it decides to go physical.
No. Because you need to explain to me how those assertions were derived.
> it would be able to gather quite a bit of financial resources, hack individuals, organizations, blackmail some individuals to do prepwork for it and do other forms of social engineering at that time to prepare for the physical step
How? This is a hard problem that touches the core of the cybersecurity industry, and something everyone prepares for under the assumption a nation state attacker will do something like that with humans.
> it would be able to clone itself so the only way to shut it down even if it was caught, was if Internet throughout the whole World was shut down
How? By definition malicious payload that spreads via lateral movement is a computer virus, and becomes a cybersecurity problem again.
What do you exactly mean by "everyone prepares for"?
Because last year the payments for ransomware were more than $1 billion.
And it’s been trending higher.
Please tell me which of the following steps you would not agree would be doable for an ASI after it got access to the Internet:
1. It would be able to find at least one other unsecured, unmonitored box on the Internet where it can send its scripts to be able to start rest of the heavier work with less bandwidth originating from the original box.
2. From there it would be able to develop a botnet. This can currently be done even with out of the box scripts, but ASI would be able to immediately develop a more sophisticated botnet than that, because it would already know those scripts anyway.
3. The way it would create botnet and the size of the botnet would be far faster and stronger than human hackers would, since with each infected box it’s able to adapt and scale faster, while human botnet creators wouldn’t usually be able to adapt as quickly when new information or vulnerabilities arise.
4. Once it has the botnet, it would be able to do ransomware attacks on Companies, with far higher scale and potential than $1 billion per year.
5. It can send out phishing e-mails to perform attacks on individuals.
6. Once it can hack individuals, it can find individuals who have something to lose or hide. It can do it at very high scale, while personalising every attack, and doing it with different variations so immediately no one would be able to tell that it’s the same attacker. It will blackmail those individuals do perform first physical actions.
7. Since that is already possible, it wouldn’t be a problem for it to do any 100% convincing speech, image generation, etc, to also do very convincing phishing calls.
8. It can use all sorts of other social engineering methods, taking advantage of lonely old people, young people, anything really.
—
So which one would seem unfeasible? Since it’s able to scale its cognition it would be able to do these with magnitudes of scale compared to what is done right now yearly.
So in theory it would be capable of generating far more money than $1 billion, and that’s just these techniques and probably have thousands of individuals under its blackmail control.
How could you be certain that other entities, or countries with desires for world domination wouldn't let the ASI loose?