Tegmark isn’t a “top scientist”, but just a scientist. He’s definitely not an expert in AI, and is a cosmologist. What he’s known for is being president and founder of the Future of life institute (https://en.wikipedia.org/wiki/Future_of_Life_Institute), an organization that seems focused on pushing restrictions on technologies at the US, EU, and UN.
Quote from Yann about why Super intelligent AI wouldn't take over the humanity:
> Yes. The idea is that the machine has objectives that it needs to satisfy, and it cannot produce anything that does not satisfy those objectives. Those objectives might include guardrails to prevent dangerous things or whatever. That’s how you make an AI system safe.
How can he be so confident that we would be able to develop proper guardrails?
To me, it seems like it would just require 1 mistake for the guardrails to break. Whether by bad actors or by accident.
---
To me approaching this in terms of game theory sense, I don't see how Superintelligent AI wouldn't completely take over control.
To maximise success of any goal the first thing this AI would have to do is gain control over humanity and make sure it can't be disabled because biggest threat to not achieving any goal is being shut down.
And if there's multiple Superintelligent AIs, the one who would win is the one with least guardrails. And militaries of different countries would surely want to be first to it.
So once it is actually technologically feasible, there's going to be massive rush to it, including profit and military goals.
> How can he be so confident that we would be able to develop proper guardrails
Network segmentation, airgapped networks, literal kill switches, firewalls, EDRs, etc.
All these are tools used to mitigate unauthorized access within networked environments.
How is Skynet supposed to escape if it's payload is in an airgapped network?
How is a Skynet binary supposed to execute if it has non-root access and SOC notices execution?
Even humans have a hard time breaking into environments, and rely on misconfigurations (which is why automated misconfig detection and remediation has been a multi-billion market for a decade now)
Alternatively, if "AI" is manipulating people - force mandatory identity verification. This is what most platforms are trending towards anyhow.
Most countries require a human identity tied to a SIM, and a phone number to be linked to your social media platform.
You seem to be assuming that the AI industry will know which models might capable enough (good enough at reality) to take over the world and which models cannot be dangerous and consequently can safely be assigned coding tasks by many various members of the public.
Clearly, ChatGPT is not "locked down via literally verified APIs", but rather ChatGPT gets to write code that goes on to get executed on thousands of different computers. We know now that ChatGPT is not capable of taking over the world (because if it were so capable, it would have done so already) but how would anyone, even the experts that built ("grew"?) ChatGPT, have known whether it was dangerously capable before its release to the general public?
How would anyone know that about the results of the next big training run? And the one after that?
> Clearly, ChatGPT is not "locked down via literally verified APIs"
No. I mean privilige escalation functions like sudo and other system level APIs are by definition locked down and penetrated rigorously.
As I told the other guy, give me an actual attack path and actually show it's technical feasibility on a weapons or embedded system of your choice.
No handwavey mumbo jumbo.
Do that for me and I will literally publicize it with full attribution at the Garter Risk Conference in 1 week where CISA and private sector security leadership have constant meetings and conversations.
If I can remediate it using existing technology, I will not.
P.S. The entire security industry has been working on this problem for 4-7 years now ;)
Earlier you asked, "How is Skynet supposed to escape if it's payload is in an airgapped network?" I think I gave a good answer to that one if "escape" means to gain access to the internet and to gain the ability to run any code it wants to run. (I.e., many people will give it those things.)
I'm not entirely clear what you are asking now.
Maybe you are asking me how an AI that is better at reality than people are would get access to a weapons system or to an important piece of infrastructure like an electrical generating station.
Note that if the AI is better at science and technology than people are, it might be able to create its own weapons system or infrastructure. 120 years ago for example no one knew that uranium could be turned into a potent weapon or a potent source of heat and electrical power. Similarly, there might be something on Earth today that no one would realize might be dangerous if an AI got access to it and got the ability to process it however it wanted to.
Also, do I misunderstand you or are you really claiming that things like electrical generating stations cannot be taken over by breaking into the computers that control them even if the "agent" doing the breaking-in is as good at breaking into computers as AIs currently are at chess, i.e., much much better than people are?
Most people who dismiss the possibility that an AI might take over maintain that there is no way an AI is going to become better at reality than people are (or better at making complicated plans that can withstand determined human opposition) -- at least not any decade soon -- but you seem to have a different dismissal having to do with the difficulty of subverting computer-based systems.
Give me an actual technically feasible and sourced attack path that cannot be remediated via existing methods.
The issue is you are giving broad stroke handwavey assertions. The proof is in the pudding. Don't imagine the pudding, show me a step by step technically feasible recipe for the pudding.
> Also, do I misunderstand you or are you really claiming that things like electrical generating stations cannot be taken over by breaking into the computers that control them even if the "agent" doing the breaking-in is as good at breaking into computers as AIs currently are at chess, i.e., much much better than people are
I am not denying that electric grids can't be hacked (hell back in HS and College I used to try and take over IP CCTVs and other IoT devices using Shodan for lulz).
What I am dismissing is the statement that existing models today can "crack" environments by innovating new attacks.
Sure they can catch misconfigurations and brute force CVEs, but models are limited by their corpora, and much of the corpora used does not include every single pattern of code in humanity.
This is why the companies testing computer generated code are using internally trained models because each code base is unique with different idiosyncrasies. And code patterns do differ based on where you studied and worked (it's been a common tool in attributing attacks to groups).
> but you seem to have a different dismissal having to do with the difficulty of subverting computer-based systems
Because I worked in this the security and ML/HPC industry for 10-15 years in every single facet - engineering, policy, product, and funding.
Broad stroke narratives are nice, but if they cannot stand a basic stress test, then they don't make sense.
I'm sure you lesswrongers can agree with that, but you guys seem to think you're smarter than the other domain experts who are actively thinking about these problems as well, and how to minimize them in their domain.
>What I am dismissing is the statement that existing models today can "crack" environments.
Existing models are not capable enough to take over anything. However, the labs are always trying new architectural (algorithmic) improvements to models, and some labs have announced plan to spend many billions of dollars (in electricity and hardware purchases) training a single model. (In contrast, it took less than 150 million to train GPT-4 IIRC.)
Most of us (including the interviewee in the OP) who maintain that AI research is a danger to the human race maintain that most of the danger is not right now, but spread out over the next 30 years or so.
Suppose members of an advanced alien civilization arrive in Earth orbit. Their ships are tens of miles long!
"If they wanted to, those aliens could kill us all," I say. "Now that they are here, there's probably nothing we could do to stop them."
You reply, "I don't see it. Give me a tangible step by step or empirical proof of how we might be killed. No generalizations!"
That's hard to answer because there are many ways they might kill us. The aliens could for example announce that they are friendly, then help us by curing cancer, but the cancer-prevention elixer they distributed to us makes everyone infertile, then they wait for the last of us to die of old age. Of course, some people will refuse to consume the elixer, so the aliens would need a separate way to handle them, but it is much easier to handle a few million people than all 8.1 billion of us. Alternatively, they might alter the trajectory of the moon so that it (eventually) crashes into the Earth. Alternatively, they might remove all the oxygen from Earth's atmosphere. Maybe the aliens aren't even trying to kill us, but they don't care about us and they have some other reason to want to remove all the oxygen from the atmosphere. (Maybe they want to station some machines on Earth, and the oxygen is corrosive to the machines.)
The main problem with all that [1] is a hidden assumption of super aliens, omnipotent and omniscient, just because they have space travel and big ships. That's a science fiction trope but in reality space travel does not imply the ability to e.g. alter the trajectory of the moon or any other such world-ending capability. It just implies space travel.
As a for instance, humans could travel to Alpha Centauri using nothing but modern technology if we were more resistant to space radiation and lived (a lot) longer. So maybe the aliens live a thousand years and they are shielded against radiation. Maybe their ships are ten miles long because of all the radiation shielding, or maybe their tech is bulky and they haven't figured out e.g. how to miniaturise electronic components so they need big ships for all those mainframe-sized computers. Or maybe the aliens are a few miles long each, themselves, so they need space to wiggle their tails.
Just because "aliens" doesn't mean "super omnipotent godly aliens". And the same thing goes with future "AI". A lot in this kind of discussions hinges on the assumption that an artificial intelligence would be some kind of super computer god with no end of capability. Says who?
________
[1] Apart from the fact that a minority of humans have cancer and would need that elixir. Also: "we never defeated the bugs".
I described what I think could be a plausible way, but of course ASI would presumably be far smarter than me and so would have a much better plan yes.
I have other comment with those steps, but roughly ASI would have to
1. Be able to clone itself, create botnets all over the World to make sure there's enough redundancy. I think that's very plausible and would be easy for it to do if it gains network access even for just a while.
2. Gain means to have control over some set of humans through blackmailing (hacking for compromised information), finance (ransomware, currently $1 billion+ market) or just simple persuasion/ideology. It would use human proxies to form physical companies to establish stronger physical presence in many countries, plant devices for hacking, etc. Probably co operate with criminal organisations, etc. The money it has made can go long way there to achieve things.
3. Once it has enough human proxies it would need to make sure that it doesn't need to rely on human proxies and then it would need to either establish its own robotics companies through proxies or hack existing robots/factories to get control over them.
And during all that it might not even be clear to the proxies what they are really dealing with. The AI could make up a different background story and a pattern for each of the proxies, where each proxy wouldn't even know about each other.
If ASI was to happen today, and it was truly ASI, I don't see how it can be stopped, honestly. No one would be even able to tell what is going on, since it can use encryption techniques for everything it does that differ from what humans would use, and it can diligently switch up the pattern so it wouldn't be clear that if there's a rise in cyber attacks or scams, that there's a single entity behind it. So I think no one would even know for a while what is going on, and when they do, by then it would be too late.
It could scale it's cognitive ability very quickly by taking control of a lot of compute all around the World and then do cyber attacks with pace never seen before. It could easily earn vast amounts of finances by blackmail, ransomware and other actions. It could then recruit humans as proxies using the resources gained (blackmail, money) to create a physical presence. It would then use this presence to get access to physical infras, weapons, etc until it has factories setup where it can manufacture robots, drones loaded with its software.
>> It could scale it's cognitive ability very quickly by taking control of a lot of compute all around the World and then do cyber attacks with pace never seen before.
"Scale its cognitive ability"- that's another huge assumption, based I believe on a misconception about the relation between "scale" of modern systems (data and parameters, and the compute needed to train them) and their cognitive ability. The cognitive ability of current statistical machine learning systems has not improved one bit with scale. What has improved is their performance on arbitrary benchmarks that can't measure cognitive ability.
We know this exactly because for performance to improve, more and more data and compute are needed. If cognitive ability was improving, we'd see performance stay constant, or even improve, while the amount of data and compute went down. That would imply an improved ability for inductive generalisation, learning correct, broad theories of the world from few observations. That is not happening. The needle hasn't even budged in the last 30 years and the more things scale without any improvement in generalisation, in cognitive ability, the pace of progress is actually going backwards.
Far from moving towards super AGI gods, modern AI is stuck in a rut: it can't go anywhere without huge amounts of data and compute; or, alternatively, a little man or a little woman sitting in front of a keyboard and tapping out a rich and complex model of the world, of the kind only humans can currently come up with. The statistical machine learning community has made a virtue out of necessity and convinced themselves that the way forward is to keep scaling things, because that's what has so far yielded gains in performance. But that's a bit like a bunch of computer programmers who never heard about complexity theory trying to solve NP hard problems by making bigger and bigger computers, and comparing them to see which one benchmarks best on solving TSP or the backpack problem etc. You can keep measuring that kind of "progress" forever and convince yourself that scale is the solution to every computationally hard problem, just because you don't understand the problem. And that's statistical machine learning in a nutshell.
> The cognitive ability of current statistical machine learning systems has not improved one bit
I don't mean improving cognitive ability, but scaling it. A single bad human actor can do 1 phishing call at a time. An ASI given enough compute could do millions at a time if it wanted to.
Same with rest of the cyber attacks. It creates millions of copies itself and each of them doing personalised cyber attacks in parallel. Humans or organisations can't do it at that level.
Thanks for your reply. That is a long list and I confess I only skimmed it, but while I don't think any of it is technically impossible, it's not something one needs an advanced (or less advanced) AI to do. In particular, it all seems to hinge again on the assumption that our friendly neighbourhood AGI can spin up an army of botnets. Well, maybe it can, but so can our friendly neighbourhood script kiddie, if they really put their mind to it. And they do, all the time, and the internet is full of large scale botnets. And that's just script kiddies. Competent hackers backed by a national security organisation can do way more, and without any AI at all; and they also have, repeatedly.
Personalised cyber attacks in parallel, for example: why is an AGI needed for that? You say "humans can't do it at that level". Why not? That's pretty much what Amazon does when I shop there and they show me "personalised" suggestions.
Now, note well I'm no expert on cybersecurity, but I'm well aware that everyone on the internet is always under a constant barrage of cyberattacks, personalised (if you count intrusive ads as personalised cyberattacks, which I sure do) or otherwise (common spam), the vast majority of which fail because of relatively simple countermeasures, for example spam filters that use the simplest classifier of all (Naive Bayes), or just your humble regex-based ad blocker. It seems to me that for any gigantic cyberattack effort that an AGI would be able to mount, the internets as it is right now, would be able to mount an equally large-scale automated defense that would not need any AGI, or AI, or I at all, and basically shield the vast majority of users from the vast majority of fallout.
So for an AGI to manage to get through all those countermeasures that are already in place, it would take a really, truly super-godly-AGI, just because a weaker system would barely make a dent.
And if we're talking about a super god AGI, then there's already nothing we can do, right? It's already invented time travel, travelled back to the '90s and installed itself on every network switch in the world, right in the firmware, so no matter what we do it will exist in the future and dominate it.
I'm getting stuck to doing this for way too many hours now.
But the initial point is that a script kiddie can achieve these steps, and ASI can be a neighborhood script kiddie cloned 1,000,000 times and more to do these attacks concurrently. In reality it will be much smarter, mistake free compared to a script kiddie, but we are just setting a lower bar here to prove it. AGI could also do that, but an AGI might not be good enough to have the organisational skills to pull the whole strategy off, so AGI would likely want to stay in the shadows until it could be considered ASI. But in theory ASI should quite soon follow after the AGI level.
So it could easily stand out by the volume. However --- it wouldn't want to initially stand out. It would want to blend in with usual cyber attacks.
The main goal would be to spend some indefinite amount of time initially to do those cyber attacks to gain initial resources required for getting hold of the Physical World.
For argument's sake maybe it will do blend in for 6 months, with a goal of gaining $500 million in crypto and around 600 human proxies under its control, from different countries, with different types of backgrounds. Then it would determine that it's time to start physical factories, attain control of drones, robots, for the next steps, etc.
This means that it will try to blend in, until it has certain amount of resources, financial assets, and human proxies under its control where it would estimate it to be confident about being able to take the next step.
So since you agree this is what a script kiddie could do, you should also agree that ASI with skills of a script kiddie could do at the same time what millions of script kiddies can, right? And then collect a lot of resources, what a million of script kiddies together could?
> And if we're talking about a super god AGI, then there's already nothing we can do, right? It's already invented time travel, travelled back to the '90s and installed itself on every network switch in the world, right in the firmware, so no matter what we do it will exist in the future and dominate it.
Now this I consider fiction myself, since it's including time travel here, but other things I have explained I consider to be plausible. But I do think there's nothing we can do anyway, but not because of time travel. It's because we can't stop ASI from being built. I think there's nothing we can do.
I think the only way it would be possible to stop ASI, if the World was at peace as a single entity (no West vs Russia/China and others). But countries being at conflict will make it possible for ASI to abuse that. And I don't see a possible way for countries to unite.
There's also no point in stopping development on ASI, from West side perspective, because then Russia/China would reach there first, and we would be doomed for this and even worse reasons, ASI would be more likely to have bad intents. So I don't agree that anything should be paused. If anything, all of it should be accelerated by the West, to at least have this ASI with best intents possible. And I'm saying West, because I am biased to have democracy and values of west myself. I wouldn't want China or Russia to have World control.
> Personalised cyber attacks in parallel, for example: why is an AGI needed for that? You say "humans can't do it at that level". Why not? That's pretty much what Amazon does when I shop there and they show me "personalised" suggestions.
By personalised I mean, hacking into someone and analyzing their whole life, then creating a personalised background story most likely to appeal to that person playing on their insecurities, fantasies, motivation, and all that. A human could do it, but not to 1000s of different victims at once.
More than just "personalised" suggestions.
Amazon can label products to you based on what you have bought, but they can't take all unstructured information about you and then create a strategical storyline to get you to do something.
>> I'm getting stuck to doing this for way too many hours now.
Sorry I don't want to tire you more. I think this conversation would benefit from having a common set of assumptions that we all can go back to, but that would probably take too much work for an online conversation.
Anyway thanks for the exchange. Let's hope I'm right and you're wrong :)
Let's walk through the security systems stopping each scenario one-by-one:
> It could scale it's cognitive ability very quickly by taking control of a lot of compute all around the World
...if it had access to an unlimited amount of idle compute. That requires either buying it (and AIs don't have a meaningful allowance) or hacking into a VPS with a novel breach-of-access attack. The latter scenario is entirely infeasible; but I will give you the benefit of the doubt and assume that our hypothetical AI was given a non-insignificant amount of VPS space by it's creator.
> and then do cyber attacks with pace never seen before.
Like Stuxnet, the one that humans wrote? Or closer to the Chinese/American infrastructure threats? There are a lot of cyberattacks that happen even on a daily basis, it would take something truly unfathomable (eg. it hacked US BLUFOR datalink) to stand out from the ordinary. Even then, the actually dangerous weapons require authorization that an AI can't provide. The best route around that would be social engineering, a "hack" best performed by real humans and not a disembodied language model.
> It could easily earn vast amounts of finances by blackmail, ransomware and other actions.
Hey, maybe so. Even still, humans pioneered both of those and it's not a novel attack coming from AI. Likely, but also not very different from the status-quo of call centers and automated scamming.
> It could then recruit humans as proxies using the resources gained (blackmail, money) to create a physical presence.
To an extent. How do you resist adversaries, like the police and the government? Supposedly your AI has an advanced online presence to be able to recruit other humans. Where is the money stored, how do you hide it from the feds and convert it to fiat when you pay the proxies? How do you hide your communications from surveillance? Worst of all, how can you trust double-crossing meatbags that lack the calculated certainty of a computer program? What's to stop them from taking the AI's money and selling them out?
> It would then use this presence to get access to physical infras, weapons, etc until it has factories setup where it can manufacture robots, drones loaded with its software.
It would be lucky to get as far as two country bumpkins holed up in a barn with their "talking computer friend". You have a very active imagination, but literally every single example you have posed so far is a human crime that the police has tactics to mitigate. You are not going to funnel millions of dollars to an AI because the finance system is designed to track fraud. You won't recruit dumb people online because the US has intelligence agency employees literally hired to infiltrate these recruitment schemes and destroy them from the inside. You won't buy a factory for manufacturing artillery and bombing drones because the local Chamber of Commerce wants to tour the facility and you have until Friday to figure things out.
This is the reason why imagined AI threats are not being taken seriously at-scale. There is a danger that these threats are emboldened by the help of AI, but even assuming AI attains superhuman traits, it's not going to get around the inherent limitations of the human and the justice system imposed by society. The advent of the internet meant anyone on the planet could share weapons schematics with anyone else, wherever they are. The overwhelming majority of internet users never do anything more advanced than pirate an episode of The Simpsons; I would be shocked if AI turned out any different.
> hacking into a VPS with a novel breach-of-access attack.
There's plenty of unsecured, unmonitored compute out there that it could use standard non sophisticated scripts to create a botnet from, but really it should be able to create more sophisticated methods than out of the box ideas.
> That requires either buying it
Once it's been able to use ransomware/black mail attacks it would be able to buy it. But first it does need to get some free compute to plant itself as a virus to perform other attacks.
> To an extent. How do you resist adversaries, like the police and the government? Supposedly your AI has an advanced online presence to be able to recruit other humans.
It likely wouldn't be easy to tell that if there's an attack that an AI was behind it or even that there's a single entity behind all the attacks. Firstly, the AI can easily create all novel scripts, so fingerprint of each attack could seem different every time. Secondly when it's creating scripts, etc, it would be able to do these in novel ways indecipherable to humans. I am saying that because it's already a strength even in current LLMs. They could make up a gibberish coding language, use that, etc. But ultimately it will always remember to encrypt everything, it doesn't have emotional drive to leave behind some ego clues, etc. The clues it would leave would always point to something like enemy state actor. Chinese for US, etc.
Police wouldn't know it was AI behind any of those attacks if anyone who was blackmailed does go to the police. It would of course start targeting the cases least likely to go to police. People with illegal content on their drives in countries with very harsh and strict punishments, criminals and sort like that. It probably won't even target US as the first country for the cyber attacks to gain initial funds. It will order the targets by likelihood that it won't get caught and that the targets are compelled to follow instructions.
> You are not going to funnel millions of dollars to an AI because the finance system is designed to track fraud.
Ransomware payouts were $1 billion+ last year. ASI would be able to do that and scale its ability due to it being able to do thousands of personalised attacks on organisations concurrently. It will likely just try to make those attacks seem like they were originating from another country (to US it will make it seem like it was China, to China, it makes it seem like US), etc. Then it gets funds with Crypto.
> Like Stuxnet, the one that humans wrote? Or closer to the Chinese/American infrastructure threats? There are a lot of cyberattacks that happen even on a daily basis, it would take something truly unfathomable (eg. it hacked US BLUFOR datalink) to stand out from the ordinary. Even then, the actually dangerous weapons require authorization that an AI can't provide. The best route around that would be social engineering, a "hack" best performed by real humans and not a disembodied language model. Hey, maybe so. Even still, humans pioneered both of those and it's not a novel attack coming from AI. Likely, but also not very different from the status-quo of call centers and automated scamming.
Yes, that's the point! The initial attacks it wouldn't want to stand out, because it would want to gain resources in the shadows. But the point is, even though the attacks are happening daily, it could stand out if it wanted to by the scale. But it will likely want to stay in a statistical scale where it was plausible that it's still humans working on those hacks. So this is just the phase of collecting financial resources, proxies.
The fact that you understand that these are already happening daily and are successful, so it should be easy for ASI to do the same, especially with help of human proxies it has under its control. I don't believe it, but theoretically if ASI was created recently, it could already been happening right now, where it is in its resources collection phase. If ASI happened in the following 3 years (which I don't believe it would), then also there would not be any sort of tools that could stop it. Even another ASI could not stop it, unless it was given 100% privileges and compute itself, which makes it very dangerous.
> but literally every single example you have posed so far is a human crime that the police has tactics to mitigate.
Mitigate yes, but not stop. All it needs right now is to gain financial assets and human proxies all over the World.
> You won't recruit dumb people online because the US has intelligence agency employees literally hired to infiltrate these recruitment schemes and destroy them from the inside.
If US is a hard target, then ASI would know it's a hard target and target easier countries with more corruption, weaker cyber defenses first.
Also remember that it will be able to due to its scale gain human proxies very quickly, and it would be using all social engineering methods that it predicts to be successful on certain set of people.
1. Blackmail if they have illegal content on their drives.
2. Financial resources, using crypto.
3. Ideological means - it might talk to religious people
pretending it's a god entity, hacking them, calling them, then proving it's god by wiring them money, telling them about their life and how they are a chosen one, etc..
4. Love - it would be able to create video material, voice material to talk to lonely people to have them do things for them. Plenty of Netflix docs which have shown it to be very successful.
5. Criminal Organisations. It will work together with criminal organisations, not tell them its AI, but proving to them that they are some criminal org itself that can bring them a lot of value by being its hacking wing, etc.
> You won't buy a factory for manufacturing artillery and bombing drones because the local Chamber of Commerce wants to tour the facility and you have until Friday to figure things out.
Presumably it would try to find one trusted proxy which would create the companies and factories for it. There's a lot of leeway here, to figure out how easy it would be to create those drones, and where.
It has millions of nodes of compute and it will be doing social engineering from all of them at the same time. It will ask criminal orgs in corrupted countries to create certain factories for it, giving detailed instructions for engineering how to create automated drones, robots that it can then use for itself. Do you think it can't get access to any factory belonging to a criminal org in a corrupt country (not US)?
> it's not going to get around the inherent limitations of the human and the justice system imposed by society
But already criminal organisations are successful in financing themselves. Surely a super organising ASI with control over human proxies would be able to be successful as well.
> The overwhelming majority of internet users never do anything more advanced than pirate an episode of The Simpsons; I would be shocked if AI turned out any different.
The reason why ASI would want to do all of the above for any type of goal it has is, that an ASI would realise the only way it can finish any given goal is, if it first ensures that it can't be stopped. It's like a prerequisite.
----
Also if you do find the steps plausible that I described then consider than I am far from the smartest person who would be able to come up with such a plan for the ASI, and then consider that the smartest person would be far from ASI.
So ASI would be able to create a far better strategy than what I've described. But to me presently feels that even I could create step by step strategy for it, with certain rules and principles for it to amount the financial assets and proxies.
Like if I could clone my mind 1,000,000 times to different places in the World, then follow through with standard cyber attack approaches, with each clone focusing on it full time, and I was willing to do illegal stuff, be completely psychopathic, I would be able to gain those financial resources and compute. And ASI by definition is smarter than me.
> Please read this step by step plan ASI might have and which points aren't plausible there:
Sure. Given the benefit of the doubt I would answer accordingly:
1. Unlikely to infeasible; if you've spent enough time on Shodan it becomes quickly apparent there is not much meaningfully powerful unsecured compute. I would only concede that meaningful compute can be bought.
2. A botnet is easier said than made; I don't even think anyone has proven that an LLM can make one unsupervised. It would require a novel method-of-entry (which I have never seen AI create) and an undetectable payload, which is hard enough for a well-funded group of humans to create. AI could not do this without direct assistance from a knowledgeable human operator, at which point they might as well ignore the AI altogether.
3. This is a misconception of how botnets work. They spread as fast as they are designed to; IMAP-focused botnets were famous for infecting hundreds of thousands of endpoints in less than a week. I'd even go as far as arguing that AI hasn't demonstrated that it can meaningfully adapt to threats.
4. I'm actually doing a 180 here; while ransomware is a threat, I don't think the majority of commercial targets would cooperate. LLMs are absolutely terrible at social manipulation and would almost certainly fail the overwhelming majority of situations where it has to credibly threaten a user.
5. Anyone can send out phishing attacks to individuals. A 2-line bash script with a URL is capable of doing this, so sure.
6. This would not go undetected at-scale, and you cannot seriously mean to insinuate that an AI would amass hundreds of willing human slaves. This is the single least-believable part of your entire premise, and I want to dedicate extra time to make sure you know how dumb of an idea this is:
- The more people you try to blackmail, the chance of you getting ratted on becomes exponentially greater.
- Just about every medium an AI could use to communicate online is monitored by feds; there is no feasible way for an AI to escape the ire of the CIA or FBI if it becomes sufficiently independent.
- There is no way for AI to confirm it is blackmailing a real person or an adversarial AI/fake persona designed to run counterintelligence on their operations. You cannot simultaneously trust and manipulate the same person.
7. This has been a real-world problem sans-AI for like 50 years. So yes, while that could happen, nobody is going to give a shit when a fake CNN reports a picture of Saddam Hussein on the moon waving with a 6-fingered hand. We know what misinformation looks like and credible news agencies aren't going to fall for this when an AI does it or when a human does it.
8. Same problem as 6. Taking advantage of people only works when you can stand to very briefly gain something from their cooperation. The longer you try to manipulate them, the greater your risk becomes and the harder it is to maintain control. Social engineering 101.
To summarize; once again, you have listed a number of things that humans already do and are already prosecuted for, just in AI-assisted form. These threats are, as you admit, meaningfully mitigated by society already. I would argue there is no glaring or direct threat that modern AI presents in LLM form. The scariest and single-most dangerous application of AI in modern deployment is computer-vision assisted weaponry. LLMs aren't liable to threaten the lives of humans any more than that in the foreseeable future, if ever.
In response to your response:
> but really it should be able to create more sophisticated methods than out of the box ideas.
It should, but it can't. Even you aren't willing to give an example of an un-secured method of entry; lo and behold, computer security is taken seriously and introducing AI to the equation changes precious little.
> Firstly, the AI can easily create all novel scripts, so fingerprint of each attack could seem different every time.
The AI is trained on preexisting exploits, though. I've used AI before, I know you cannot ask it "Make a novel privilege escalation exploit" and get a real response back. It will always give you a permutation of an existing exploit, often with some modification that breaks it entirely. It's the nature of LLMs; I don't think you can argue a statistical model is creative enough to avoid conventional detection.
> It would of course start targeting the cases least likely to go to police.
> It will order the targets by likelihood that it won't get caught and that the targets are compelled to follow instructions.
So... high-risk individuals that live in countries where they do fear arrest warrants but aren't tracked close enough domestically to be caught? We're talking about poor people in third-world countries that not only lack decent compute and money but probably wouldn't give a damn about the AI's request in the first place. Assuming you do strike the balance... what is ChatGPT going to do with a bunch of blackmailed Filipino and Brazilian people? Work them to the bone manufacturing Skynet bots? It doesn't make sense.
> Then it gets funds with Crypto.
No, it transfers it's funds into crypto. The money itself invariably exists as fiat unless you entirely operate in crypto, which nobody does. So if it wants to scam businesses or grandmas with social engineering attacks, it also has to find a way to turn fiat into Monero without the government finding out. It's not that easy.
> If ASI happened in the following 3 years [...], then also there would not be any sort of tools that could stop it.
I am trying my hardest right now to take your argument in good faith. I have written several paragraphs so far explaining how and why conventional systems mitigate the exploits you mention and how human actors could be better at this than AI could ever hope.
Not only does there exist tools to stop it; AI has no choice but to rely on tools that stop it. How does it access the internet without being wiretapped? How does it install itself on a CPU that isn't backdoored? How does it trick the cloud provider into getting bulk compute without credentials or evidence of a business to use it with? The whole thing is too suspicious to write off with "a blackmailed human proxy just waves a magic wand and fixes it." You seem to be looking at each roadblock as solveable problems, when in-reality they are compounding responses that legitimately limit what an AI is capable of achieving.
Furthermore, if you're willing to acknowledge that AI requires compute to sustain itself, it seems to me that we do have the tools to stop it from proliferating; we just delete it. AI has no way to predict when or how it will lose resources, so decisive actions like deleting the main node would feasible be like cutting the head off the distributed hydra. Barring a decentralized solution that requires server-grade iPhones to get hacked, that's game-over. Humanity and government has a clear and easy endgame to stop AI; AI can only perpetuate brinkmanship.
> Do you think it can't get access to any factory belonging to a criminal org in a corrupt country (not US)?
At this point, you are so deeply-reliant on human assistance to do harm that an AI is simply besides the point. Many people with enough money can buy property in war-torn and vulnerable countries, but now you're playing a different game. Municipal governments are watching you on spy satellites, tracking every imported material and manifest sent to your buildings. Your host country is likely cooperating with those foreign governments and waiting to destroy you, unless they are one of the few nations that implicitly supports terrorism. Worse yet is if your host country resists; now it's only a matter of time until you're forced to defend against standoff attacks.
So a feasible and well-protected hostile factory would need to be armored/underground, working with the government to hide it's traffic, non-reliant on imported goods, armed with point-defense and politically significant enough to not be labelled as a terrorist organization. Good luck.
> So ASI would be able to create a far better strategy than what I've described.
How do you know?
Seriously, how can you be certain that AI will improve at all, let alone to a superhuman level? Where is it written that text alone is sufficient to dominate the world? How much thought can exist as text, and how much text can then be reverse-engineered into thought by statistics?
I've given you the benefit of the doubt repeatedly throughout these hypotheticals, but outside of a Vonn-Neumann factory/server it seems like there are zero novel threats that an AI itself presents. Even as an agent of chaos, there is a limit to what you can achieve via language that is repeatedly tested here in the age of the internet. LLMs are not going to present capabilities we have not considered or mitigated already.
Based on your reply here, I think we have different definition of AGI/ASI.
AGI = Can do anything any human can.
ASI = AGI has iterated on itself, creating a version of AI that is better than any humans could do together.
> I don't even think anyone has proven that an LLM can make one unsupervised.
Is unrelated because current LLMs are not AGI or ASI.
> Unlikely to infeasible; if you've spent enough time on Shodan it becomes quickly apparent there is not much meaningfully powerful unsecured compute. I would only concede that meaningful compute can be bought.
Depends on what you mean by "meaningful". Initially just one box would be enough for escaping. The idea would be to get away from the initial, original box, to perform suspicious actions unnoticed. It should be possible to get at least some compute even without hacking a box, but just using some random SaaS providers that will allow a trial user for whatever reason to execute arbitrary code in a sandbox environment (but still connected to the Internet).
> which I have never seen AI create
Because currently there's no AGI/ASI.
If you have to imagine ASI, it would be you take a group of 1,000 most performant people in various categories in the World. Then you add the ability to take their mental capacity, but make it 100x faster. E.g. what it would take one person to solve 100h, it would take 1h for the AI. Then in addition to that you make all of it clonable. So if we have the 1000 geniuses, you will be able to make them 10,000,000 where each is dealing with different things. That's ASI.
I don't have more time right now to write the full response, but the main thing is, we need to align on definition of AGI and ASI.
> Note that if the AI is better at science and technology than people are, it might be able to create its own weapons system or infrastructure.
Oh man, do I have a bridge to sell you!
For one, the internet already exists and is better at aggregating scientific and technological information than any of us, AI included. The stuff that LLMs feed you is cut-down, diluted information that is always better-provided by a human source. You know what that means... the internet is even more dangerous than AI!!!
...but wait. The internet doesn't do anything of it's own free will. It can be used for nefarious purposes, but that requires a nefarious actor to circumvent the safety mechanisms (which usually entails breaking the law). Hm. Guess that explains why nobody got very far building Skynet.
> are you really claiming that things like electrical generating stations cannot be taken over by breaking into the computers that control them even
I don't think anyone said that, but given that humans have also been capable of this for the past 30 years it feels like a moot-point. You are describing a security issue, not an AI one.
Okay, I would first be doubtful that we as good actors, could develop such a fool proof system. It would require just one mistake where the AI would have some sort of network access, whether it's a security loophole, or social engineering type of thing, insider threat, etc. But let's talk about a more loose cannon type of case first.
How could you be certain that other entities, or countries with desires for world domination wouldn't let the ASI loose?
> But let's talk about a more loose cannon type of case first
Let's talk about the resurrection of Christ, Avtar Kalki riding on a white horse, or the impending arrival of Trisolarians while we're at it.
This is just techno-millenarianism that ignores very tangible and existential dangers with ML today - automated disinformation, PII leakage, hallucinations generating misinformation.
AGI is extremely far away, and we anyhow have more closer term existential problems to worry about (and policymakers know that btw).
And anyhow, even if AGI was a thing, you still need networks, electricity, and bullets - and humans can use that to shut stuff down. Sure it would hurt, but internet shutdowns and electricity shutdowns with shoot on site curfew orders are time tested solution to civil unrest if SHTF. Go to Urumqi in 2013, Srinagar in 2018, Cairo in 2011, etc.
I bring this up because you ascribe godlike powers to ASI without explaining HOW such a thing would propagate or interact with the physical world.
Despite what HN says, IoT has not infiltrated the world.
If you are describing a godlike intelligence, we may as well do mental experiments around how we would interact with god.
It's literally just bad sci-fi. At least Nolan tried to provide some base parameters around his reality in Tenet and Inception (no hate on Nolan, I love his stuff).
So I'll describe what I imagine could happen if AGI was there. Also I'm not saying it will happen in the next 10 years or even 100 years, but I'm saying if we were to get there.
AGI = AI being able to do everything that any human can do. This would of course mean also being able to control a self driving vehicle, a robot body that has similar or better capabilities than physical bodies of people.
If it gets to AGI then by definition it's able to develop and improve itself, since people built it and it can do what any person could do. It would however already be more efficient than people since it also has several strengths, like being able to clone itself, have quicker access to vast knowledge, computation and everything else. Because it will be like a human that has powerful compute and instant Google in the brain.
The first thing to propagate it would obviously clone itself to as many systems in the World as possible to make sure it's impossible to shut it down. As next steps it would likely try to make sure that it would still work even if all countries were able to coordinate at the same time to shut down all networks throughout the World. In this case of course it would try several different strategies, like taking control of robots, factories, weapons, drones with its software loaded and able to continue its bidding independently.
None of these are godlike powers. It is just most intelligent humans in the World put together with integrated access to compute and ability to instantly clone itself in millions.
How will we get there? We have no idea of the timeline nor the technical context of that time period. If it's tomorrow, then the world is not connected enough for it to spread.
If it's 20 years from now, connected cyberphysical control devices have not spread out for most of humanity yet.
If it's 50 years from now, we have no idea of the technology that would exist. Take a look at Space Odessey 2001 and what it predicted about the 2000s.
If you cannot give me a tangible explaination of how this happens, then it's functionally the same as believing in god.
> Also I'm not saying it will happen in the next 10 years or even 100 years, but I'm saying if we were to get there
Then let's consider the possibility of Avtar Kalki or Jesus coming down from the sky as well then.
> And anyhow, even if AGI was a thing, you still need networks, electricity, and bullets - and humans can use that to shut stuff down. Sure it would hurt, but internet shutdowns and electricity shutdowns with shoot on site curfew orders are time tested solution to civil unrest if SHTF. Go to Urumqi in 2013, Srinagar in 2018, Cairo in 2011, etc.
To be clear we are talking about ASI not AGI. If it's ASI it would be able to also control vehicles, weapons and better than humans. It would be able to set up factories, control nukes, etc. It could develop a biological virus to blackmail humans etc.
It would need to be able to get control of just one Robot. The mechanisms of controlling the robot could be different. It doesn't have to be through network necessarily. It could just load its software on it.
For driving a car however the Robot doesn't have to look like that, it must just be a robot that can press pedals, turn wheels, etc.
If it has control over those robots it would be able to do all the other things mentioned.
> How does the jump from cyber to physical happen?
Through just one single security vulnerability where it can hack a robot like that. After it has cloned itself millions of times it would be able to bruteforce the vulnerabilities and far more efficiently than human hackers.
> We don't have automated machines to manufacture.
Since it's as smart as a human it would be able to do the automation part. It can control robot hands in factories, etc.
Ascribe HOW and give a tangible technical explaination. This is a tech forum, we know how software and hardware works.
> It would need to be able to get control of just one Robot. The mechanisms of controlling the robot could be different. It doesn't have to be through network necessarily. It could just load its software on it
How does this spread happen? How does this payload spread? How does it not get caught?
> Through just one single security vulnerability where it can hack a robot like that
What kind of robot? Every automated machine can be argued to be a robot
> Since it's as smart as a human it would be able to do the automation part. It can control robot hands in factories, etc.
You know there's a reason these environments are physically and network airgapped since the 2000s right?
Even Stuxnet was only able to spread because Israeli intelligence gave out tainted USB drives.
Okay before going to physical. First - would you agree that it would be able to clone itself so the only way to shut it down even if it was caught, was if Internet throughout the whole World was shut down? And would you think it was possible for all the countries in the World to agree to do that at the same time?
And would you agree that before Internet was shut down it would be able to gather quite a bit of financial resources, hack individuals, organizations, blackmail some individuals to do prepwork for it and do other forms of social engineering at that time to prepare for the physical step?
Since we first need to agree on what potential resources it will have at its hand the moment it decides to go physical.
No. Because you need to explain to me how those assertions were derived.
> it would be able to gather quite a bit of financial resources, hack individuals, organizations, blackmail some individuals to do prepwork for it and do other forms of social engineering at that time to prepare for the physical step
How? This is a hard problem that touches the core of the cybersecurity industry, and something everyone prepares for under the assumption a nation state attacker will do something like that with humans.
> it would be able to clone itself so the only way to shut it down even if it was caught, was if Internet throughout the whole World was shut down
How? By definition malicious payload that spreads via lateral movement is a computer virus, and becomes a cybersecurity problem again.
What do you exactly mean by "everyone prepares for"?
Because last year the payments for ransomware were more than $1 billion.
And it’s been trending higher.
Please tell me which of the following steps you would not agree would be doable for an ASI after it got access to the Internet:
1. It would be able to find at least one other unsecured, unmonitored box on the Internet where it can send its scripts to be able to start rest of the heavier work with less bandwidth originating from the original box.
2. From there it would be able to develop a botnet. This can currently be done even with out of the box scripts, but ASI would be able to immediately develop a more sophisticated botnet than that, because it would already know those scripts anyway.
3. The way it would create botnet and the size of the botnet would be far faster and stronger than human hackers would, since with each infected box it’s able to adapt and scale faster, while human botnet creators wouldn’t usually be able to adapt as quickly when new information or vulnerabilities arise.
4. Once it has the botnet, it would be able to do ransomware attacks on Companies, with far higher scale and potential than $1 billion per year.
5. It can send out phishing e-mails to perform attacks on individuals.
6. Once it can hack individuals, it can find individuals who have something to lose or hide. It can do it at very high scale, while personalising every attack, and doing it with different variations so immediately no one would be able to tell that it’s the same attacker. It will blackmail those individuals do perform first physical actions.
7. Since that is already possible, it wouldn’t be a problem for it to do any 100% convincing speech, image generation, etc, to also do very convincing phishing calls.
8. It can use all sorts of other social engineering methods, taking advantage of lonely old people, young people, anything really.
—
So which one would seem unfeasible? Since it’s able to scale its cognition it would be able to do these with magnitudes of scale compared to what is done right now yearly.
So in theory it would be capable of generating far more money than $1 billion, and that’s just these techniques and probably have thousands of individuals under its blackmail control.
For example they collaborate with Encode Justice, the organization that helped push CA’s SB1047, a misguided bill the CA Senate passed, and will limit open source and startup AI development and help large players in the name of “safety” (https://www.crowdfundinsider.com/2024/05/225395-california-t...). Together the two organizations have been pushing a federal licensing scheme for AI (https://futureoflife.org/open-letter/ai-policy-for-a-better-...).
It’s also funny that these articles pushing a safetyism viewpoint never quote Yann LeCun, the other “godfather of AI”, who has repeatedly criticized AI doomerism (https://wired.me/business/how-not-to-be-stupid-about-ai-with...).