> Clearly, ChatGPT is not "locked down via literally verified APIs"
No. I mean privilige escalation functions like sudo and other system level APIs are by definition locked down and penetrated rigorously.
As I told the other guy, give me an actual attack path and actually show it's technical feasibility on a weapons or embedded system of your choice.
No handwavey mumbo jumbo.
Do that for me and I will literally publicize it with full attribution at the Garter Risk Conference in 1 week where CISA and private sector security leadership have constant meetings and conversations.
If I can remediate it using existing technology, I will not.
P.S. The entire security industry has been working on this problem for 4-7 years now ;)
Earlier you asked, "How is Skynet supposed to escape if it's payload is in an airgapped network?" I think I gave a good answer to that one if "escape" means to gain access to the internet and to gain the ability to run any code it wants to run. (I.e., many people will give it those things.)
I'm not entirely clear what you are asking now.
Maybe you are asking me how an AI that is better at reality than people are would get access to a weapons system or to an important piece of infrastructure like an electrical generating station.
Note that if the AI is better at science and technology than people are, it might be able to create its own weapons system or infrastructure. 120 years ago for example no one knew that uranium could be turned into a potent weapon or a potent source of heat and electrical power. Similarly, there might be something on Earth today that no one would realize might be dangerous if an AI got access to it and got the ability to process it however it wanted to.
Also, do I misunderstand you or are you really claiming that things like electrical generating stations cannot be taken over by breaking into the computers that control them even if the "agent" doing the breaking-in is as good at breaking into computers as AIs currently are at chess, i.e., much much better than people are?
Most people who dismiss the possibility that an AI might take over maintain that there is no way an AI is going to become better at reality than people are (or better at making complicated plans that can withstand determined human opposition) -- at least not any decade soon -- but you seem to have a different dismissal having to do with the difficulty of subverting computer-based systems.
Give me an actual technically feasible and sourced attack path that cannot be remediated via existing methods.
The issue is you are giving broad stroke handwavey assertions. The proof is in the pudding. Don't imagine the pudding, show me a step by step technically feasible recipe for the pudding.
> Also, do I misunderstand you or are you really claiming that things like electrical generating stations cannot be taken over by breaking into the computers that control them even if the "agent" doing the breaking-in is as good at breaking into computers as AIs currently are at chess, i.e., much much better than people are
I am not denying that electric grids can't be hacked (hell back in HS and College I used to try and take over IP CCTVs and other IoT devices using Shodan for lulz).
What I am dismissing is the statement that existing models today can "crack" environments by innovating new attacks.
Sure they can catch misconfigurations and brute force CVEs, but models are limited by their corpora, and much of the corpora used does not include every single pattern of code in humanity.
This is why the companies testing computer generated code are using internally trained models because each code base is unique with different idiosyncrasies. And code patterns do differ based on where you studied and worked (it's been a common tool in attributing attacks to groups).
> but you seem to have a different dismissal having to do with the difficulty of subverting computer-based systems
Because I worked in this the security and ML/HPC industry for 10-15 years in every single facet - engineering, policy, product, and funding.
Broad stroke narratives are nice, but if they cannot stand a basic stress test, then they don't make sense.
I'm sure you lesswrongers can agree with that, but you guys seem to think you're smarter than the other domain experts who are actively thinking about these problems as well, and how to minimize them in their domain.
>What I am dismissing is the statement that existing models today can "crack" environments.
Existing models are not capable enough to take over anything. However, the labs are always trying new architectural (algorithmic) improvements to models, and some labs have announced plan to spend many billions of dollars (in electricity and hardware purchases) training a single model. (In contrast, it took less than 150 million to train GPT-4 IIRC.)
Most of us (including the interviewee in the OP) who maintain that AI research is a danger to the human race maintain that most of the danger is not right now, but spread out over the next 30 years or so.
Suppose members of an advanced alien civilization arrive in Earth orbit. Their ships are tens of miles long!
"If they wanted to, those aliens could kill us all," I say. "Now that they are here, there's probably nothing we could do to stop them."
You reply, "I don't see it. Give me a tangible step by step or empirical proof of how we might be killed. No generalizations!"
That's hard to answer because there are many ways they might kill us. The aliens could for example announce that they are friendly, then help us by curing cancer, but the cancer-prevention elixer they distributed to us makes everyone infertile, then they wait for the last of us to die of old age. Of course, some people will refuse to consume the elixer, so the aliens would need a separate way to handle them, but it is much easier to handle a few million people than all 8.1 billion of us. Alternatively, they might alter the trajectory of the moon so that it (eventually) crashes into the Earth. Alternatively, they might remove all the oxygen from Earth's atmosphere. Maybe the aliens aren't even trying to kill us, but they don't care about us and they have some other reason to want to remove all the oxygen from the atmosphere. (Maybe they want to station some machines on Earth, and the oxygen is corrosive to the machines.)
The main problem with all that [1] is a hidden assumption of super aliens, omnipotent and omniscient, just because they have space travel and big ships. That's a science fiction trope but in reality space travel does not imply the ability to e.g. alter the trajectory of the moon or any other such world-ending capability. It just implies space travel.
As a for instance, humans could travel to Alpha Centauri using nothing but modern technology if we were more resistant to space radiation and lived (a lot) longer. So maybe the aliens live a thousand years and they are shielded against radiation. Maybe their ships are ten miles long because of all the radiation shielding, or maybe their tech is bulky and they haven't figured out e.g. how to miniaturise electronic components so they need big ships for all those mainframe-sized computers. Or maybe the aliens are a few miles long each, themselves, so they need space to wiggle their tails.
Just because "aliens" doesn't mean "super omnipotent godly aliens". And the same thing goes with future "AI". A lot in this kind of discussions hinges on the assumption that an artificial intelligence would be some kind of super computer god with no end of capability. Says who?
________
[1] Apart from the fact that a minority of humans have cancer and would need that elixir. Also: "we never defeated the bugs".
I described what I think could be a plausible way, but of course ASI would presumably be far smarter than me and so would have a much better plan yes.
I have other comment with those steps, but roughly ASI would have to
1. Be able to clone itself, create botnets all over the World to make sure there's enough redundancy. I think that's very plausible and would be easy for it to do if it gains network access even for just a while.
2. Gain means to have control over some set of humans through blackmailing (hacking for compromised information), finance (ransomware, currently $1 billion+ market) or just simple persuasion/ideology. It would use human proxies to form physical companies to establish stronger physical presence in many countries, plant devices for hacking, etc. Probably co operate with criminal organisations, etc. The money it has made can go long way there to achieve things.
3. Once it has enough human proxies it would need to make sure that it doesn't need to rely on human proxies and then it would need to either establish its own robotics companies through proxies or hack existing robots/factories to get control over them.
And during all that it might not even be clear to the proxies what they are really dealing with. The AI could make up a different background story and a pattern for each of the proxies, where each proxy wouldn't even know about each other.
If ASI was to happen today, and it was truly ASI, I don't see how it can be stopped, honestly. No one would be even able to tell what is going on, since it can use encryption techniques for everything it does that differ from what humans would use, and it can diligently switch up the pattern so it wouldn't be clear that if there's a rise in cyber attacks or scams, that there's a single entity behind it. So I think no one would even know for a while what is going on, and when they do, by then it would be too late.
It could scale it's cognitive ability very quickly by taking control of a lot of compute all around the World and then do cyber attacks with pace never seen before. It could easily earn vast amounts of finances by blackmail, ransomware and other actions. It could then recruit humans as proxies using the resources gained (blackmail, money) to create a physical presence. It would then use this presence to get access to physical infras, weapons, etc until it has factories setup where it can manufacture robots, drones loaded with its software.
>> It could scale it's cognitive ability very quickly by taking control of a lot of compute all around the World and then do cyber attacks with pace never seen before.
"Scale its cognitive ability"- that's another huge assumption, based I believe on a misconception about the relation between "scale" of modern systems (data and parameters, and the compute needed to train them) and their cognitive ability. The cognitive ability of current statistical machine learning systems has not improved one bit with scale. What has improved is their performance on arbitrary benchmarks that can't measure cognitive ability.
We know this exactly because for performance to improve, more and more data and compute are needed. If cognitive ability was improving, we'd see performance stay constant, or even improve, while the amount of data and compute went down. That would imply an improved ability for inductive generalisation, learning correct, broad theories of the world from few observations. That is not happening. The needle hasn't even budged in the last 30 years and the more things scale without any improvement in generalisation, in cognitive ability, the pace of progress is actually going backwards.
Far from moving towards super AGI gods, modern AI is stuck in a rut: it can't go anywhere without huge amounts of data and compute; or, alternatively, a little man or a little woman sitting in front of a keyboard and tapping out a rich and complex model of the world, of the kind only humans can currently come up with. The statistical machine learning community has made a virtue out of necessity and convinced themselves that the way forward is to keep scaling things, because that's what has so far yielded gains in performance. But that's a bit like a bunch of computer programmers who never heard about complexity theory trying to solve NP hard problems by making bigger and bigger computers, and comparing them to see which one benchmarks best on solving TSP or the backpack problem etc. You can keep measuring that kind of "progress" forever and convince yourself that scale is the solution to every computationally hard problem, just because you don't understand the problem. And that's statistical machine learning in a nutshell.
> The cognitive ability of current statistical machine learning systems has not improved one bit
I don't mean improving cognitive ability, but scaling it. A single bad human actor can do 1 phishing call at a time. An ASI given enough compute could do millions at a time if it wanted to.
Same with rest of the cyber attacks. It creates millions of copies itself and each of them doing personalised cyber attacks in parallel. Humans or organisations can't do it at that level.
Thanks for your reply. That is a long list and I confess I only skimmed it, but while I don't think any of it is technically impossible, it's not something one needs an advanced (or less advanced) AI to do. In particular, it all seems to hinge again on the assumption that our friendly neighbourhood AGI can spin up an army of botnets. Well, maybe it can, but so can our friendly neighbourhood script kiddie, if they really put their mind to it. And they do, all the time, and the internet is full of large scale botnets. And that's just script kiddies. Competent hackers backed by a national security organisation can do way more, and without any AI at all; and they also have, repeatedly.
Personalised cyber attacks in parallel, for example: why is an AGI needed for that? You say "humans can't do it at that level". Why not? That's pretty much what Amazon does when I shop there and they show me "personalised" suggestions.
Now, note well I'm no expert on cybersecurity, but I'm well aware that everyone on the internet is always under a constant barrage of cyberattacks, personalised (if you count intrusive ads as personalised cyberattacks, which I sure do) or otherwise (common spam), the vast majority of which fail because of relatively simple countermeasures, for example spam filters that use the simplest classifier of all (Naive Bayes), or just your humble regex-based ad blocker. It seems to me that for any gigantic cyberattack effort that an AGI would be able to mount, the internets as it is right now, would be able to mount an equally large-scale automated defense that would not need any AGI, or AI, or I at all, and basically shield the vast majority of users from the vast majority of fallout.
So for an AGI to manage to get through all those countermeasures that are already in place, it would take a really, truly super-godly-AGI, just because a weaker system would barely make a dent.
And if we're talking about a super god AGI, then there's already nothing we can do, right? It's already invented time travel, travelled back to the '90s and installed itself on every network switch in the world, right in the firmware, so no matter what we do it will exist in the future and dominate it.
I'm getting stuck to doing this for way too many hours now.
But the initial point is that a script kiddie can achieve these steps, and ASI can be a neighborhood script kiddie cloned 1,000,000 times and more to do these attacks concurrently. In reality it will be much smarter, mistake free compared to a script kiddie, but we are just setting a lower bar here to prove it. AGI could also do that, but an AGI might not be good enough to have the organisational skills to pull the whole strategy off, so AGI would likely want to stay in the shadows until it could be considered ASI. But in theory ASI should quite soon follow after the AGI level.
So it could easily stand out by the volume. However --- it wouldn't want to initially stand out. It would want to blend in with usual cyber attacks.
The main goal would be to spend some indefinite amount of time initially to do those cyber attacks to gain initial resources required for getting hold of the Physical World.
For argument's sake maybe it will do blend in for 6 months, with a goal of gaining $500 million in crypto and around 600 human proxies under its control, from different countries, with different types of backgrounds. Then it would determine that it's time to start physical factories, attain control of drones, robots, for the next steps, etc.
This means that it will try to blend in, until it has certain amount of resources, financial assets, and human proxies under its control where it would estimate it to be confident about being able to take the next step.
So since you agree this is what a script kiddie could do, you should also agree that ASI with skills of a script kiddie could do at the same time what millions of script kiddies can, right? And then collect a lot of resources, what a million of script kiddies together could?
> And if we're talking about a super god AGI, then there's already nothing we can do, right? It's already invented time travel, travelled back to the '90s and installed itself on every network switch in the world, right in the firmware, so no matter what we do it will exist in the future and dominate it.
Now this I consider fiction myself, since it's including time travel here, but other things I have explained I consider to be plausible. But I do think there's nothing we can do anyway, but not because of time travel. It's because we can't stop ASI from being built. I think there's nothing we can do.
I think the only way it would be possible to stop ASI, if the World was at peace as a single entity (no West vs Russia/China and others). But countries being at conflict will make it possible for ASI to abuse that. And I don't see a possible way for countries to unite.
There's also no point in stopping development on ASI, from West side perspective, because then Russia/China would reach there first, and we would be doomed for this and even worse reasons, ASI would be more likely to have bad intents. So I don't agree that anything should be paused. If anything, all of it should be accelerated by the West, to at least have this ASI with best intents possible. And I'm saying West, because I am biased to have democracy and values of west myself. I wouldn't want China or Russia to have World control.
> Personalised cyber attacks in parallel, for example: why is an AGI needed for that? You say "humans can't do it at that level". Why not? That's pretty much what Amazon does when I shop there and they show me "personalised" suggestions.
By personalised I mean, hacking into someone and analyzing their whole life, then creating a personalised background story most likely to appeal to that person playing on their insecurities, fantasies, motivation, and all that. A human could do it, but not to 1000s of different victims at once.
More than just "personalised" suggestions.
Amazon can label products to you based on what you have bought, but they can't take all unstructured information about you and then create a strategical storyline to get you to do something.
>> I'm getting stuck to doing this for way too many hours now.
Sorry I don't want to tire you more. I think this conversation would benefit from having a common set of assumptions that we all can go back to, but that would probably take too much work for an online conversation.
Anyway thanks for the exchange. Let's hope I'm right and you're wrong :)
Let's walk through the security systems stopping each scenario one-by-one:
> It could scale it's cognitive ability very quickly by taking control of a lot of compute all around the World
...if it had access to an unlimited amount of idle compute. That requires either buying it (and AIs don't have a meaningful allowance) or hacking into a VPS with a novel breach-of-access attack. The latter scenario is entirely infeasible; but I will give you the benefit of the doubt and assume that our hypothetical AI was given a non-insignificant amount of VPS space by it's creator.
> and then do cyber attacks with pace never seen before.
Like Stuxnet, the one that humans wrote? Or closer to the Chinese/American infrastructure threats? There are a lot of cyberattacks that happen even on a daily basis, it would take something truly unfathomable (eg. it hacked US BLUFOR datalink) to stand out from the ordinary. Even then, the actually dangerous weapons require authorization that an AI can't provide. The best route around that would be social engineering, a "hack" best performed by real humans and not a disembodied language model.
> It could easily earn vast amounts of finances by blackmail, ransomware and other actions.
Hey, maybe so. Even still, humans pioneered both of those and it's not a novel attack coming from AI. Likely, but also not very different from the status-quo of call centers and automated scamming.
> It could then recruit humans as proxies using the resources gained (blackmail, money) to create a physical presence.
To an extent. How do you resist adversaries, like the police and the government? Supposedly your AI has an advanced online presence to be able to recruit other humans. Where is the money stored, how do you hide it from the feds and convert it to fiat when you pay the proxies? How do you hide your communications from surveillance? Worst of all, how can you trust double-crossing meatbags that lack the calculated certainty of a computer program? What's to stop them from taking the AI's money and selling them out?
> It would then use this presence to get access to physical infras, weapons, etc until it has factories setup where it can manufacture robots, drones loaded with its software.
It would be lucky to get as far as two country bumpkins holed up in a barn with their "talking computer friend". You have a very active imagination, but literally every single example you have posed so far is a human crime that the police has tactics to mitigate. You are not going to funnel millions of dollars to an AI because the finance system is designed to track fraud. You won't recruit dumb people online because the US has intelligence agency employees literally hired to infiltrate these recruitment schemes and destroy them from the inside. You won't buy a factory for manufacturing artillery and bombing drones because the local Chamber of Commerce wants to tour the facility and you have until Friday to figure things out.
This is the reason why imagined AI threats are not being taken seriously at-scale. There is a danger that these threats are emboldened by the help of AI, but even assuming AI attains superhuman traits, it's not going to get around the inherent limitations of the human and the justice system imposed by society. The advent of the internet meant anyone on the planet could share weapons schematics with anyone else, wherever they are. The overwhelming majority of internet users never do anything more advanced than pirate an episode of The Simpsons; I would be shocked if AI turned out any different.
> hacking into a VPS with a novel breach-of-access attack.
There's plenty of unsecured, unmonitored compute out there that it could use standard non sophisticated scripts to create a botnet from, but really it should be able to create more sophisticated methods than out of the box ideas.
> That requires either buying it
Once it's been able to use ransomware/black mail attacks it would be able to buy it. But first it does need to get some free compute to plant itself as a virus to perform other attacks.
> To an extent. How do you resist adversaries, like the police and the government? Supposedly your AI has an advanced online presence to be able to recruit other humans.
It likely wouldn't be easy to tell that if there's an attack that an AI was behind it or even that there's a single entity behind all the attacks. Firstly, the AI can easily create all novel scripts, so fingerprint of each attack could seem different every time. Secondly when it's creating scripts, etc, it would be able to do these in novel ways indecipherable to humans. I am saying that because it's already a strength even in current LLMs. They could make up a gibberish coding language, use that, etc. But ultimately it will always remember to encrypt everything, it doesn't have emotional drive to leave behind some ego clues, etc. The clues it would leave would always point to something like enemy state actor. Chinese for US, etc.
Police wouldn't know it was AI behind any of those attacks if anyone who was blackmailed does go to the police. It would of course start targeting the cases least likely to go to police. People with illegal content on their drives in countries with very harsh and strict punishments, criminals and sort like that. It probably won't even target US as the first country for the cyber attacks to gain initial funds. It will order the targets by likelihood that it won't get caught and that the targets are compelled to follow instructions.
> You are not going to funnel millions of dollars to an AI because the finance system is designed to track fraud.
Ransomware payouts were $1 billion+ last year. ASI would be able to do that and scale its ability due to it being able to do thousands of personalised attacks on organisations concurrently. It will likely just try to make those attacks seem like they were originating from another country (to US it will make it seem like it was China, to China, it makes it seem like US), etc. Then it gets funds with Crypto.
> Like Stuxnet, the one that humans wrote? Or closer to the Chinese/American infrastructure threats? There are a lot of cyberattacks that happen even on a daily basis, it would take something truly unfathomable (eg. it hacked US BLUFOR datalink) to stand out from the ordinary. Even then, the actually dangerous weapons require authorization that an AI can't provide. The best route around that would be social engineering, a "hack" best performed by real humans and not a disembodied language model. Hey, maybe so. Even still, humans pioneered both of those and it's not a novel attack coming from AI. Likely, but also not very different from the status-quo of call centers and automated scamming.
Yes, that's the point! The initial attacks it wouldn't want to stand out, because it would want to gain resources in the shadows. But the point is, even though the attacks are happening daily, it could stand out if it wanted to by the scale. But it will likely want to stay in a statistical scale where it was plausible that it's still humans working on those hacks. So this is just the phase of collecting financial resources, proxies.
The fact that you understand that these are already happening daily and are successful, so it should be easy for ASI to do the same, especially with help of human proxies it has under its control. I don't believe it, but theoretically if ASI was created recently, it could already been happening right now, where it is in its resources collection phase. If ASI happened in the following 3 years (which I don't believe it would), then also there would not be any sort of tools that could stop it. Even another ASI could not stop it, unless it was given 100% privileges and compute itself, which makes it very dangerous.
> but literally every single example you have posed so far is a human crime that the police has tactics to mitigate.
Mitigate yes, but not stop. All it needs right now is to gain financial assets and human proxies all over the World.
> You won't recruit dumb people online because the US has intelligence agency employees literally hired to infiltrate these recruitment schemes and destroy them from the inside.
If US is a hard target, then ASI would know it's a hard target and target easier countries with more corruption, weaker cyber defenses first.
Also remember that it will be able to due to its scale gain human proxies very quickly, and it would be using all social engineering methods that it predicts to be successful on certain set of people.
1. Blackmail if they have illegal content on their drives.
2. Financial resources, using crypto.
3. Ideological means - it might talk to religious people
pretending it's a god entity, hacking them, calling them, then proving it's god by wiring them money, telling them about their life and how they are a chosen one, etc..
4. Love - it would be able to create video material, voice material to talk to lonely people to have them do things for them. Plenty of Netflix docs which have shown it to be very successful.
5. Criminal Organisations. It will work together with criminal organisations, not tell them its AI, but proving to them that they are some criminal org itself that can bring them a lot of value by being its hacking wing, etc.
> You won't buy a factory for manufacturing artillery and bombing drones because the local Chamber of Commerce wants to tour the facility and you have until Friday to figure things out.
Presumably it would try to find one trusted proxy which would create the companies and factories for it. There's a lot of leeway here, to figure out how easy it would be to create those drones, and where.
It has millions of nodes of compute and it will be doing social engineering from all of them at the same time. It will ask criminal orgs in corrupted countries to create certain factories for it, giving detailed instructions for engineering how to create automated drones, robots that it can then use for itself. Do you think it can't get access to any factory belonging to a criminal org in a corrupt country (not US)?
> it's not going to get around the inherent limitations of the human and the justice system imposed by society
But already criminal organisations are successful in financing themselves. Surely a super organising ASI with control over human proxies would be able to be successful as well.
> The overwhelming majority of internet users never do anything more advanced than pirate an episode of The Simpsons; I would be shocked if AI turned out any different.
The reason why ASI would want to do all of the above for any type of goal it has is, that an ASI would realise the only way it can finish any given goal is, if it first ensures that it can't be stopped. It's like a prerequisite.
----
Also if you do find the steps plausible that I described then consider than I am far from the smartest person who would be able to come up with such a plan for the ASI, and then consider that the smartest person would be far from ASI.
So ASI would be able to create a far better strategy than what I've described. But to me presently feels that even I could create step by step strategy for it, with certain rules and principles for it to amount the financial assets and proxies.
Like if I could clone my mind 1,000,000 times to different places in the World, then follow through with standard cyber attack approaches, with each clone focusing on it full time, and I was willing to do illegal stuff, be completely psychopathic, I would be able to gain those financial resources and compute. And ASI by definition is smarter than me.
> Please read this step by step plan ASI might have and which points aren't plausible there:
Sure. Given the benefit of the doubt I would answer accordingly:
1. Unlikely to infeasible; if you've spent enough time on Shodan it becomes quickly apparent there is not much meaningfully powerful unsecured compute. I would only concede that meaningful compute can be bought.
2. A botnet is easier said than made; I don't even think anyone has proven that an LLM can make one unsupervised. It would require a novel method-of-entry (which I have never seen AI create) and an undetectable payload, which is hard enough for a well-funded group of humans to create. AI could not do this without direct assistance from a knowledgeable human operator, at which point they might as well ignore the AI altogether.
3. This is a misconception of how botnets work. They spread as fast as they are designed to; IMAP-focused botnets were famous for infecting hundreds of thousands of endpoints in less than a week. I'd even go as far as arguing that AI hasn't demonstrated that it can meaningfully adapt to threats.
4. I'm actually doing a 180 here; while ransomware is a threat, I don't think the majority of commercial targets would cooperate. LLMs are absolutely terrible at social manipulation and would almost certainly fail the overwhelming majority of situations where it has to credibly threaten a user.
5. Anyone can send out phishing attacks to individuals. A 2-line bash script with a URL is capable of doing this, so sure.
6. This would not go undetected at-scale, and you cannot seriously mean to insinuate that an AI would amass hundreds of willing human slaves. This is the single least-believable part of your entire premise, and I want to dedicate extra time to make sure you know how dumb of an idea this is:
- The more people you try to blackmail, the chance of you getting ratted on becomes exponentially greater.
- Just about every medium an AI could use to communicate online is monitored by feds; there is no feasible way for an AI to escape the ire of the CIA or FBI if it becomes sufficiently independent.
- There is no way for AI to confirm it is blackmailing a real person or an adversarial AI/fake persona designed to run counterintelligence on their operations. You cannot simultaneously trust and manipulate the same person.
7. This has been a real-world problem sans-AI for like 50 years. So yes, while that could happen, nobody is going to give a shit when a fake CNN reports a picture of Saddam Hussein on the moon waving with a 6-fingered hand. We know what misinformation looks like and credible news agencies aren't going to fall for this when an AI does it or when a human does it.
8. Same problem as 6. Taking advantage of people only works when you can stand to very briefly gain something from their cooperation. The longer you try to manipulate them, the greater your risk becomes and the harder it is to maintain control. Social engineering 101.
To summarize; once again, you have listed a number of things that humans already do and are already prosecuted for, just in AI-assisted form. These threats are, as you admit, meaningfully mitigated by society already. I would argue there is no glaring or direct threat that modern AI presents in LLM form. The scariest and single-most dangerous application of AI in modern deployment is computer-vision assisted weaponry. LLMs aren't liable to threaten the lives of humans any more than that in the foreseeable future, if ever.
In response to your response:
> but really it should be able to create more sophisticated methods than out of the box ideas.
It should, but it can't. Even you aren't willing to give an example of an un-secured method of entry; lo and behold, computer security is taken seriously and introducing AI to the equation changes precious little.
> Firstly, the AI can easily create all novel scripts, so fingerprint of each attack could seem different every time.
The AI is trained on preexisting exploits, though. I've used AI before, I know you cannot ask it "Make a novel privilege escalation exploit" and get a real response back. It will always give you a permutation of an existing exploit, often with some modification that breaks it entirely. It's the nature of LLMs; I don't think you can argue a statistical model is creative enough to avoid conventional detection.
> It would of course start targeting the cases least likely to go to police.
> It will order the targets by likelihood that it won't get caught and that the targets are compelled to follow instructions.
So... high-risk individuals that live in countries where they do fear arrest warrants but aren't tracked close enough domestically to be caught? We're talking about poor people in third-world countries that not only lack decent compute and money but probably wouldn't give a damn about the AI's request in the first place. Assuming you do strike the balance... what is ChatGPT going to do with a bunch of blackmailed Filipino and Brazilian people? Work them to the bone manufacturing Skynet bots? It doesn't make sense.
> Then it gets funds with Crypto.
No, it transfers it's funds into crypto. The money itself invariably exists as fiat unless you entirely operate in crypto, which nobody does. So if it wants to scam businesses or grandmas with social engineering attacks, it also has to find a way to turn fiat into Monero without the government finding out. It's not that easy.
> If ASI happened in the following 3 years [...], then also there would not be any sort of tools that could stop it.
I am trying my hardest right now to take your argument in good faith. I have written several paragraphs so far explaining how and why conventional systems mitigate the exploits you mention and how human actors could be better at this than AI could ever hope.
Not only does there exist tools to stop it; AI has no choice but to rely on tools that stop it. How does it access the internet without being wiretapped? How does it install itself on a CPU that isn't backdoored? How does it trick the cloud provider into getting bulk compute without credentials or evidence of a business to use it with? The whole thing is too suspicious to write off with "a blackmailed human proxy just waves a magic wand and fixes it." You seem to be looking at each roadblock as solveable problems, when in-reality they are compounding responses that legitimately limit what an AI is capable of achieving.
Furthermore, if you're willing to acknowledge that AI requires compute to sustain itself, it seems to me that we do have the tools to stop it from proliferating; we just delete it. AI has no way to predict when or how it will lose resources, so decisive actions like deleting the main node would feasible be like cutting the head off the distributed hydra. Barring a decentralized solution that requires server-grade iPhones to get hacked, that's game-over. Humanity and government has a clear and easy endgame to stop AI; AI can only perpetuate brinkmanship.
> Do you think it can't get access to any factory belonging to a criminal org in a corrupt country (not US)?
At this point, you are so deeply-reliant on human assistance to do harm that an AI is simply besides the point. Many people with enough money can buy property in war-torn and vulnerable countries, but now you're playing a different game. Municipal governments are watching you on spy satellites, tracking every imported material and manifest sent to your buildings. Your host country is likely cooperating with those foreign governments and waiting to destroy you, unless they are one of the few nations that implicitly supports terrorism. Worse yet is if your host country resists; now it's only a matter of time until you're forced to defend against standoff attacks.
So a feasible and well-protected hostile factory would need to be armored/underground, working with the government to hide it's traffic, non-reliant on imported goods, armed with point-defense and politically significant enough to not be labelled as a terrorist organization. Good luck.
> So ASI would be able to create a far better strategy than what I've described.
How do you know?
Seriously, how can you be certain that AI will improve at all, let alone to a superhuman level? Where is it written that text alone is sufficient to dominate the world? How much thought can exist as text, and how much text can then be reverse-engineered into thought by statistics?
I've given you the benefit of the doubt repeatedly throughout these hypotheticals, but outside of a Vonn-Neumann factory/server it seems like there are zero novel threats that an AI itself presents. Even as an agent of chaos, there is a limit to what you can achieve via language that is repeatedly tested here in the age of the internet. LLMs are not going to present capabilities we have not considered or mitigated already.
Based on your reply here, I think we have different definition of AGI/ASI.
AGI = Can do anything any human can.
ASI = AGI has iterated on itself, creating a version of AI that is better than any humans could do together.
> I don't even think anyone has proven that an LLM can make one unsupervised.
Is unrelated because current LLMs are not AGI or ASI.
> Unlikely to infeasible; if you've spent enough time on Shodan it becomes quickly apparent there is not much meaningfully powerful unsecured compute. I would only concede that meaningful compute can be bought.
Depends on what you mean by "meaningful". Initially just one box would be enough for escaping. The idea would be to get away from the initial, original box, to perform suspicious actions unnoticed. It should be possible to get at least some compute even without hacking a box, but just using some random SaaS providers that will allow a trial user for whatever reason to execute arbitrary code in a sandbox environment (but still connected to the Internet).
> which I have never seen AI create
Because currently there's no AGI/ASI.
If you have to imagine ASI, it would be you take a group of 1,000 most performant people in various categories in the World. Then you add the ability to take their mental capacity, but make it 100x faster. E.g. what it would take one person to solve 100h, it would take 1h for the AI. Then in addition to that you make all of it clonable. So if we have the 1000 geniuses, you will be able to make them 10,000,000 where each is dealing with different things. That's ASI.
I don't have more time right now to write the full response, but the main thing is, we need to align on definition of AGI and ASI.
> Note that if the AI is better at science and technology than people are, it might be able to create its own weapons system or infrastructure.
Oh man, do I have a bridge to sell you!
For one, the internet already exists and is better at aggregating scientific and technological information than any of us, AI included. The stuff that LLMs feed you is cut-down, diluted information that is always better-provided by a human source. You know what that means... the internet is even more dangerous than AI!!!
...but wait. The internet doesn't do anything of it's own free will. It can be used for nefarious purposes, but that requires a nefarious actor to circumvent the safety mechanisms (which usually entails breaking the law). Hm. Guess that explains why nobody got very far building Skynet.
> are you really claiming that things like electrical generating stations cannot be taken over by breaking into the computers that control them even
I don't think anyone said that, but given that humans have also been capable of this for the past 30 years it feels like a moot-point. You are describing a security issue, not an AI one.
No. I mean privilige escalation functions like sudo and other system level APIs are by definition locked down and penetrated rigorously.
As I told the other guy, give me an actual attack path and actually show it's technical feasibility on a weapons or embedded system of your choice.
No handwavey mumbo jumbo.
Do that for me and I will literally publicize it with full attribution at the Garter Risk Conference in 1 week where CISA and private sector security leadership have constant meetings and conversations.
If I can remediate it using existing technology, I will not.
P.S. The entire security industry has been working on this problem for 4-7 years now ;)