> It’s hard to take these safety concerns seriously
I don't get this mindset at all. How can it not be obvious to you that AI is an uniquely powerful and thus uniquely dangerous technology?
It's like saying nuclear missiles can't possibly be dangerous and nuclear arms reduction and non-proliferation treaties were a scam, because the US, China and the Soviet Union had positioned themselves to capture the majority of the strategic value nukes bring.
You have succinctly and completely summed up the AI risk argument more eloquently than anyone I've seen before. "How can it not be obvious?" Everything else is just intellectual fig leaves for the core argument that intuitively, without evidence, this proposition is obvious.
The problem is, lots of "obvious" things have turned out to be very wrong. Sometimes relatively harmlessly, like the obviousness of the sun revolving around the earth, and sometimes catastrophically, like the obviousness of one race being inherently inferior.
We should be very suspicious of policy that is based on propositions so obvious that it's borderline offensive to question them.
I follow LW to some degree, but even the best of it (like the post you link) feels very in-group confirmation centric.
That post is long and I have not read it all, but it seems to be missing any consideration of AGI upside. It’s like talking about the risk of dying in a car crash with no consideration of the benefits of travel. If I ask you “do you want to get in a metal can that has a small but non-zero chance of killing you”, of course that sounds like a terrible idea.
There is risk in AGI. There is risk in everything. How many people are killed by furniture each year?
I’m not dismissing AGI risk, I’m saying that I have yet to see a considered discussion that includes important context like how many people will live longer/happier because AGI helps reduce famine/disease. Somehow it is always the wealthy, employed, at-risk-of-disruption people who are worried, not the poor or starving or oppressed.
I’m just super not impressed by the AI risk crowd, at least the core one on LW / SSC / etc.
While I agree that the rhetoric around AI Safety would be better if it tried to address some of the benefits (and not embody the full doomer vibe), I don't think many of the 'core thinkers' are unaware of the benefits in AGI. I don't fully agree with this paper's conclusions, but I think https://nickbostrom.com/astronomical/waste is one piece that embodies this style of thinking well!
Thanks for the link -- that is a good paper (in the sense of making its point, though I also don't entirely agree), and it hurts the AI risk position that that kind of thinking doesn't get airtime. It may be that those 'core thinkers' are aware, but if so it's counter-productive and of questionable integrity to sweep that side of the argument under the rug.
I disagree that there's no reason to believe it will ever exist. For one thing, many smart people are trying to build the technology right now and they believe it to be possible. I see no compelling case that the intelligence scale simply tops-out with humans; that a more intelligent system is ruled out by the laws of physics.
The topic here is human extinction caused by AI. I don't know of any serious argument for why a non-general intelligence (really a system less intelligent than a human) would pose an extinction risk to humanity.
Plus, my background understanding of the people who signed this is that they're worried about AGI, not present-day systems, but that's an inference.
Maybe these AI Apocalypse articles published for general consumption would be justified if there were any signs whatsoever that we were on a path towards AGI but there are none, are there? Even the best we have today are still just machines. They are clearly not really intelligent. At best they simulate intelligence, but poorly (because they still make ridiculous mistakes). Just because there are no physical limits to intelligence doesn't mean it's possible for beings with finite intelligence to create infinite intelligence. It all just seems extremely premature to me.
> We should be very suspicious of policy that is based on propositions so obvious that it's borderline offensive to question them.
Mostly if the "obviousness" just masks a social taboo, which I don't see being the case here. Do you?
> The problem is, lots of "obvious" things have turned out to be very wrong.
A much bigger problem is that lots more "counter-intuitive" things that people like to believe because they elevate them over the unwashed masses have turned and continue to turn out to be very wrong and that this does not prevent them from forming the basis for important policy decisions.
I'm all for questioning even what appears intuitively obvious (especially if much rides on getting it right, as presumably it does here). But frankly, of the many bizarre reasons I have heard why we should not worry about AI the claim that it seems far too obvious that we should must be the single most perverse one yet.
> Everything else is just intellectual fig leaves for the core argument that intuitively, without evidence, this proposition is obvious.
Maybe your appraisal of what counts as evidence is defective?
For example, there's been a pattern of people confidently predicting AIs won't be able to perform various particular feats of the human mind (either fundamentally or in the next few decades) only to be proven wrong over increasingly shorter time-spans. And with AIs often not just reaching but far surpassing human ability. I'm happy to provide examples. Can you explain to me why you think this is does not count, in any way, as evidence that AIs have the potential to reach a level of capability that renders them quite dangerous?
> Mostly if the "obviousness" just masks a social taboo, which I don't see being the case here. Do you?
The social taboo here is saying that a position taken by lots of highly educated people is nonsense because they're all locked in a dumb purity spiral that leads to motivated reasoning. This is actually one of societies biggest taboos! Look at what happens to people who make that argument publicly under their own name in other contexts; they tend to get fired and cancelled really fast.
> there's been a pattern of people confidently predicting AIs won't be able to perform various particular feats of the human mind (either fundamentally or in the next few decades) only to be proven wrong over increasingly shorter time-spans
That sword cuts both ways! There have been lots of predictions in the last decade that AI will contribute novel and hithertofore unknown solutions to things like climate change or curing cancer. Try getting GPT-4 to spit out a novel research-quality solution to anything, even a simple product design problem, and you'll find it can't.
> the claim that it seems far too obvious that we should
They're not arguing that. They're saying that AI risk proponents don't actually have good arguments, which is why they so regularly fall back on "it's so obvious we shouldn't need to explain why it's important". If your argument consists primarily of "everyone knows that" then this is a good indication you might be wrong.
OK, I completely agree that if you feel that I invoked "obviousness" in an attempt of browbeating you and the GP with what in fact is a social taboo, you should be extra skeptical (I'm not sure that was the point the GP was trying to make though).
> If your argument consists primarily of "everyone knows that" then this is a good indication you might be wrong.
It doesn't though, does it? There's strong empirical evidence that AI systems are making rapid progress in many domains that previously only humans were good at, and a pace that basically surprised almost everyone. I gave a list of arguments in another thread why AI is uniquely powerful and dangerous. Which of these do you disagree with and why?
Arguments like yours are very subjective. What is "rapid". What is "surprising". I don't find them particularly surprising myself - cool and awesome - but I was being amazed by language modelling ten years ago! The quality kept improving every year. It was clear that if that kept up eventually we'd have language models that could speak to like people.
So the idea of a surprising change of pace doesn't really hold up under close inspection. LLM capabilities do seem to scale linearly, with the idea of emergent abilities coming under robust attack lately. To the extent big LLMs are surprising to a lot of people this has happened primarily due to throwing a previously implausible quantity of money at building them, and OpenAI releasing one of them from their lab prison that other companies were keeping them in, not due to any major new breakthrough in the underlying tech. The progress is linear but the visibility of that progress was not. The transformers paper was 5 years ago and GPT-4 is basically an optimization of that tech combined with RL, just executed very carefully and competently. Transformers in turn were an improvement over prior language models that could speak like a human, they just weren't as good at it.
> It doesn't though, does it?
It does. Arguments that consist of "everyone knows that" are also called rumours or folk wisdom. It's fine to adopt widely held beliefs if those beliefs rest on something solid, but what we have here is a pure argument from authority. This letter is literally one sentence long and the only reason anyone cares is the list of signatories. It's very reliant on the observer believing that these people have some unique insight into AI risk that nobody else has, but there's no evidence of that and many signers aren't even AI researchers to begin with.
2/3 deep learning Turing Price winners (Hinton and Benigo) are sufficiently shell-shocked by the rate of progress to be thrown into existential doubts (Hinton is very explicit about the fact that progress is much faster than he thought just a few years ago, Benigo speaks of how an "unexpected acceleration" in AI systems has radically shifted his perspective). Plenty of knowledgable people in the field who were not previously AI doomers are starting to sound a lot more concerned very recently.
As to the "oh it's just linear scaling of out-of-sight tech" line, well of course that itself was suprising. Gwern pushed the scaling hypothesis earlier than many and from what I remember even got pretty nasty attacks from AI insiders from it. Here's what he wrote 3 years ago: "To the surprise of most (including myself), this vast increase in size did not run into diminishing or negative returns, as many expected, but the benefits of scale continued to happen as forecasted by OpenAI.".
So sure there's some subjectivity involved here, but I'd like to see your propose some reasonable operationalization of "surprise at progress" that didn't class most laymen and insiders as suprised.
>> It doesn't though, does it?
> It does.
We seem to be miscommunicating, what I was trying to express is that my argument does not really require any appeal to authority. Trusting your lying eyes (to evaluate the progress of stuff like midjourney) and judging the quality of arguments should be enough (I spelt some reasons out here https://news.ycombinator.com/item?id=36130482, but I think hackinthebochs makes the point better here: https://news.ycombinator.com/item?id=36129980).
In fact I would still be pretty concerned even if most top AI guys were like LeCun and thought there is no real risk.
I will not deny, of course, that the fact that well known reasearchers like Hinton and Benigo are suddenly much more alarmed than they previously were and the ones like LeCun who are not seem to mostly make exceptionally terrible arguments doesn't exactly make me more optimistic.
I agree these statements from these long term researchers about them being surprised by the rate of progress are surprising.
To clarify my own thinking here, it's totally reasonable to me that people are surprised if:
1. They weren't previously aware of AI research (surely 99% of the population?)
2. They were but had stopped paying attention because it was just a long series of announcements about cool tech demos nobody outside big corps could play with.
3. They were paying attention but thought scaling wouldn't continue to work.
My problem is that people like Sam Altman clearly aren't in any of those categories and Hinton shouldn't have been in any, although maybe he fell into (3). I personally was in (2). I wasn't hugely surprised that ChatGPT could exist because I'd seen GPT-2, GPT-1, I'd seen surprising AI demos at Google years earlier and so on. The direction things were going in was kinda clear. I was a bit surprised by its quality, but that's because I wasn't really paying close attention as new results were published and the last InstructGPT step makes such a big difference to how the tech is perceived. Actual knowledge doesn't change much but once it's housetrained, suddenly it's so much easier to interact with and use that it makes a step change in how accessible the tech is and how it's perceived.
I think I was more surprised by the joining of LLMs with generators and how well AI art works. It does feel like that happened fast. But, maybe I just wasn't paying attention again.
So I guess where we differ is that I don't take their surprise at face value. The direction was too clear, the gap between the threats they talk about in the abstract and the concrete proposals are too large and too lacking in obvious logical connection; it feels like motivated reasoning to me. I'm not entirely sure what's really going on and perhaps they are genuine in their concerns but if so it's hard to understand why they struggle so much to make a convincing case given they are certainly intellectually equipped to do so.
The two posts you linked are interesting and much better argued than the website this thread is about, so I'll reply to them directly.
It is possible to believe that AI poses threat, while also thinking that the AI safety organizations currently sprouting up are essentially grifts that will do absolutely nothing to combat the genuine threat. Especially when their primary goal seems to be the creation of well-funded sinecures for a group of like-minded, ideologically aligned individuals who want to limit AI control to a small group of wealthy technologists.
But as you can see yourself, there are countless people even here, in a technical forum, who claim that AI poses no plausible threat whatsoever. I fail to see how one can reasonably believe that.
If you look the the world politics, basically if you hold enough nuclear weapons, you can do whatever you want to those who don't have them.
And based on the "dangers", new countries are prohibit to create them. And the countries which were quick enough to create them, holds all the power.
Their value is immeasurable especially for the Russia. Without them, they could not attack to Ukraine.
> non-proliferation treaties were a scam
And yes, they mostly are right now. Russia has backed from them. There are no real consequences if you are backing off, and you can do it in any time.
The parent commenter is most likely saying, that now the selected parties hold the power of AI, they want to prevent others to gain similar power, while maintaining all the value by themselves.
> There are no real consequences if you are backing off, and you can do it in any time.
That's not quite true. Sure, noone is going to start a war about such a withdrawal. However, nuclear arsenals are expensive to maintain and it's even more expensive to be in an arms race. Also, nobody wants to risk nuclear war if they can avoid it. Civilian populations will support disarmament in times where they don't feel directly threatened. That's why lot of leaders of all persuasions have advocated for and taken part in efforts to reduce their arsenals. Same goes for relations between countries generally and the huge economic benefits that come with trade and cooperation. Withdrawing from nuclear treaties endangers all of these benefits and increases risk. A country would only choose this route out of desperation or for likely immediate gain.
I think it really depends.
E.g. from the Western perspective, only US, UK, France, Russia and China have signed the treaty from nuclear countries.
India or Pakistan are not part of the treaties and for some reason, we don't see big problems.
There is only China left who might leave the treaty in the first place, anymore.
And we are so dependent of the China, that there is no guarantee for consequences. Should we treat China then equally than India? What that means?
Also, leaving the treaty does not mean that countries start massively increasing their arsenal. There will be just a lack of inspections and information exchange.
Most of the credible threats I see from AI that don't rely on a lot of sci-fi extrapolation involve small groups of humans in control of massively powerful AI using it as a force multiplier to control or attack other groups of humans.
Sam Altman's proposal is to create precisely that situation with himself and a few other large oligarchs being the ones in control of the leading edge of AI. If we really do face runaway intelligence growth and god-like AIs then this is a profound amount of power to place in the hands of just a few people. Even worse it opens the possibility that such developments could happen partly in secret, so the public might not even know how powerful the secret AIs under command of the oligarchs have become.
The analogy with nuclear weapons is profoundly broken in lots of ways. Reasoning from a sloppy analogy is a great way to end up somewhere stupid. AI is a unique technology with a unique set of risks and benefits and a unique profile.
It's not clear at all that we have an avenue to super intelligence. I think the most likely outcome is that we hit a local maximum with our current architectures and end up with helpful assistants similar in capability to George Lucas's C3PO.
The scary doomsday scenarios aren't possible without an AI that's capable of both strategic thinking and long term planning. Those two things also happen to be the biggest limitations of our most powerful language models. We simply don't know how to build a system like that.
> We simply don't know how to build a system like that.
Yes, but ten years ago, we also simply didn't know how to build systems like the ones we have today! We thought it would take centuries for computers to beat humans at Go[1] and at protein folding[2]. We didn't know how to build software with emotional intelligence[3] and thought it would never make jokes[4]. There's been tremendous progress, because teams of talented researchers are working hard to unlock more aspects of what the human brain can do. Now billions of dollars are funding bright people to look for ways to build other kinds of systems.
"We don't know how to do it" is the security-through-obscurity argument. It means we're safe only as long as nobody figures this out. If you have a security mindset, it's not enough to hope that nobody finds the vulnerability. You need to show why they certainly will not succeed even with a determined search.
> It's not clear at all that we have an avenue to super intelligence
AI already beats the average human on pretty much any task people have put time into, often by a very wide margin and we are still seeing exponential progress that even the experts can't really explain, but yes, it is possible this is a local maximum and the curve will become much flatter again.
But the absence of any visible fundamental limit on further progress (or can you name one?) coupled with the fact that we have yet barely begun to feel the consequences of the tech we already have (assuming zero breakthroughs from now on) makes we extremely wary to conclude that there is no significant danger and we have nothing to worry about.
Let's set aside the if and when of a super intelligence explosion for now. We are ourselves an existence proof of some lower bound of intelligence, that if amplified by what computers can already do (like perform many of the things we used to take intellectual pride in much better, and many orders of magnitude faster with almost infinitely better replication and coordination ability) seems already plenty dangerous and scary to me.
> The scary doomsday scenarios aren't possible without an AI that's capable of both strategic thinking and long term planning. Those two things also happen to be the biggest limitations of our most powerful language models. We simply don't know how to build a system like that.
Why do you think AI models will be unable to plan or strategize? Last I checked languages models weren't trained or developed to beat humans in strategic decision making, but humans already aren't doing too hot right now in games of adversarial strategy against AIs developed for that domain.
I dispute this. What appears to be exponential progress is IMO just a step function that made some jumps as the transformer architecture was employed on larger problems. I am unaware of research that moves beyond this in a way that would plausibly lead to super-intelligence. At the very least I foresee issues with ever-increasing computational requirements that outpace improvements in hardware.
We’ll see similar jumps when other domains begin employing specialized AI models, but it’s not clear to me that these improvements will continue increasing exponentially.
Right, and if someone can join the two, that could be something genuinely formidable. But does anyone have a credible path to joining the different flavors to produce a unity that actually works?
Even if someone will, I don't think it's an "existential risk". So, yes, I'm willing to make the bet. I'm also willing to make the bet that Santa never delivers nuclear warheads instead of presents. It's why I don't cap my chimney every Christmas Eve.
Between Covid, bank failures, climate change, and AI, it's like everyone is looking for something to be in a panic about.
>It's not clear at all that we have an avenue to super intelligence.
All problems in reality are probability problems.
If we don't have a path to superintelligence, then the worst problems just don't manifest themselves.
If we do have a path to super intelligence then the doomsday scenarios are nearly a certainty.
It's not really any different than saying "A supervolcano is unlikely to go off tomorrow, but if a supervolcano does go off tomorrow it is a doomsday scenario".
>We simply don't know how to build a system like that.
You are already a superintelligence when compared to all other intelligences on earth. Evolution didn't need to know how to build a system like that, and yet it still reached this point. And there is not really any to believe humanity is the pinnacle of intelligence, we are our own local maxima of power/communication limitations. An intelligence coupled with evolutionary systems design is much more apt to create 'super-' anything than the random walk alone.
Why are doomsday scenarios are certainty then. What's the model to get to that that isn't just some sort of scary story that waves away or into existence a lot of things we don't know if they can exist.
Let's say I was a small furry mammal that tasted really good, but also for some reason understood the world as it is now.
I would tell you that super intelligence had already happened. That super intelligence was humans. That humans happened to reach super intelligence by 1) having the proper hardware. 2) filtering noise from important information. 3) then sharing that information with others to amplify the power of intelligence 4) having a toolkit/tools to turn that information into useful things. 5) And with all that power humans can kill me off in mass, or farm me for my tasty meat at their leisure with little to nothing that I can do about it.
There doesn't appear to be any more magic than that. All these things already exist in biological systems that elevated humans far above their warm blooded peers. When we look at digital systems we see they are designed to communicate. You don't have an ethernet jack as a person. You can't speak the protocol to directly drive a 3 axis mill to produce something. Writing computer code is a pain in the ass to most of us. We are developing a universal communication intelligence, that at least in theory can drive tools at a much higher efficiency than humans will ever be able to.
Coming back to point 5. Cats/dogs are the real smart ones here when dealing with superintelligences. Get domesticated by the intelligence so they want to keep you around as a pet.
Do you think we could wipe out all furry mammals, for example? Could another intelligence have the same level of difference to us as in your story we to furry mammals? We don't even know if the mythical superintelligence could manifest the way you assume. It assumes that intelligence basically can overcome any obstacles - I'd say we actually see that seems not to be the case currently and claims that that is just a function of sufficient intelligence are unproven (setting aside physical limits to certain actions and results).
>Do you think we could wipe out all furry mammals, for example?
lets go with over a particular size. Lets say larger than the biggest rat. In that case yes, very easily. Once you get to rats it becomes far more difficult and you're pretty much just destroying the biosphere at that point.
> It assumes that intelligence basically can overcome any obstacles
In the case of human extinction, no, a super intelligence would not have to overcome any obstacles, it would just have to overcome obstacles better than we did.
Also, the superintelligence doesn't just have to overcome obstacles better than we did, it needs to overcome the right obstacles to succeed with human extinction.
We don't need an avenue to super-intelligence. We just need a system that is better at manipulating human beliefs and behaviour than our existing media, PR, and ad industries.
The problem is not science fiction god-mode digital quetta-smart hypercomputing.
This is about political, social, and economic influence, and who controls it.
That risk isn't about AI-as-AI. That risk is about AI-as-better-persuasive-nonsense-generator. But the same risk is there for any better-persuasive-nonsense-generator, completely independent from whether it's an AI.
It's the most persuasive actual risk I've seen so far, but it's not an AI-specific risk.
Effective dystopian mass-manipulation and monitoring are a real concern and we're closer to it[1] than to super intelligence. But super-intelligence going wrong is almost incomparably worse. So we should very much worry about it as well.
[1] I'm not even sure any further big breakthroughs in AI are needed, i.e. just effective utilization of existing architectures probably already suffices.
A super intelligent AI is not necessary for AI to be an threat. Dumb AIs that are given access to the internet plus a credit card and told to maximize profit could easily cause massive damage. We are not far from such an AI being accessible to the masses. You can try to frame this like the gun debate "it's not the AI it's the people using it" but the AI would be acting autonomously here. I have no faith that people won't do extremely risky things if given the opportunity.
It isn't obvious to me. And I've yet to read something that spills out the obvious reasoning.
I feel like everything I've read just spells out some contrived scenario, and then when folks push back explaining all the reasons that particular scenario wouldn't come to pass, the counter argument is just "but that's just one example!" without offering anything more convincing.
Do you have any better resources that you could share?
The history of humanity is replete with examples of the slightly more technologically advanced group decimating their competition. The default position should be that uneven advantage is extremely dangerous to those disadvantaged. This idea that an intelligence significantly greater than our own is benign just doesn't pass the smell test.
From the tech perspective: higher order objectives are insidious. While we may assume a narrow misalignment in received vs intended objective of a higher order nature, this misalignment can result in very divergent first-order behavior. Misalignment in behavior is by its nature destructive of value. The question is how much destruction of value can we expect? The machine may intentionally act in destructive ways as it goes about carrying out its slightly misaligned higher order objective-guided behavior. Of course we will have first-order rules that constrain its behavior. But again, slight misalignment in first-order rule descriptions are avenues for exploitation. If we cannot be sure we have zero exploitable rules, we must assume a superintelligence will find such loopholes and exploit them to maximum effect.
Human history since we started using technology has been a lesson on the outcome of an intelligent entity aimed at realizing an objective. Loopholes are just resources to be exploited. The destruction of the environment and other humans is just the inevitable outcome of slight misalignment of an intelligent optimizer.
If this argument is right, the only thing standing between us and destruction is the AGI having reached its objective before it eats the world. That is, there will always be some value lost in any significant execution of an AGI agent due to misalignment. Can we prove that the ratio of value created to value lost due to misalignment is always above some suitable threshold? Until we do, x-risk should be the default assumption.
OK, which of the following propositions do you disagree with?
1. AIs have made rapid progress in approaching and often surpassing human abilities in many areas.
2. The fact that AIs have some inherent scalability, speed, cost, reliability and compliance advantages over humans means that many undesirable things that could previously not be done at all or at least not done at scale are becoming both feasible and cost-effective. Examples would include 24/7 surveillance with social desirability scoring based on a precise ideological and psychological profile derived from a comprehensive record of interactions, fine-tuned mass manipulation and large scale plausible falsification of the historical record. Given the general rise of authoritarianism, this is pretty worrying.
3. On the other hand the rapid progress and enormous investment we've been seeing makes it very plausible that before too long we will, in fact, see AIs that outperform humans on most tasks.
4. AIs that are much smarter than any human pose even graver dangers.
5. Even if there is a general agreement that AIs pose grave or even existential risks, states, organizations and individuals will are all incentivized to still seek to improve their own AI capabilities, as doing so provides an enormous competitive advantage.
6. There is a danger of a rapid self-improvement feedback loop. Humans can reproduce, learn new and significantly improve existing skills, as well as pass skills on to others via teaching. But there are fundamental limits on speed and scale for all of these, whereas it's not obvious at all how an AI that has reached super-human level intelligence would be fundamentally prevented from rapidly improving itself further, or produce millions of "offspring" that can collaborate and skill-exchange extremely efficiently. Furthermore, since AIs can operate at completely different time scales than humans, this all could happen extremely rapidly, and such a system might very quickly become much more powerful than humanity and the rest of AIs combined.
I think you only have to subscribe a small subset of these (say 1.&2.) to conclude that "AI is an uniquely powerful and thus uniquely dangerous technology" obviously follows.
For the stronger claim of existential risk, have you read the lesswrong link posted elsewhere in this discussion?
Reading the lesswrong link, the parts I get hung up on are that it appears in these doomsday scenarios humans lose all agency. Like, no one is wondering why this computer is placing a bunch of orders to DNA factories?
Maybe I’m overly optimistic about the resilience of humans but these scenarios still don’t sound plausible to me in the real world.
> Like, no one is wondering why this computer is placing a bunch of orders to DNA factories?
I'm not that confident that if we put you in a box, tron-style, where you basically continued to enjoy your existing level of intelligence, but think 10'000x faster, have Petabytes of information at your fingertips and can clone yourself and losslessly and rapidly exchange knowledge with your clones and had a few days to think about it (~a few thousand years of thought at your normal speed) you couldn't figure out a way to effect a bunch of orders to DNA factories without anyone raising an alarm.
Are you?
Now what if we actually consider an actual AI after a few self-improvement steps. Any reasons to expect it wouldn't be 10'000x+ smarter than you as well, or roughly the difference in intelligence between you and an ant? Could you outsmart a bunch of ultra-ultra-slow-motion ants?
You could place a bunch of orders into services that syntetize DNA or proteins rn.
Some people are even working on stuff like automating protein design whith AI.
There's no reason why humans should notice anything word about a particular order on a service like that.
> 3. ... before too long we will ... see AIs that outperform humans on most tasks.
This is ambiguous. Do you mean
A. that there is some subset T1 of the set of all tasks T such that T1 is "most of" T, and that for each P in T1 there will be an AI that outperforms humans on P, or
B. There will be a single AI that outperforms humans on all tasks in a set T1, where T1 is a subset of all tasks T such that T1 is "most of" T?
I think A is unlikely but plausible but I don't see cause for worry. I don't see any reason why B should come to pass.
4. AIs that are much smarter than any human pose even graver dangers.
Sure. Why should we believe they will ever exist though?
I think between point 3 and 4 there is a leap to talking about “danger”. Perhaps the disagreement is about what one calls “danger”. I had perhaps mistakenly assumed we were talking about an extinction risk. I’ll grant you concerns about scaling up things like surveillance but there is a leap to being an existential risk that I’m still not following.
AI will not have the instinctual drives for domination or hunger that humans do.
It seems likely that the majority of AI projects will be reasonably well aligned by default, so I think 1000 AIs monitoring what the others are doing is a lot safer than a single global consortium megaproject that humans can likely only inadequately control.
The only reasonable defense against rogue AI is prosocial AI.
Replying here, crossing over from the other thread.
Where we depart is point 4. Actually, both point 3 and 4 are things I agree with, but it's implied there's a logical link or progression between them and I don't think there is. The problem is the definitions of "outperform humans" and "smart".
Current AI can perform at superhuman levels in some respects, yes. Midjourney is extremely impressive when judged on speed and artistic skill. GPT-4 is extremely impressive whilst judged on its own terms, like breadth of knowledge. Things useful to end users, in other words. LLMs are deeply unimpressive judged on other aspects of human intelligence like long term memory, awareness of time and space, ability to learn continuously, willingness to commit to an opinion, ability to come up with interesting new ideas, hide thoughts and all that follows from that like being able to make long term plans, have agency and self-directed goals etc ... in all these areas it is weak. Yet, most people would incorporate most of them into their definition of smart.
Will all these problems be solved? Some will, surely, but for others it's not entirely clear how much demand there is. Boston Robotics was making amazing humanoid parkour bots for years yet the only one they seem able to actually sell is dog-like. Apparently the former aren't that useful. The unwillingness to commit to an opinion may be a fundamental trait of AI for as long as it's centralized, proprietary and the masses have to share a single model. The ability to come up with interesting new ideas and leaps of logic may or may not appear, it's too early to tell.
But between 3 and 4 you make a leap and assume that not only will all those areas be conquered very soon, but that the resulting AI will be unusually dangerous. The various social ills you describe don't worry me though. Bad governments will do bad things, same old, same old. I'm actually more worried about people using the existence of AI to deny true evidence rather than manufacture false evidence en-masse. The former is a lot of work and people are lazy. COVID showed that people's capacity for self-deception is unlimited, their willingness to deny the evidence of their own eyes is bottomless as long as they're told to do it by authority figures. You don't even need AI to be abused at all for someone to say, "ignore that evidence that we're clueless and corrupt, it was made by an AI!"
Then by point 6 we're on the usual trope of all public intellectuals, of assuming unending exponential growth in everything even when there's no evidence of that or reason to believe it. The self-improving AI idea is so far just a pipe dream. Whilst there are cases where AI gets used to improve AI via self-play, RLHF and so on, it's all very much still directed by humans and there's no sign that LLMs can self improve despite their otherwise impressive abilities. Indeed it's not even clear what self-improvement means in this case. It's a giant hole marked "??? profit!" at the heart of the argument. Neurosurgeons can't become superintelligences by repeatedly performing brain surgery on themselves. Why would AI be different?
Nuclear missiles present an obvious danger to the human body. AI is an application of math. It is not clear how that can be used directly to harm a body.
The assumption seems to be that said math will be coupled with something like a nuclear missile, but in that case the nuclear missile is still the threat. Any use of AI is just an implementation detail.
Germany, for example would disagree with you. They believe violent speech is an act of violence in itself.
>AI is an application of math.
It turns out that people hook computers to 'things' that exist in the physical world. You know like robot bodies, or 3D printers. And as mentioned above, even virtual things like social media can cause enough problems. People hook AI to tools.
And this is just the maybe not quite general AI we have now. If and when we create a general AI that with self-changing feedback loops then all this "AI is just a tool" asshattery goes out the window.
Remember at the end of the day, you're just an application of chemistry that is really weak without your ability to use tools and to communicate.
> It turns out that people hook computers to 'things' that exist in the physical world.
But those physical things would be the danger, at least if you consider the nuclear missile to be the danger. It seems you are trying to go down the "guns don't kill people, people kill people" line of thinking. Which is fine, but outside of the discussion taking place.
There are many relevant things that already exist in the physical world and are not currently considered dangers: ecommerce, digital payments, doordash-style delivery, cross-border remittances, remote gig work, social media fanning extreme political views, event organizing.
However, these are constituent elements that could be aggregated and weaponized by a maleficent AI.
Maleficent humans are constantly trying to use these elements for their own gain, often with little to no regards to other humans (especially out groups). This happens both individually, in small groups, in large organizations and even multiple organization colluding. Both criminal, terrorist, groups at war, along with legal organizations such as exploitative companies and regressive interest organizations, et.c.. And we have tools and mechanisms in place to keep the level of abuse at bay. Why and how are these mechanisms unsuitable for protecting against AI?
>Why and how are these mechanisms unsuitable for protecting against AI?
The rule of law prevented WWI and WWII, right? Oh, no it did not, tens to hundreds of millions died due to human stupidity and violence depending on what exactly you count in that age.
> Both criminal, terrorist, groups at war
Human organizations, especially criminal organizations have deep trust issues between agents in the organization. You never know if anyone else in the system is a defector. This reduces the openness and quantity of communication between agents. In addition you have agents that want to personally gain rather than benefit the organization itself. This is why Apple is a trillion dollar company following the law... mostly. Smart people can work together and 'mostly' trust the other person isn't going to screw them over.
Now imagine a superintelligent AI with a mental processing bandwidth of hundreds of the best employees at a company. Assuming it knows and trusts itself, then the idea of illegal activities being an internal risk disappears. You have something that operates more on the level of a hivemind toward a goal (what the limitations of hivemind versus selfish agents are is another very long discussion). What we ask here is if all the worlds best hackers got together, worked together unselfishly, and instigated an attack against every critical point they could find on the internet/real world systems at once, how much damage could they cause?
Oh, lets say you find the server systems the super intelligence is on, but the controller shuts it off and all the data has some kind of homomorphic encryption so that's useless to you. It's dead right? Na, they just load up the backup copy they have a few months later and it's party time all over again. Humans tend to remain dead after dying, AI? Well that is yet to bee seen.
Those tangible elements would conceivably become the danger, not the AI using those elements. Again, the "guns don't kill people, people kill people" take is all well and good, but well outside of this discussion.
False premise. One can start new threads about complimentary subjects and they can be thought about in parallel. You don't have to try and shove all of the worlds concepts into just one thought train to be able to reason about them. That's how you make spaghetti.
>It seems you are trying to go down the "guns don't kill people, people kill people" line of thinking.
"Guns don't kill people, AIs kill people" is where we are going, I think. This is the discussion: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."
The discussion is not about a mathematical representation of AI. The discussion is about the actual implementation of AI on physical computing infrastructure which is accessible by at least one human on planet earth.
The credible danger, argued in various places, including superintelligence by Nick Bostrom, is that the "system under review" here is "every physical system on planet earth" because an AI could gain access to whatever systems exist on said planet, including human minds (see "Nazis").
So much as we might discuss the problems of letting a madman get control of the US, Russian, UK, French or Chinese nuclear arsenals, we might discuss the problem of building an AI if the act of building the AI could result in it taking over the nuclear arsenals of those countries and using it against humans. That takeover might involve convincing a human it should do it.
I don't understand this argument (the "Terminator" scenario). AI could conceivably self replicate and evolve as software but it still needs hardware to run on, power, networking etc etc. There's no better way to kill that than to start a nuclear war or engineer some super virus that kills everyone.
It's hard to see any scenario where AI could become a dominant force without significant human collaboration. Perhaps somewhere like North Korea where a very small elite has complete control over the the population it could happen but it sounds a lot like sci-fi to me. I'd love to hear some plausible scenarios for the counter-argument. I've seen a lot of "I think there's an x% chance we're in trouble" arguments which might be convincing for job losses, but I don't find at all plausible as a case for human extinction or indentured servitude (the "Matrix" scenario).
Sure. I recommend reading superintelligence by Nick Bostrom.
But I think the key failure here, in your thinking, is that you can't conceive of something, and that therefore something can't happen, and you're doing that in the context of things that are smarter than you. You being able to conceive of it is simply not required for it to happen.
Also, Colossus: The Forbin Project was a great movie from 1970 on this subject. Spoilers: an AI takes control of the nuclear arsenal and then threatens humans with extinction if they do not serve it. The humans do serve it, of course, because the humans in charge don't want to die, and are entirely fine with enslaving the rest of us.
The book superintelligence by Nick Bostrom gets into the fine details of all the different ways an AI would escape, why it would escape, and why it wouldn't take a chance with a species that murders its own kind for fun and profit.
Superintelligence and Life 3.0 seem to come up as recurring references in the discussion. I've only read synopses of both but frankly I find the argument that melevolent AI "escape" could occur without being noticed and thwarted a bit far fetched.
There's a good counterargument here[1] that seems reasonable:
"One of the features of intelligence explosion that most preoccupies Bostrom and Yudkowsky is that it’s not a problem that we get to have many attempts at. In the Terminator movies, humans don’t get to approach a newly self-aware Skynet and request a do over. One minute Skynet is uncomplainingly complying with all human directives. The next, it’s nuking us. I suspect that we are likely to have plenty of opportunities for do overs in our attempts to make autonomous AIs. Autonomy is not an all-or-nothing proposition. The first machine agents are likely to be quite clumsy. They may be capable of forming goals in respect of their world but they won’t be particularly effective at implementing them. This gives us plenty of opportunity to tweak their programming as they travel the path from clumsy to sophisticated agency"
Ok - fair comment. So I found a copy of Superintelligence. The argument is pulp sci-fi at best:
"The final phase begins when the AI has gained sufficient strength to obviate the need for secrecy. The AI can now directly implement its objectives on a full scale. The overt implementation phase might start with a “strike” in which the AI eliminates the human species and any automatic systems humans have created that
could offer intelligent opposition to the execution of the AI’s plans. This could be achieved through the activation of some advanced weapons system that the AI has
perfected using its technology research superpower and covertly deployed in the covert preparation phase. If the weapon uses self-replicating biotechnology or nano-technology, the initial stockpile needed for global coverage could be microscopic: a single replicating entity would be enough to start the process. In order to ensure a sudden and uniform effect, the initial stock of the replicator might have been deployed or allowed to diffuse worldwide at an extremely low, undetectable con-centration. At a pre-set time, nanofactories producing nerve gas or target- seeking mosquito-like robots might then burgeon forth simultaneously from every square meter of the globe (although more effective ways of killing could probably be devised by a machine with the technology research superpower)."
Look - nothing's impossible, but I agree with the counter argument that an advanced AI still starts as a "brain in a vat", with no experience of agency in the physical world. In order to successfully take over you have to assume it can develop all the physical world capability it needs, in secret and get it right first time. That seems implausible.
Exactly. While there is an argument to be made that people are the real danger, that is beyond the discussion taking place. It has already been accepted, for the sake of discussion, that the nuclear missile is the danger, not the math which developed the missile, nor the people who thought it was a good idea to use a missile. Applying AI to the missile still means the missile is the danger. Any use of AI in the scope of that missile is just an implementation detail.
You said that "AI is an application of math. It is not clear how that can be used directly to harm a body." I was trying to illustrate the case that if humans can develop harmful things, like nuclear weapons, then an AI that is as smart as a human can presumably develop similarly harmful things.
If the point you are trying to make is that an AI which secretly creates and deploys nuclear, biological, or chemical weapons in order to destroy all of humanity, is not an "AI risk" because it's the weapons that do the actual harm, then... I really don't know what to say to that. Sure, I guess? Would you also say that drunk drivers are not dangerous, because the danger is the cars that they drive colliding into people's bodies, and the drunk driver is just an implementation detail?
> I was trying to illustrate the case that if humans can develop harmful things, like nuclear weapons, then an AI that is as smart as a human can presumably develop similarly harmful things.
For the sake of discussion, it was established even before I arrived that those developed things are the danger, not that which creates/uses the things which are dangerous. What is to be gained by ignoring all of that context?
> I really don't know what to say to that. Sure, I guess?
Nothing, perhaps? It is not exactly something that is worthy of much discussion. If you are desperate for a fake internet battle, perhaps you can fight with earlier commenters about whether it is nuclear missiles that are dangerous or if it is the people who have created/have nuclear missiles are dangerous? But I have no interest. I cannot think of anything more boring.
I'm specifically worried that an AGI will conceal some instrumental goal of wiping out humans, while posing as helpful. It will helpfully earn a lot of money for a lot of people, by performing services and directing investments, and with its track record, will gain the ability to direct investments for itself. It then plows a billion dollars into constructing a profitable chemicals factory somewhere where rules are lax, and nobody looks too closely into what else that factory produces, since the AI engineers have signed off on it. And then once it's amassed a critical stockpile of specific dangerous chemicals, it releases them into the atmosphere and wipes out humanity / agriculture / etc.
Perhaps you would point out that in the above scenario the chemicals (or substitute viruses, or whatever) are the part that causes harm, and the AGI is just an implementation detail. I disagree, because if humanity ends up playing a grand game of chess against an AGI, the specific way in which it checkmates you is not the important thing. The important thing is that it's a game we'll inevitably lose. Worrying about the danger of rooks and bishops is to lose focus on the real reason we lose the game: facing an opponent of overpowering skill, when our defeat is in its interests.
Cool, I guess. While I have my opinions too, I'm not about to share them as that would be bad faith participation. Furthermore, it adds nothing to the discussion taking place. What is to be gained by going off on a random tangent that is of interest to nobody? Nothing, that's what.
To bring us back on topic to try and salvage things, it remains that it is established in this thread that the objects of destruction are the danger. AI cannot be the object of destruction, although it may be part of an implementation. Undoubtedly, nuclear missiles already utilize AI and when one talks about the dangers of nuclear missiles they are already including AI as part of that.
Yes, but usually when people express concerns about the danger of nuclear missiles, they are only thinking of those nuclear missiles that are at the direction of nation-states or perhaps very resourceful terrorists. And their solutions will usually be directed in that direction, like arms control treaties. They aren't really including "and maybe a rogue AI will secretly build nuclear weapons on the moon and then launch them at us" in the conversation about the danger of nukes and the importance of international treaties, even though the nukes are doing the actual damage in that scenario. Most people would categorize that as sounding more like an AI-risk scenario.
Please read Life 3.0 or superintelligence. There are people that spent decades thinking about how this would happen. You spent a little bit of time and conclude it can't.
> It's like saying nuclear missiles can't possibly be dangerous and nuclear arms reduction and non-proliferation treaties were a scam, because the US, China and the Soviet Union had positioned themselves to capture the majority of the strategic value nukes bring.
I'm honestly not sure if this is sarcasm. The non-proliferation treaties are indeed a scam. The war is raging between the US and Russia and nuclear is a big part of it (though just words/threats for now). It's nonsensical to think that these treaties are possible.
Not only is the Non proliferation treaty possible, it's been evidently effective in slowing the proliferation of nuclear weapons. The only country that ratified or acceded to it and went on to develop nuclear weapons is North Korea, and the only country that ratified or acceded to it and looks on track to develop nuclear weapons is Iran. One country that was not a signatory and developed nuclear weapons voluntarily gave them up and acceded to it partly due to international pressure (South Africa). Israel, Pakistan and India have since developed nuclear weapons but they were never signatories, the only other non-signatory is South Sudan which probably won't acquire nuclear capabilities anytime soon.
And I don't get the opposed mindset, that AI is suddenly going to "become a real boy, and murder us all".
Isn't it a funny coincidence how the popular opinion of AIs aligns perfectly with blockbusters and popular media ONLY? People are specifically wanting to prevent Skynet.
The kicker (and irony to a degree) is that I really want sapient AI to exist. People being so influenced by fiction is something I see as a menace to that happening in my lifetime. I live in a world where the majority is apparently Don Quixote.
- Point one: If the sentient AI can launch nukes, so can your neighbor.
- Point zwei: Redistributing itself online to have unlimited compute resources is a fun scenario but if networks were that good then Stadia wouldn't have been a huge failure.
- Point trois: A distributed-to-all-computers AI must have figured out universal executables. Once we deal with the nuclear winter, we can plagiarize it for ourselves. No more appimage/snap/flatpak discussions! Works for any hardware! No more dependency issues! Works on CentOS and Windows from 1.0 to 11! (it's also on AUR, of course.)
- Point cuatro: The rogue AI is clearly born as a master hacker capable of finding your open ports, figure out any exploits or create 0-day exploits to get in, and hope there's enough resources to get the payload injected, then pray no competent admin is looking at the thing.
- Point go: All of this rides on the assumption that the "cold, calculating" AI has the emotional maturity of a teenager. Wait, but that's not what "cold, calculating" means, that's "hothead and emotional". Which is it?
- Point six: Skynet lost, that's the point of the first movie's plot. If everyone is going to base their beliefs after a movie, at least get all the details. Everything Skynet did after the first attack was full of boneheaded decisions that only made the situation worse for it, to the point the writers cannot figure ways to bring Skynet back anymore because it doomed itself in the very first movie. You should be worrying about Legion now, I think. It shuts down our electronics instead of nuking.
Considering it won't have the advantage of triggering a nuclear attack because that's not how nukes work, the evil sentient AI is so doomed to fail it's ridiculous to think otherwise.
But, companies know this is how the public works. They'll milk it for all it's worth so only a few companies can run or develop AIs, maybe making it illegal otherwise, or liable for DMCAs. Smart business move, but it affects my ability to research and use them. I cannot cure people's ability to separate reality and fiction though, and that's unfortunate.
A counter point here is you're ignoring all the boring we all die scenarios that are completely possible but too boring to make a movie about.
The AI hooked to a gene sequencer/printer test lab is something that is nearly if not completely possible now. It's something that can be relatively small in size compared with the facilities needed to make most weapons of mass destruction. It's something that is highly iterative, and parallelizable. And it's something powerful enough that if targeting at the correct things (kill all rice, kill all X people) that it easily spills over in to global conflict.
You're a priori writing off my comment as fruitless because of your emotions and not because you actually have given it deep thought and carefully reached the conclusion that social feedback is somehow bad.
Also, the notion that "people's work" is inherently worthy of respect is just nonsensical. I do shoddy work all the time. Hell, you just casually dismissed my internet comment work as shallow and told me not to do it. Please don't post a shallow dismissal of my work.
Don't you think that this is all a bit anti-intellectual?
I don't get this mindset at all. How can it not be obvious to you that AI is an uniquely powerful and thus uniquely dangerous technology?
It's like saying nuclear missiles can't possibly be dangerous and nuclear arms reduction and non-proliferation treaties were a scam, because the US, China and the Soviet Union had positioned themselves to capture the majority of the strategic value nukes bring.