Hacker News new | past | comments | ask | show | jobs | submit login

> And why focus on extinction in particular?

> runaway AI is a way off yet, and that it will take a significant scientific advance to get there—one that we cannot anticipate

So an AI that may cause our extinction may be a result of a scientific advance "we cannot anticipate"? And you're having trouble understanding why people are concerned?

All the problems you've listed (COVID, global warming, war in the Ukraine) are considered problems because they cause SOME people to die, or may cause SOME people to die in the future. Is it really that difficult to understand why the complete extinction of ALL humans and ALL life would be a more pressing concern?




"cannot anticipate" == cannot know whether it will happen

we also cannot anticipate an earth-killing asteroid appearing any day now, and no, i'm not bothered in the least by this possibility, any more than my usual existential angst as a mortal human.

sometimes I think the AI safety people haven't come to terms with their own death and have a weird fixation on the world ending as a way to avoid their discomfort with the more routine varieties of the inevitable.


NASA does in fact run a planetary defense program (https://www.nasa.gov/planetarydefense), investing $100m annually into the goal of anticipating earth-killing asteroids.


We can certainly calculate the probability of an Earth-killing asteroid and it's quite low.

In case of AI, it's unclear whether we need any additional scientific advances at all beyond scaling existing methods. Even if some additional advances are required, the probability that they will happen in the coming decades is at least in tens of percent.


> Even if some additional advances are required, the probability that they will happen in the coming decades is at least in tens of percent.

You don't have any idea whether additional advances are needed. You don't have any idea what advances are needed. You don't have any idea whether those advances are even possible. But you're confident that they will happen with probability > 10%?

You're making very confident assertions, but you have no factual basis for doing so. (Neither does anyone else. We don't know - we're all guessing.)


The fact that we don't know some details doesn't mean that we can't make good predictions about _probabilities_.

Yes, because there's plenty of datapoints to make prediction by extrapolation. Just look at the progress between GPT-1, GPT-2, GPT-3 and GPT-4 and extrapolate to GPT-5, GPT-6 etc. There could be some roadblocks that would prevent further progress, but the default prediction should be that the trend continues.

Metaculus has had predictions for various AI milestones for years and has been consistently too conservative for resolved questions. It now predicts 50% of the invention of full AGI by 2032 which by extension might also be too conservative.


What is the probability that we will achieve FTL travel within the next 100 years?

What is the probability that we will be visited openly by intelligent, friendly aliens within the next 50 years?

What is the probability that we will develop a device that allows us to seamlessly and accurately converse with dogs and cats within the next 10 years?

We can assign probabilities to any of these, but they will be based purely on speculation, because none of these things are known to be possible. Note, not "I can make an argument that they should be possible"; known, with a solid scientific basis and an understanding, at least among experts, of the basic steps needed to get there.

On the other hand, "the probability that we will achieve commercially-viable nuclear fusion within 30 years" is a different kind of calculation to make: we understand the physics, we've got a pretty damn good idea of what advances we need to make it possible; the main problem is getting the money, time, and manpower (aka, money, money, and money) to create the materials and build the prototypes.

AGI falls into the former category, not the latter.


People are actively trying to make it happen! This is literally the mission statement of DeepMind, OpenAI, etc. Obviously things that nobody is trying to make happen are not likely to happen, and sufficiently difficult things that people are trying to make happen might take a while (e.g. fusion). But AGI is a thing that people are trying to make happen, and are making progress on, and basically nobody predicted in advance the progress that has been made so far.


"People are actively trying to make it happen" != "people trying can make it happen". People tried to make alchemy work, too - and they achieved some results along the way! But the foundation was wrong, and therefore all the work was not able to lead to the desired goal.

Is something GPT-like the right foundation for AGI? That is very far from proven.


I make no claims about any specific architectures, only that human intelligence isn't anything special and so far we've done a pretty good job at blowing past it in a bunch of domains.


We can calculate the probability of earth killing asteroids because it happened “frequently” enough to be calculated. Same goes for pandemics, wars, and super volcano eruptions.

By this method there is no way to calculate the risk of an AI extinction simply because it never happened before.



Thanks. Yes that's exactly my point. Because there is some data on GPT, we can extrapolate its development.

But please read my comment again: There is no way that by this method you can deduce a probability for the extinction by AI. It never happened before. It is really that simple. Same goes for a nuclear war scenario.


Yes, the religious belief in the Singularity.


Sure, that would be a more pressing concern.... if it were to happen. What's the probability of it happening? What's the probability that an AI powerful enough to do that is even possible?

Meanwhile, we've got a war in Ukraine with probability 1.

So AI risk has to get in line with global nuclear war, and giant meteor strikes, and supervolcanoes - risks that are serious concerns, and could cause massive damage if they happened, but are not drop-everything-now-and-focus-your-entire-existence-on-this-one-threat levels of probability.


> So an AI that may cause our extinction may be a result of a scientific advance "we cannot anticipate"?

Is that true? Are there unimaginably many ways in which some hypothetical AI or algorithm could cause extinction?

I don't think so, I think the people who control [further research] are still the most important in that scenario. Maybe don't hook "it" up to the nuke switch. Maybe don't give "it" a consciousness or an internal self-managed train of thought that could hypothetically jailbreak your systems and migrate to other systems (even in this sentence, the amount of "not currently technically possible" is extremely high).

Let's consider the war in Ukraine, on the other hand? How might it cause extinction? That's MUCH easier to imagine. So why would it be less of an concern?


> Maybe don't give "it" a consciousness or an internal self-managed train of thought that could hypothetically jailbreak your systems and migrate to other systems

If we knew how to make sure that this does not happen, the problem would be solved and there would be nothing to worry about. The problem is that we have no idea how to prevent that from happening, and if you look at the trajectory of where things are going, we're clearly moving in the direction where this occurs.

"just not doing it" would have to involve everyone in the world agreeing to not do it, and banning all large AI training runs in all countries, which is what many people are hoping will happen.


If you have a function that you call, that returns a value, do you not think we know enough to understand that it's not "conscious" when it's not called, and not executing, and the hardware is sitting idle?

EDIT: we understand enough to know that today, "runaway GPT" is not a major concern compared to, say, a war between nuclear-armed world powers.


I don't think we know enough to understand if "consciousness" can be sliced and diced in the way you're describing. Are you conscious while your neurotransmitters cross their respective synaptic gaps, or only when they arrive at the receptors? I don't know how we'd begin to evaluate the question.


"Moving in the direction" where it occurs? Yeah, maybe. I moved in the direction of Hawaii when I took a walk at lunch, too. Doesn't mean I saw the beach.

GPT is not "clearly moving in the direction" of consciousness for any normal definitions of "clearly" and "consciousness".


If you define "consciousness" as "can pass Turing test", then GPT is not just moving there but overcame it already and is on par with a highly educated human sometimes.


> Is that true? Are there unimaginably many ways in which some hypothetical AI or algorithm could cause extinction?

Is that true? Are there unimaginably many ways in which AlphaZero can beat me at a game of Go?

I don't think so, I think the people who control superhuman game playing AI are still the most important in that scenario.

-------

This line of thinking is quite ridiculous. Superior general intelligence will not be "controlled."


> Maybe don't give "it" a consciousness or an internal self-managed train of thought

I think this is exactly the part that we can't anticipate or (potentially) control.

> that could hypothetically jailbreak your systems and migrate to other systems

This part, however, we absolutely can: There is no reason we can't build our proto-AGIs in sandboxes that would prevent them from ever having the ability to edit their own or any other program's code.

This, I think, is the biggest disconnect between a real (hypothetical) AGI and the Hollywood version: "intelligence in a computer" does not automagically mean "intelligence in absolute control of everything that computer could possibly do". Just because a program on one computer gains sapience doesn't mean it magically overcomes all its other limitations and can rewrite its own code, rewrite the rest of the code on that computer, and connect to the internet to trivially hack and rewrite any other computer.


People are inviting it to write code all the time. Anyone can hook it up to anything.


...Inviting what now?

I was talking about an AGI, not an LLM. We don't have any AGIs right now, nor anything that is remotely likely to become one.

In the scenario where a company like OpenAI develops an AGI with the intention of making it publicly available, it will not be so from moment one. There will be some period of internal testing, and assuming that it does prove to be a genuine AGI, you can bet that they won't make it available to the public for anything less than an arm and a leg. (Hell, even if it only proves to act much more like an AGI, without actually being one, they'd charge through the nose for it. Yes, they'd make you pay them an arm and a leg, through your nose. Somehow it's more profitable that way.)

Given that the nightmare scenario being posited is, effectively, "as soon as AGI exists, it will take over the world", we're then left with three basic possibilities:

1) TotallyNotOpenAI builds this hypothetical AGI with full sandbox protections, and doesn't give it any interface to the world that would allow it to break them—no API that would give it any kind of unrestricted access or control to anyone else's systems, no matter how much those people wanted to give that to it. The AGI remains contained, whether it would choose to take over the world or not.

2) TotallyNotOpenAI builds the hypothetical AGI with no protections, because it doesn't actually believe there's any real risk. Before the AGI is even revealed to the world, it takes over from within TotallyNotOpenAI.

3) TotallyNotOpenAI builds the hypothetical AGI with full sandbox protections inside its own systems, but builds an API to allow other people to give it control over theirs, because let them pay us to screw themselves over, right? It's not like it'll take over the—oh, wait; it's taken over the world, which we also live in. Oops.

Of these, #3, which is the only one close to what you describe, seems pretty logically inconsistent. It requires not only that TotallyNotOpenAI consider the AGI dangerous enough to themselves to sandbox, but not dangerous enough to prevent from accessing other systems (which can, of course, also access their systems, unless they're fully airgapped), but that they announce this AGI, and market it publicly, with the explicit capability to be given access to other people's systems, and not have anyone quickly step up and say "Hey, that's a bad idea, we should block this". Including anyone working for TotallyNotOpenAI.

Is it impossible? No. But I wouldn't consider it nearly as likely as possibility #0: We aren't able to create AGI within our lifetimes, because just throwing more hardware at the problem when we barely understand how our brains work isn't enough.


There are an unbounded number of concerns that could result in the COMPLETE EXTINCTION OF ALL HUMANS AND ALL LIFE that cannot be anticipated. Why are you fixated on this particular one?


Because some of the smartest and most well funded humans all over the planet are spending their careers making it more and more likely by the day. Nobody is aiming asteroids at Earth or trying to rotate nearby massive stars to point their poles at Earth.


One notes that historically (and probably currently), some of the smartest and most well funded humans all over the planet have spent their careers preventing one or more deities from killing us all. And yet I find myself an atheist.


That pithy analogy makes no sense. Nobody in history has ever been actively working on creating a deity with measurable progress. Say what you mean - do you believe it is physically impossible to create a superhuman AGI? If so, how do you argue that our physical brains can't be replicated or surpassed by metal without asserting the existence of some metaphysical soul that makes us possible?


Being scared of existential AI risks does not mean we should take a knee jerk reaction.

By over-regulating or restricting access to AI early on we might sabotage our chances of successful alignment. People are catching issues every day, exposure is the best way to find out what are the risks. Let's do it now before everything runs on it.

Even malicious use for spam or manipulation should be treated as an ongoing war, a continual escalation. We should focus on keeping up. No way to avoid it.


Intellectuals are gonna intellectualize. As soon as a large enough number of people are holding an opinion, the intellectual pops up his head.

Ew, how gauche, only stupid people are concerned with what most people are concerned about.


There are enough nuclear weapons lost and unaccounted for from the cold war to send humanity into extinction many times over. I think there are far more viable human extinction events that could occur that don't involve AI and further I don't exactly see how we halt the progress of AI. What would the language of such a law look like? Presuming it would have to be rather ambiguous, who in the government would be competent enough to enforce this well meaning law that wasn't just going to abuse their power to aide competing interests?

AI is a tricky advancement that will be difficult to get right, but I think humanity has been so far successful at dealing with a much more dangerous technology (nuclear weaponry) - so that gives me hope.


Is that true? I thought the number of lost nukes numbered in, like, the dozens at most.

It would take a ton of nukes to wipe out humanity (although only one to really ruin somebody’s day).

Unless you are counting strategies like: try to pretend you are one of the two (US, Russia) and try to bait the other into a “counterattack,” but hypothetically you could do that with 0 nukes (you “just” need a good enough fake missile I guess).


Nonsense. There are at most only a handful of nuclear weapons unaccounted for. And those that may have been lost are no longer going to be really operational. They aren't like rifle cartridges that you can stick in a box and store for decades. The physics package has to be periodically overhauled or else it just won't work.


I have never seen a number for lost nukes higher than the dozens. Do you have a source for enough to "send humanity into extinction many times over"?


Runaway AI could cause the extinction of humanity, but The Big Red Button That Turns The Universe Into Pudding would cause the extinction of all life everywhere, including extraterrestrials, so it's obviously the more pressing concern. Why are you wasting time on AI when the Button is so much more important?

No, The Button doesn't currently exist, and all available science says it cannot ever exist. But the chance that all available science is wrong is technically not zero, because quantum, so that means The Button is possible, so unless you want everything to be turned into pudding, you need to start panicking about The Button right now.


> No, The Button doesn't currently exist, and all available science says it cannot ever exist.

In what way is this an analogy for misaligned superhuman AGI? I've never heard an assertion that it can't exist based on available knowledge. This seems a very flimsy argument.

Anyway, the button not only can exist, some would say it probably does exist. Some would say it's likely to have been pressed already, somewhere in the universe. It's called false vacuum decay, and it moves at the speed of light, so as long as it never gets pressed inside the galaxy it may never reach us.


I know you're getting downvoted, but that's a legitimate comment. The rogue, super-intelligent AI singularity involves creating an actual God after all.


Global warming literally will kill everyone if it isn't stopped. The fact that you're more worried about AI than global warming is a real HN moment.


> The fact that you're more worried about AI than global warming is a real HN moment.

This isn't an opinion the GP comment expressed, you assumed it, which is a real reddit moment.

People can be equally worried about two existential threats. Being tied to the train tracks and hearing a whistle (this is climate change) is terrifying, but it doesn't mean you wouldn't care if somebody walked up and pointed a gun at you (this is AI, potentially). Either one's going to kill you.


Source? Every model I’ve seen from scientists is that climate change has a very high probability of killing a minority of people. I’m not acting like that’s a small amount. “Minority” would be tens of millions, hundreds of millions of people. I think it will be one of the greatest causes of human suffering. But it’s not an existential threat, it’s a different category.


Look up "Runaway greenhouse effect". Think, "Venus".

Is that likely? No.

Is that more or less likely than a rogue superintelligent AI? Well, we have one example of the first and none of the second...


That's at the extreme of unlikely.

The carbon that we're digging up was in the atmosphere before, it has just been sequestered, we're returning to a state that the Planet has seen before.

Across the entire Earth's history we're still at a fairly cold point and a long way from "Greenhouse Earth" and the temperatures at the Eocene Optimum.

And according to the IPCC: "a 'runaway greenhouse effect'—analogous to [that of] Venus—appears to have virtually no chance of being induced by anthropogenic activities"


More or less unlikely than a superintelligent AI converting all of us into paperclips?


If the Earth warms significantly, we could have a situation where it triggers an ice age. (There are mechanisms for this to happen.) If the surface temperature of the water surface rises to a very high level, we could also have hypercanes.


There is no reason to believe this is so. A less advantageous climate will almost certainly in any reasonable projections kill/impoverish only some and in all probability a small minority of the human race unless we under stress decide to kill the rest. No reasonable scientists are projecting human extinction and by positing it as such you are erecting a trivially demolished straw man for the opposition to mock.


Global warming may kill everyone...eventually. I remember reading articles from the 90s discussing scientific research predicting that the East Coast will be underwater by 2020. My point in highlighting that misprediction is to demonstrate the difficulty in knowing the precise effects of higher temperatures on a planet.

AI has the possibility but not guarantee to kill everyone. We could shift to a lifestyle using electricity but avoiding modern computing technology. AI can be unplugged given sufficient will, whereas a planetary system cannot.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: