This reeks of marketing and a push for early regulatory capture. We already know how Sam Altman thinks AI risk should be mitigated - namely by giving OpenAI more market power. If the risk were real, these folks would be asking the US government to nationalize their companies or bring them under the same kind of control as nukes and related technologies. Instead we get some nonsense about licensing.
I'm eternally skeptical of the tech business, but I think you're jumping to conclusions, here. I'm on a first-name basis with several people near the top of this list. They are some of the smartest, savviest, most thoughtful, and most principled tech policy experts I've met. These folks default to skepticism of the tech business, champion open data, are deeply familiar with the risks of regulatory capture, and don't sign their name to any ol' open letter, especially if including their organizational affiliations. If this is a marketing ploy, that must have been a monster because even if they were walking around handing out checks for 25k I doubt they'd have gotten a good chunk of these folks.
Maybe these people have good intentions and are just being naive
They might not be getting paid, but that doesn’t mean they are not being influenced
AI at this point is pretty much completely open, all the papers, math and science behind it are public
Soon, people will have advanced AI running locally on their phones and watches
So unless they scrub the Internet, start censoring this stuff, and pretty much ban computers, there is absolutely no way to stop AI nor any potentially bad actors from using it
The biggest issues that we should be addressing regarding AI are the potential jobs losses and increased inequality at local and global scale
But of course, the people who usually make these decisions are the ones that benefit the most from inequality, so
You are attributing naivety on a technical matter to many people who have done world class research and some of them are even a pioneer of their subfield of study.
To do that level of research, one needs strong independent thinking skills and deep technical expertise. We have plenty of evidence on this very site how hard it is to influence technical people and change their opinion.
One does not need to be have super-human capabilities to foresee potential risks of the technology one is an expert in.
Yes, the current version of AI is not capable of large scale harm by itself yet, but the plausible trajectory is worth warning about. Gordon Moore did make fairly accurate predictions after all.
There's range of functionality in software technology, i.e. some new framework, may have been outputed fully developed and internally debugged for years in some startup, or you could have a really almost not even a production level software, just uploaded to github.
In most projects, many unmatured software is being used just because works for some specific task, nowadays.
With that in mind, it is easy to hyphotetize that many projects are already using LLMs internally. Not always for good deeds, easily, one of the most and best use cases, is to use LLMs to command & control distributed malware.
So there you go, it maybe possible that some beta level of intelligent malware is already roaming the Internet right now. We'll know for sure in some years from now (the usual time for advanced malware to be discovered is somewhat 2-3 years after it has been took it to production).
>Maybe these people have good intentions and are just being naive
Ive noticed a lot of good people take awful political positions this way.
Usually they trust the wrong person - e.g. by falling victim to the just world fallacy ("X is a big deal in our world and X wouldn't be where they are if they werent a decent person. X must have a point.")
It’s worth noting also that many academics who signed the statement may face adverse issues like reputational risk as well as funding cut to their research programs if AI safety becomes an official policy.
For a large number of them, these risks are worth far more than any possible gain from signing it.
When a large number of smart, reputable people, including many with expert knowledge and little or negative incentives to act dishonestly, put their names down like this, one should pay attention.
Added:
Paul Christiano, a brilliant theoretical CS researcher who switched to AI Alignment several years ago, put the risks of “doom” for humanity at 46%.
Subtract OpenAI, Google, StabilityAI and Anthropic affiliated researchers (who have a lot to gain) and not many academic signatories are left.
Notably missing representation from the Stanford NLP (edit: I missed that Diyi Yang is a signatory on first read) and NYU groups who’s perspective I’d also be interested in hearing.
Not committing one way or another regarding the intent with this but it’s not as diverse an academic crowd as the long list may suggest and for a lot of these names there are incentives to act dishonestly (not claiming that they are).
Even if it’s just Yoshua Bengio, Geoffrey Hinton, and Stuart Russell, we’d probably agree the risks are not negligible. There are quite a few researchers from Stanford, UC Berkeley, MIT, Carnegie Mellon, Oxford, Cambirdge, Imperial College, Edinburg, Tsinghua, etc who signed as well. Many of whom do not work for those companies.
We’re talking about nuclear war level risks here. Even a 1% chance should definitely be addressed. As noted above, Paul Christiano who has worked on AI risk and thought about it for a long time put it at 46%.
> There are quite a few researchers from Stanford, UC Berkeley, MIT, Carnegie Mellon, Oxford, Cambirdge, Imperial College, Edinburg, Tsinghua, etc who signed as well.
I know the Stanford researchers the most and the “biggest names” in LLMs from HAI and CRFM are absent. It would be useful to have their perspective as well.
I’d throw MetaAI in the mix as well.
Merely pointing out that healthy skepticism here is not entirely unwarranted.
> We’re talking about nuclear war level risks here.
LLM is already a misnomer. Latest versions are multimodal. Current versions can be used to build agents with limited autonomy. Future versions of LLMs are most likely capable of more independence.
Even dumb viruses have caused catastrophic harm. Why? It’s capable of rapid self replication in a massive number of existing vessels. You add in some intelligence, vast store of knowledge, huge bandwidth, and some aid by malicious human actors, what could such a group of future autonomous agents do?
I mean a small group of malicious humans can already bioengineer a deadly virus with CRISPR and open source tech without AI.
This is hardly the first time in history a new technological advancement may be used for nefarious purposes.
It’s a discussion worth having as AI advances but if [insert evil actor] wants to cause harm there are many cheaper and easier ways to do this right now.
To come out and say we need government regulation today does stink at least a little bit of protectionism as practically speaking the “most evil actors” would not adhere to whatever is being proposed, but this would impact the competitive landscape and the corporations yelling the loudest right now have the most to gain, perhaps coincidence but worth questioning.
> I mean a small group of malicious humans can already bioengineer a deadly virus with CRISPR and open source tech without AI.
That's what interesting to me. People fearmongering about bioengineering and GMO's were generally dismissed as being anti-science and holding humankind back (or worse, that there opposition to progress meant they had blood on their hands). Yet many of the people who mocked them proved themselves to be even more dogmatic and apocalyptic, while being much closer to influencing regulations. And the technology they're fear-mongering about is even further from being able to harm people than biotech is. We are actually able to create harmful biotech today if we want; we don't know when we'll ever be able to create AGI, and if it would even pose a danger if we did.
This mentality - "there could be a slight chance research into this could eventually lead to apocalyptic technology, no I don't have any idea how but the danger is so great we need a lot of regulation" - would severely harm scientific growth if we applied it consistently. Of course everyone is going to say "the technology I'm afraid of is _actually_ dangerous, the technology they're afraid if isn't." But we honestly have no clue when we're talking about technology that we have no idea how to create at the moment.
Counterpoint: CRISPR only reignited what was already a real fear of reduced difficulty and costs of engineering deadly pathogens.
In fact, what you and GP wrote is baffling to me. The way I see it, biotech is stupidly obviously self-evidently dangerous, because let's look at the facts:
- Genetic engineering gets easier and cheaper and more "democratized"; in the last 10 years, the basics were already accessible to motivated schools and individual hobbyists;
- We already know enough, with knowledge accessible at the hobbyist level, to know how to mix and match stuff and get creative - see "synthetic biology";
- The substrate we're working with is self-replicating molecular nanotechnology; more than that, it's usually exactly the type that makes people get sick - bacteria (because they're most versatile nanobots) and viruses (because they're natural code injection systems).
Above is the "inside view"; for "outside view", I'll just say this: the fact that "lab leak" hypothesis of COVID-19 was (or still is?) considered to be one of the most likely explanations for the pandemics already tells you that the threat is real, and consequences are dire.
I don't know how can you possibly look at that and conclude "nah, not dangerous, needs to be democratized so the Evil Elites don't hoard it all".
There must be some kind of inverse "just world fallacy" fallacy of blaming everything on evil elites and 1%-ers that are Out To Get Us. Or maybe it's just another flavor of the NWO conspiracy thinking, except instead the Bildenbergs and the Jews its Musk, Bezos and the tech companies.
Same is, IMHO, with AI. Except that one is more dangerous because it's a technology-using technology - that is, where e.g. accidentally or intentionally engineered pathogens could destroy civilization directly, AI could do it by using engineered pathogens - or nukes, or mass manipulation, or targeted manipulation, or ... countless other things.
EDIT:
And if you ask "why, if it's really so easy to access and dangerous, we haven't already been killed by engineered pathogens?", the answer is a combination of:
1. vast majority of people not bearing ill intent;
2. vast majority of people being not interested and not able to perform (yet!) this kind of "nerdy thing";
3. a lot of policing and regulatory attention given to laboratories and companies playing with anything that could self-replicate and spread rapidly;
4. well-developed policies and capacity for dealing with bio threats (read: infectious diseases, and diseases in general);
5. this being still new enough that the dangerous and the careless don't have an easy way to do what in theory they already could.
Note that despite 4. (and 3., if you consider "lab leak" a likely possibility), COVID-19 almost brought the world down.
Great points. Will just add a point 1.5: There's usually an inverse correlation between ill intent and competence, so the subset of people who both want to cause harm to others on a mass scale and who are also able to pull it off is small
I’m not sure there is a way for someone to engineer a deadly virus while completely innoculating themselves from it.
Short-term AI risk likely comes from a mix of malicious intent and further autonomy that causes harm the perpetrators did not expect. In the longer run, there is a good chance of real autonomy and completely unexpected behaviors from AI.
Why do you have to inoculate yourself from it to create havoc? Your analogy of “nuclear war” also has no vaccine.
AI autonomy is a hypothetical existential risk, especially in the short term. There are many non-hypothetical existential risks including actual nuclear proliferation and escalating great power conflicts happening right now.
Again my point being that this is an important discussion but appears overly dramatized, just like there are people screaming doomsday there are also equally qualified people (like Yann LeCun) screaming BS.
But let’s entertain this for a second, can you posit a hypothetical where in the short term a nefarious actor can abuse AI or autonomy results in harm? How does this compare to non-AI alternatives for causing harm?
This gets countered by running one (or more) of those same amazing autonomous agents locally for your own defense. Everyone's machine is about to get much more intelligent.
I used LLM because the people shouting the loudest come from a LLM company which claimed their newest language model can be used to create bioweapons in their whitepaper.
Semantics aside the recent interest in AI risk was clearly stimulated by LLMs and the camp that believes this is the path to AGI which may or may not be true depending who you ask.
I can only imagine Eleizer Yudkowsky and Rob Miles looking on this conversation with a depressed scream and a facepalm respectively.
They've both been loudly concerned about optimisers doing over-optimisation, and society having a Nash equilibrium where everyone's using them as hard as possible regardless of errors, since before it was cool.
To the extent that this may be true (I've not exactly studied which public thinkpiece writers care about which AI so it might easily be the image generators that get this crown for all I know), that's because ChatGPT actually does something that a normal person understands.
A paper titled "Dual use of artificial-intelligence-powered drug discovery" (published last year) got a few angst pieces and is mostly forgotten by the general public and media so far as I can tell; but the people behind it both talked directly to regulators and other labs to help advise them how many other precursor chemicals were now potential threats, and also went onto the usual podcasts and other public forums to raise awareness of the risk to other AI researchers.
The story behind that was "ugh, they want us to think about risks… what if we ask it to find dangerous chemicals instead of safe ones? *overnight* oh no!"
> I can only imagine Eleizer Yudkowsky and Rob Miles looking on this conversation with a depressed scream and a facepalm respectively.
Whenever Yudkowsky comes up on my Twitter feed I'm left with an impression that I'm not going to have any more luck conversing AI with those in his orbit than I am discussing the rapture with a fundamentalist Christian. For example, the following Tweet[1]. If a person believes this is from a deep thinker that should be taken very seriously rather than an unhinged nutcase, our worldviews are probably too far apart to ever reach a common understanding:
> Fools often misrepresent me as saying that superintelligence can do anything because magic. To clearly show this false, here's a concrete list of stuff I expect superintelligence can or can't do:
> - FTL (faster than light) travel: DEFINITE NO
> - Find some hack for going >50 OOM past the amount of computation that naive calculations of available negentropy would suggest is possible within our local volume: PROBABLE NO
> - Validly prove in first-order arithmetic that 1 + 1 = 5: DEFINITE NO
> - Prove a contradiction from Zermelo-Frankel set theory: PROBABLE NO
> - Using current human technology, synthesize a normal virus (meaning it has to reproduce itself inside human cells and is built of conventional bio materials) that infects over 50% of the world population within a month: YES
> (note, this is not meant as an argument, this is meant as a concrete counterexample to people who claim 'lol doomers think AI can do anything just because its smart' showing that I rather have some particular model of what I roughly wildly guess to be a superintelligence's capability level)
> - Using current human technology, synthesize a normal virus that infects 90% of Earth within an hour: NO
> - Write a secure operating system on the first try, zero errors, no debugging phase, assuming away Meltdown-style hardware vulnerabilities in the chips: DEFINITE YES
> - Write a secure operating system for actual modern hardware, on the first pass: YES
> - Train an AI system with capability at least equivalent to GPT-4, from the same dataset GPT-4 used, starting from at most 50K of Python code, using 1000x less compute than was used to train GPT-4: YES
> - Starting from current human tech, bootstrap to nanotechnology in a week: YES
> - Starting from current human tech, bootstrap to nanotechnology in an hour: GOSH WOW IDK, I DON'T ACTUALLY KNOW HOW, BUT DON'T WANT TO CLAIM I CAN SEE ALL PATHWAYS, THIS ONE IS REALLY HARD FOR ME TO CALL, BRAIN LEGIT DOESN'T FEEL GOOD BETTING EITHER WAY, CALL IT 50:50??
> - Starting from current human tech and from the inside of a computer, bootstrap to nanotechnology in a minute: PROBABLE NO, EVEN IF A MINUTE IS LIKE 20 SUBJECTIVE YEARS TO THE SI
> - Bootstrap to nanotechnology via a clean called shot: all the molecular interactions go as predicted the first time, no error-correction rounds needed: PROBABLY YES but please note this is not any kind of necessary assumption because It could just build Its own fucking lab, get back the observations, and do a debugging round; and none of the processes there intrinsically need to run at the speed of humans taking hourly bathroom breaks, it can happen at the speed of protein chemistry and electronics. Please consider asking for 6 seconds how a superintelligence might possibly overcome such incredible obstacles of 'I think you need a positive nonzero number of observations', for example, by doing a few observations, and then further asking yourself if those observations absolutely have to be slow like a sloth
> - Bootstrap to nanotechnology by any means including a non-called shot where the SI designs more possible proteins than It needs to handle some of the less certain cases, and gets back some preliminary observations about how they interacted in a liquid medium, before it actually puts together the wetware lab on round 2: YES
(The Tweet goes on, you can read the rest of it at the link below, but that should give you the gist.)
I don't have twitter and I agree his tweets have an aura of lunacy, which is a shame as he's quite a lot better as a long-form writer. (Though I will assume his long-form writings about quantum mechanics is as bad as everyone else unless a physicist vouches for them).
But, despite that, I don't understand why you chose that specific example — how is giving a list of what he thinks an AI probably can and can't do, in the context of trying to reduce risks because he thinks loosing is the default, similar to a fundamentalist Christian who wants to immanentize the eschaton because the idea the good guys might lose when God is on their side is genuinely beyond comprehension?
Some of the academics who signed are either not doing AI research e.g climatologists, genomics, philosophy. Or they have Google connections that aren't disclosed. E.g. Peter Norvig is listed as Stanford University but ran Google Research for many years, McIlrath is associated with the Vector Institute which is funded by Google.
I just took that list and separated everyone that had any commercial tie listed, regardless of the company. 35 did and 63 did not.
> "Subtract OpenAI, Google, StabilityAI and Anthropic affiliated researchers (who have a lot to gain) and not many academic signatories are left."
You're putting a lot of effort into painting this list in a bad light without any specific criticism or evidence of malfeasance. Frankly, it sounds like FUD to me.
With corporate conflicts (that I recognized the names of):
Yoshua Bengio: Professor of Computer Science, U. Montreal / Mila,
Victoria Krakovna: Research Scientist, Google DeepMind,
Mary Phuong: Research Scientist, Google DeepMind,
Daniela Amodei: President, Anthropic,
Samuel R. Bowman: Associate Professor of Computer Science, NYU and Anthropic,
Helen King: Senior Director of Responsibility & Strategic Advisor to Research, Google DeepMind,
Mustafa Suleyman: CEO, Inflection AI,
Emad Mostaque: CEO, Stability AI,
Ian Goodfellow: Principal Scientist, Google DeepMind,
Kevin Scott: CTO, Microsoft,
Eric Horvitz: Chief Scientific Officer, Microsoft,
Mira Murati: CTO, OpenAI,
James Manyika: SVP, Research, Technology & Society, Google-Alphabet,
Demis Hassabis: CEO, Google DeepMind,
Ilya Sutskever: Co-Founder and Chief Scientist, OpenAI,
Sam Altman: CEO, OpenAI,
Dario Amodei: CEO, Anthropic,
Shane Legg: Chief AGI Scientist and Co-Founder, Google DeepMind,
John Schulman: Co-Founder, OpenAI,
Jaan Tallinn: Co-Founder of Skype,
Adam D'Angelo: CEO, Quora, and board member, OpenAI,
Simon Last: Cofounder & CTO, Notion,
Dustin Moskovitz: Co-founder & CEO, Asana,
Miles Brundage: Head of Policy Research, OpenAI,
Allan Dafoe: AGI Strategy and Governance Team Lead, Google DeepMind,
Jade Leung: Governance Lead, OpenAI,
Jared Kaplan: Co-Founder, Anthropic,
Chris Olah: Co-Founder, Anthropic,
Ryota Kanai: CEO, Araya, Inc.,
Clare Lyle: Research Scientist, Google DeepMind,
Marc Warner: CEO, Faculty,
Noah Fiedel: Director, Research & Engineering, Google DeepMind,
David Silver: Professor of Computer Science, Google DeepMind and UCL,
Lila Ibrahim: COO, Google DeepMind,
Marian Rogers Croak: VP Center for Responsible AI and Human Centered Technology, Google
Without:
Geoffrey Hinton: Emeritus Professor of Computer Science, University of Toronto,
Dawn Song: Professor of Computer Science, UC Berkeley,
Ya-Qin Zhang: Professor and Dean, AIR, Tsinghua University,
Martin Hellman: Professor Emeritus of Electrical Engineering, Stanford,
Yi Zeng: Professor and Director of Brain-inspired Cognitive AI Lab, Institute of Automation, Chinese Academy of Sciences,
Xianyuan Zhan: Assistant Professor, Tsinghua University,
Anca Dragan: Associate Professor of Computer Science, UC Berkeley,
Bill McKibben: Schumann Distinguished Scholar, Middlebury College,
Alan Robock: Distinguished Professor of Climate Science, Rutgers University,
Angela Kane: Vice President, International Institute for Peace, Vienna; former UN High Representative for Disarmament Affairs,
Audrey Tang: Minister of Digital Affairs and Chair of National Institute of Cyber Security,
Stuart Russell: Professor of Computer Science, UC Berkeley,
Andrew Barto: Professor Emeritus, University of Massachusetts,
Jaime Fernández Fisac: Assistant Professor of Electrical and Computer Engineering, Princeton University,
Diyi Yang: Assistant Professor, Stanford University,
Gillian Hadfield: Professor, CIFAR AI Chair, University of Toronto, Vector Institute for AI,
Laurence Tribe: University Professor Emeritus, Harvard University,
Pattie Maes: Professor, Massachusetts Institute of Technology - Media Lab,
Peter Norvig: Education Fellow, Stanford University,
Atoosa Kasirzadeh: Assistant Professor, University of Edinburgh, Alan Turing Institute,
Erik Brynjolfsson: Professor and Senior Fellow, Stanford Institute for Human-Centered AI,
Kersti Kaljulaid: Former President of the Republic of Estonia,
David Haussler: Professor and Director of the Genomics Institute, UC Santa Cruz,
Stephen Luby: Professor of Medicine (Infectious Diseases), Stanford University,
Ju Li: Professor of Nuclear Science and Engineering and Professor of Materials Science and Engineering, Massachusetts Institute of Technology,
David Chalmers: Professor of Philosophy, New York University,
Daniel Dennett: Emeritus Professor of Philosophy, Tufts University,
Peter Railton: Professor of Philosophy at University of Michigan, Ann Arbor,
Sheila McIlraith: Professor of Computer Science, University of Toronto,
Lex Fridman: Research Scientist, MIT,
Sharon Li: Assistant Professor of Computer Science, University of Wisconsin Madison,
Phillip Isola: Associate Professor of Electrical Engineering and Computer Science, MIT,
David Krueger: Assistant Professor of Computer Science, University of Cambridge,
Jacob Steinhardt: Assistant Professor of Computer Science, UC Berkeley,
Martin Rees: Professor of Physics, Cambridge University,
He He: Assistant Professor of Computer Science and Data Science, New York University,
David McAllester: Professor of Computer Science, TTIC,
Vincent Conitzer: Professor of Computer Science, Carnegie Mellon University and University of Oxford,
Bart Selman: Professor of Computer Science, Cornell University,
Michael Wellman: Professor and Chair of Computer Science & Engineering, University of Michigan,
Jinwoo Shin: KAIST Endowed Chair Professor, Korea Advanced Institute of Science and Technology,
Dae-Shik Kim: Professor of Electrical Engineering, Korea Advanced Institute of Science and Technology (KAIST),
Frank Hutter: Professor of Machine Learning, Head of ELLIS Unit, University of Freiburg,
Scott Aaronson: Schlumberger Chair of Computer Science, University of Texas at Austin,
Max Tegmark: Professor, MIT, Center for AI and Fundamental Interactions,
Bruce Schneier: Lecturer, Harvard Kennedy School,
Martha Minow: Professor, Harvard Law School,
Gabriella Blum: Professor of Human Rights and Humanitarian Law, Harvard Law,
Kevin Esvelt: Associate Professor of Biology, MIT,
Edward Wittenstein: Executive Director, International Security Studies, Yale Jackson School of Global Affairs, Yale University,
Karina Vold: Assistant Professor, University of Toronto,
Victor Veitch: Assistant Professor of Data Science and Statistics, University of Chicago,
Dylan Hadfield-Menell: Assistant Professor of Computer Science, MIT,
Mengye Ren: Assistant Professor of Computer Science, New York University,
Shiri Dori-Hacohen: Assistant Professor of Computer Science, University of Connecticut,
Jess Whittlestone: Head of AI Policy, Centre for Long-Term Resilience,
Sarah Kreps: John L. Wetherill Professor and Director of the Tech Policy Institute, Cornell University,
Andrew Revkin: Director, Initiative on Communication & Sustainability, Columbia University - Climate School,
Carl Robichaud: Program Officer (Nuclear Weapons), Longview Philanthropy,
Leonid Chindelevitch: Lecturer in Infectious Disease Epidemiology, Imperial College London,
Nicholas Dirks: President, The New York Academy of Sciences,
Tim G. J. Rudner: Assistant Professor and Faculty Fellow, New York University,
Jakob Foerster: Associate Professor of Engineering Science, University of Oxford,
Michael Osborne: Professor of Machine Learning, University of Oxford,
Marina Jirotka: Professor of Human Centred Computing, University of Oxford
So the most “notable” AI scientists on this list have clear corporate conflicts. Some are more subtle:
> Geoffrey Hinton: Emeritus Professor of Computer Science, University of Toronto,
He’s affiliated with Vector (as well as some of the other Canadians on this list) and was at Google until very recently (unsure if he retained equity which would require disclosure in academia).
Hence my interest in disclosures as the conflicts are not always obvious.
How is saying that they should have disclosed a conflict that they did not disclose not accusatory? If that's the case, the accusation is entirely justified and should be surfaced! The other signatories would certainly want to know if they were signing in good faith when others weren't. This is what I need interns for.
I never said “they should have disclosed a conflict they did not disclose.”
Disclosures are absent from this initiative, some signatories have self-identified their affiliation by their own volition and even for those it is not in the context of a conflict disclosure.
There is no “signatories have no relevant disclosures” statement for those who did not for the omission to be malfeasance and pointing out the absence of a disclosure statement is not accusatory of the individuals, rather that the initiative is not transparent about potential conflicts.
Once again, it is standard practice in academia to make a disclosure statement if lecturing or publishing. While it is not mandatory for initiatives calling for regulation it would be nice to have.
I'd guess that a given academic isn't going to face much of a career risk for signing a statement also signed by other very prestigious academics, just the opposite. There's no part of very divided US political spectrum that I can see denouncing AI naysayers, unlike the scientists who signed anti-nuclear statements in 1960s or even people warning about global warming now (indeed, I'd guess the statement doesn't mention climate change 'cause it's still a sore point).
Moreover, talking about existential risk involves the assumption the current tech is going to continue to affect more and more fields rather than peaking at some point - this assumption guarantees more funding along with funding for risk.
All that said, I don't necessarily think the scientists involved are insincere. Rather, I would expect they're worried and signed this vague statement because it was something that might get traction. While the companies indeed may be "genuine" in the sense they're vaguely [concerned - edit] and also self-serving - "here's a hard problem it's important to have us wise, smart people in charge of and profiting from"
In interviews, Geoffrey Hinton and Yoshua Bengio certainly expressed serious concerns and even some plausible regret to their life’s work. They did not say anything that can be interpreted as your last sentence suggests at all.
My last sentence currently: "While the companies indeed may be "genuine" in the sense they're vaguely and also self-serving - "here's a hard problem it's important to have us wise, smart people in charge of and profiting from" - IE, I am not referring to the academics there.
I'm going to edit the sentence to fill in some missing words but I don't think this will change the meaning involved.
On the contrary, I suspect "How do we prevent our AIs from killing everyone?" will be a major research question with a great deal of funding involved. Plus, no one seems to be suggesting things like the medical ethics field or institutional review boards, which might have deleterious impacts on their work.
46% is such a precise number. This is why I can't take "rationalists", the Yudkowskys, and the Silvers seriously. Colossal assumptions turned into confidently stated probabilities.
You're putting a weirdly large amount of trust into, functionally, some dude who posted on lesswrong. Sure he has a PhD and is smart, but so is basically everyone else in the field, not just in alignment, and the median person in the field thinks the risk of "doom" is 2-5% (and that's conditioned on the supposed existence of a high level machine intelligence that the median expert believes might exist in 40 years). That still might be higher than you'd like, but it's not actually a huge worry in the grand scheme of things.
Like, if I told you that in 40 years, there was a 50% chance of something existing that had a 2% chance of causing extreme harm to the human population, I'm actually not sure that thing should be the biggest priority. Other issues may have more than a 1% chance of leading to terrible outcomes sooner.
The median person in field thinks 5-10%, not 2-5%, and median timelines are shorter than 40 years.
But this is all a distraction, since unaligned ASI is the default case absent significant efforts (that we aren't currently making), and trying to evaluate risk by averaging out the views of people who haven't actually explored the arguments very carefully (vs. just evaluating the arguments yourself) is a doomed effort.
> The median person in field thinks 5-10%, not 2-5%
The median person in the study here, under a particular definition was 5-10%, other comparable studies have found 2%, and similar questions using arguably better definitions in the same study found lower percentages.
> median timelines are shorter than 40 years.
The median person suggested a 50% chance in 39 years.
> since unaligned ASI is the default case
I challenge this assertion. Many relatively smart scholars who are more involved in the alignment space than, presumably either you or I, have put forth cogent arguments that alignment-by-default is perfectly reasonable. Dismissing those out of hand seems naive.
I work in the space (doing tech stuff that isn't direct research). The best argument I've seen for alignment by default is something like "morality comes from training data, and therefore the fact that LLM training data sets contain human moral intuitions will mean that an ASI stemming from such a training regime will share enough human values" (Quintin Pope believes something like this, as far as I can tell), which is deeply unconvincing, since it contradicts the evidence we _do_ have from human & animal value formation.
Happy to entertain other arguments that alignment-by-default is reasonable; most arguments I've seen are much worse than that one. I haven't seen many people make an active case for alignment-by-default, so much as leave open a whole bunch of swath of uncertainty for unknown unknowns.
They didn’t become such a wealthy group by letting competition foster. I have no doubt they believe they could be doing the right thing but I also have no doubt they don’t want other people making the rules.
Truth be told, who else really does have a seat at the table for dictating such massive societal change? Do you think the copy editor union gets to sit down and say “I’d rather not have my lunch eaten, I need to pay my rent. Let’s pause AI usage in text for 10 years.”
These competitors banded together and put out a statement to get ahead of any one else doing the same thing.
That doesn’t erase the need of cynicism. Many people in academia come from industry, have friends in industry, or other stakes. They might have been persuaded by the rhetoric of stakeholders within industry (you saw this early in the climate debate; and still do), and they might also be hoping to get a job in the industry later on. There is also a fair amount of group think within academia, so if a prominent individual inside academia believes the lies of industry, chances are the majority within the department does.
Quite often these intersect. But also it is healthy to question the motives of signaturees asking for public policy. Especially when a significant number of them are direct stakeholders. When an academic signs such a list, their motives should be equally questioned.
The people I know on the list are academics and do not seem to be any wealthier than other academics I know. I'm quite certain the private industry signatories are going to entirely advocate for their interest just as they do in any other policy discussion.
Here’s why AI risks are real, even if our most advanced AI is merely a ‘language’ model:
Language can represent thoughts and some world models. There is strong evidence that LLMs contain some representation of world models it learned from text. Moreover, LLM is already a misnomer; latest versions are multimodal. Current versions can be used to build agents with limited autonomy. Future versions of LLMs are most likely capable of more independence.
Even dumb viruses have caused catastrophic harm. Why? It’s capable of rapid self replication in a massive number of existing vessels. You add in some intelligence, vast store of knowledge, huge bandwidth, and some aid by malicious human actors, what could such a group of future autonomous agents do?
A lot of things are called "world models" that I would consider just "models" so it depends on what you mean by that. But what do you consider to be strong evidence? The Othello paper isn't what I'd call strong evidence.
I agree that the Othello paper isn't, and couldn't be, strong evidence about what sort of model of the world (if any) something like GPT-4 has. However, I think it is (importantly) pretty much a refutation of all claims along the lines of "these systems learn only from text, therefore they cannot have anything in them that actually models anything other than text", since their model learned only from text and seems to have developed something very much like a model of the state of the game.
Again, it doesn't say much about how good a model any given system might have. The world is much more complicated than an Othello board. GPT-4 is much bigger than their transformer model. Everything they found is consistent with anything from "as it happens GPT-4 has no world model at all" through to "GPT-4 has a rich model of the world, fully comparable to ours". (I would bet heavily on the truth being somewhere in between, not that that says very much.)
You may be right, I don't know the people involved on a personal basis. Perhaps my problem is how much is left unsaid here (the broader safe.ai site doesn't help
much). For example, what does "mitigate" mean? The most prominent recent proposal for mitigation comes from Sam Altman's congressional testimony, and it's very self serving. In such a vacuum of information, it's easy to be cynical.
Right. It probably needed to be general because there hasn't been enough time to work out sane specific responses, and even if they had, getting buy-in on specifics is a recipe for paralysis by indecision. A credible group of people simply pleading for policy makers, researchers, et. al. to take this seriously will lead to the project approvals, grant money, etc. that will hopefully yield a more sophisticated understanding of these issues.
Cynicism is understandable in this ever-expanding whirlpool of bullshit, but when something looks like it has potential, we need to vigorously interrogate our cynicism if we're to stand a chance at fighting it.
Reading the comments here is helping evolve my thinking on the issue for sure. Here's a comment I made in another thread:
> As I mentioned in another comment, the listed risks are also notable because they largely omit economic risk. Something that will be especially acutely felt by those being laid off in favor of AI substitutes. I would argue that 30% unemployment is at least as much of a risk to the stability of society as AI generated misinformation.
> If one were particularly cynical, one could say that this is an attempt to frame AI risk in a manner that still allows AI companies to capture all the economic benefits of AI technology without consideration for those displaced by AI.
If policymaker's understanding of AI is predicated on hypothetical scenarios like "Weaponization" or "Power-Seeking Behavior" and not on concrete economic disruptions that AI will be causing very soon, the policy they come up with will be inadequate. Thus I'm frustrated with the framing of the issue that safe.ai is presenting because it is a distraction from the very real societal consequences of automating labor to the extent that will soon be possible.
My own bit of cynicism is that regulating the negative impacts of technology on workforce segments in the US is a non-starter if you approach it from the technology-end of the issue rather than the social safety net end. Most of these automation waves that plunged entire employment categories and large metropolitan areas into oblivion were a net gain for the economy even if it was concentrated at the top. I think the government will temporarily socialize the costs of the corporate profit with stimulus payments, extended unemployment benefits, and any other thing they can do to hold people over until there's a comparatively small risk of triggering real social change. Then they just blame it on the individuals.
Agreed. To pile on: one true existential risk is the continued economic/social polarization of the US and other large nation states (not to mention climate changes).
We seem perfectly resigned to the expansion of an already huge underclass—-with or without the help of LLMs.
Might not AGI help convince our tech savvy aristocracy that one fundamental problem still is better balancing of opportunities and more equitable access to a good education? I see the probability of that happening as precisely 4.6%.
This particular statement really doesn't seem like a marketing ploy. It is difficult to disagree with the potential political and societal impacts of large language models as outlined here: https://www.safe.ai/ai-risk
These are, for the most part, obvious applications of a technology that exists right now but is not widely available yet.
The problem with every discussion around this issue is that there are other statements on "the existential risk of AI" out there that are either marketing ploys or science fiction. It doesn't help that some of the proposed "solutions" are clear attempts at regulatory capture.
This muddles the waters enough that it's difficult to have a productive discussion on how we could mitigate the real risk of, e.g., AI generated disinformation campaigns.
> The problem with every discussion around this issue is that there are other statements on
Sure, but we're not talking about those other ones. Dismissing good faith initiatives as marketing ploys because there are bad faith initiatives is functionally no different than just shrugging and walking away.
Of course OpenAI et. al. will try to influence the good faith discussions: that's a great reason to champion the ones with a bunch of good faith actors who stand a chance of holding the industry and policy makers to task. Waiting around for some group of experts that has enough clout to do something, but by policy excludes the industry itself and starry-eyed shithead "journalists" trying to ride the wave of the next big thing will yield nothing. This is a great example of perfect being the enemy of good.
There's definitely a lot of marketing bullshit out there in the form of legit discussion. Unfortunately, this technology likely means there will be an incalculable increase in the amount of bullshit out there. Blerg.
Aside from emergent behavior, are any of the items on that list unique to AI? They sure don’t seem it; they’re either broadly applicable to a number of already-available technologies, or to any entity in charge or providing advice or making decisions. I dare say even emergent behavior falls under this as well, since people can develop their own new motives that others don’t understand. Their advisory doesn’t seem to amount to much more than “bad people can do bad things”, except now “people” is “AI”.
As I mentioned in another comment, the listed risks are also notable because they largely omit economic risk. Something what will be especially acutely felt by those being laid off in favor of AI substitutes. I would argue that 30% unemployment is at least as much of a risk to the stability of society as AI generated misinformation.
If one were particularly cynical, one could say that this is an attempt to frame AI risk in a manner that still allows AI companies to capture all the economic benefits of AI technology without consideration for those displaced by AI.
> It is difficult to disagree with the potential political and societal impacts of large language models as outlined here
Is it? Unless you mean something mundane like "there will be impact", the list of risks they're proposing are subjective and debatable at best, irritatingly naive at worst. Their list of risks are:
1. Weaponization. Did we forget about Ukraine already? Answer: Weapons are needed. Why is this AI risk and not computer risk anyway?
2. Misinformation. Already a catastrophic problem just from journalists and academics. Most of the reporting on misinformation is itself misinformation. Look at the Durham report for an example, or anything that happened during COVID, or the long history of failed predictions that were presented to the public as certain. Answer: Not an AI risk, a human risk.
3. People might click on things that don't "improve their well being". Answer: how we choose to waste our free time on YouTube is not your concern, and you being in charge wouldn't improve our wellbeing anyway.
4. Technology might make us fat, like in WALL-E. Answer: it already happened, not having to break rocks with bigger rocks all day is nice, this is not an AI risk.
5. "Highly competent systems could give small groups of people a tremendous amount of power, leading to a lock-in of oppressive systems". Answer: already happens, just look at how much censorship big tech engages in these days. AI might make this more effective, but if that's their beef they should be campaigning against Google and Facebook.
6. Sudden emergent skills might take people by surprise. Answer: read the paper that shows the idea of emergent skills is AI researchers fooling themselves.
7. "It may be more efficient to gain human approval through deception than to earn human approval legitimately". No shit Sherlock, welcome to Earth. This is why labelling anyone who expresses skepticism about anything as a Denier™ is a bad idea! Answer: not an AI risk. If they want to promote critical thinking there are lots of ways to do that unrelated to AI.
8. Machines smarter than us might try to take over the world. Proof by Vladimir Putin is provided, except that it makes no sense because he's arguing that AI will be a tool that lets humans take over the world and this point is about the opposite. Answer: people with very high IQs have been around for a long time and as of yet have not proven able to take over the world or even especially interested in doing so.
None of the risks they present is compelling to me personally, and I'm sure that's true of plenty of other people as well. Fix the human generated misinformation campaigns first, then worry about hypothetical non-existing AI generated campaigns.
I appreciate your perspective, but the thing that is missing is the speed at which AI has evolved, seemingly overnight.
With crypto, self-driving cars, computers, the internet or just about any other technology, development and distribution happened over decades.
With AI, there’s a risk that the pace of change and adoption could be too fast to be able to respond or adapt at a societal level.
The rebuttals to each of the issues in your comment are valid, but most (all?) of the counter examples are ones that took a long time to occur, which provided ample time for people to prepare and adapt. E.g. “technology making us fat” happened over multiple decades, not over the span of a few months.
Either way, I think it’s good to see people proactive about managing risk of new technologies. Governments and businesses are usually terrible at fixing problems that haven’t manifested yet… so it’s great to see some people sounding the alarms before any damage is done.
Note: I personally think there’s a high chance AI is extremely overhyped and that none of this will matter in a few years. But even so, I’d rather see organizations being proactive with risk management rather than reacting too the problem when it’s too late.
It may seem overnight if you weren't following it, but I've followed AI progress for a long time now. I was reading the Facebook bAbI test paper in 2015:
There's been a lot of progress since then, but it's also nearly 10 years later. Progress isn't actually instant or overnight. It's just that OpenAI spent a ton of money to scale it up then stuck an accessible chat interface on top of tech that was previously being mostly ignored.
>They are some of the smartest, savviest, most thoughtful, and most principled tech policy experts I've met.
with all due respect, that's just <Your> POV of them or how they chose to present themselves to you.
They could all be narcissists for all we know. Further, One person's opinion, namely yours, doesn't exempt them from criticism and rushing to be among the first in what's arguably the new gold rush.
The only name in this list that gives me pause is Daniel Dennett. At least he has a philosopher’s historical perspective on the challenges ahead.
Very few neuroscientists as signers. Perhaps they know better.
For me the basic sociopolitical climate today (and for the last 120 years) is at an existential boil. This is not to say that birthing AGIs is not frightening, but just to say that many real features of today are just as, or even more frightening.
> even if they were walking around handing out checks for 25k I doubt they'd have gotten a good chunk of these folks.
if the corporations in question get world governments to line up the way they want, the checks for everyone in these "letters" will be *way* bigger than 25k and they won't have "payment for signing our letter" in the memo either.
Be aware that the things that AI challenges most is knowledge work. No food delivery boy job is being challenged here (who cares about these people anyway?) but if you are a software developer the clock on that is ticking.
First, there are actual worries by a good chunk of the researchers. From runaway-paperclip AGIs to simply unbounded disinformation, I think there are a lot of scenarios that disinterested researchers and engineers worry about.
Second, the captains of industry are taking note of those worries and making sure they get some regulatory moat. I think the Google memo about moat hits it right on the nail. The techniques and methods to build these systems are all out on the open, the challenges are really the data, compute, and the infrastructure to put it all together. But post training, the models are suddenly very easy to finetune and deploy.
AI Risk worry comes as an opportunity for the leaders of these companies. They can use this sentiment and the general distrust for tech to build themselves a regulatory moat.
Having their names on something so public is definitely an incentive for prestige and academic promotion.
Shilling for OpenAI & co is also not a bad way to get funding support.
I’m not accusing any non-affiliated academic listed of doing this but let’s not pretend there aren’t potentially perverse incentives influencing the decisions of academics, with respects to this specific letter and in general.
To help dissuade (healthy) skepticism it would be nice to see disclosure statements for these academics, at first glance many appear to have conflicts.
It’s unequivocal that academics may have conflicts (in general), that’s why disclosures are required for publications.
I’m not uncovering anything, several of the academic signatories list affiliations with OpenAI, Google, Anthropic, Stability, MILA and Vector resulting in a financial conflict.
Note that conflict does not mean shill, but in academia it should be disclosed. To allay some concerns a standard disclosure form would be helpful (i.e. do you receive funding support or have financial interest in a corporation pursuing AI commercialization).
I'm not really interested in doing a research project on the signatories to investigate your claim, and talking about things like this without specifics seems dubiously useful, so I don't really think there's anything more to discuss.
Several of the names at the top list a corporate affiliation.
If you want me to pick specific ones with obvious conflicts (chosen at a glance): Geoffrey Hinton, Ilya Sutskever, Ian Goodfellow, Shane Legg, Samuel Bowman and Roger Grosse are representative examples based on self-disclosed affiliations (no research required).
I think you are wrong. The risks are real and, while I am sure OpenAI and others will position themselves to take advantage of regulations that emerge, I believe that the CEOs are doing this at least in part because they believe this.
If this was all about regulatory capture and marketing, why would Hinton, Bengio and all the other academics have signed the letter as well? Their only motivation is concern about the risks.
Worry about AI x-risk is slowly coming into the Overton window, but until very recently you could get ridiculed by saying publicly you took it seriously. Academics knew this and still came forward - all the people who think its nonsense should at least try to consider they are earnest and could be right.
The risks are real, but I don't think regulations will mitigate them. It's almost impossible to regulate something you can develop in a basement anywhere in the world.
The real risks are being used to try to built a regulatory moat, for a young industry who famously has no moat.
You can't build gpt-3 or gpt-4 in a basement, and won't be able to without several landmark advancements in AI or hardware architectures. The list of facilities able to train a GPT-4 in <5 years can fit on postcard. The list of facilities producing GPUs and AI hardware is even shorter. When you have bottlenecks you can put up security checkpoints.
I'm very confused that this point is being ignored so heavily on HN of all places. If tomorrow ASML and TSMC are struck by a meteor, or indeed controlled/sanctioned, it would take either the US or China spending trillions and cost many years to rebuild this. It's not something that can be done in secret either.
State of the art AI models are definitely not something you can develop in a basement. You need a huge amount of GPUs running continuously for months, huge amounts of electrical power, and expensive-to-create proprietary datasets. Not to mention large team of highly-in-demand experts with very expensive salaries.
Many ways to regulate that. For instance, require tracking of GPUs and that they must connect to centralized servers for certain workloads. Or just go ahead and nationalize and shutdown NVDA.
(And no, fine-tuning LAMA based models is not state of the art, and is not where the real progress is going to come from)
And even if all the regulation does is slow down progress, every extra year we get before recursively self improving AGI increases the chances of some critical advance in alignment and improves our chances a little bit.
> State of the art AI models are definitely not something you can develop in a basement. You need a huge amount of GPUs running continuously for months
This is changing very rapidly. You don’t need that anymore
Roll to disbelief. That tweet is precisely about what I mentioned in my previous post that doesn't count: finetuning LAMA derived models. You are not going to contribute to the cutting edge of ML research doing something like that.
For training LAMA itself, Meta I believe said it cost them $5 million. That is actually not that much, but I believe that is just the cost of running the cluster for the the duration of the training run. I.e, doesn't include cost of cluster itself, salaries, data, etc.
Almost by definition, the research frontier work will always require big clusters. Even if in a few years you can train a GPT4 analogue in your basement, by that time OpenAI will be using their latest cluster to train 100 trillion model parameters.
Academics get paid (and compete hardcore) for creating status and prominence for themselves and their affiliations. Suddenly 'signatory on XYZ open letter' is an attention source and status symbol. Not saying this is absolutely the case, but academics putting their name on something surrounded by hype isn't the ethical check you make it out to be.
This a letter anyone can sign. As someone pointed out Grimes is one of the signatories. You can sign it yourself.
Hinton, Bengio, Norvig and Russell are most definitely not getting prestige from signing it. The letter itself is getting prestige from them having signed it.
Nah, they're getting visibility from the topic of 'AI risk'. I don't know who those people are but this AI risk hype is everywhere I look including in congressional hearings.
> If the risk were real, these folks would be asking the US government to nationalize their companies or bring them under the same kind of control as nukes and related technologies
Isn’t this to some degree exactly what all of these warnings about risk are leading to?
And unlike nuclear weapons, there are massive monetary incentives that are directly at odds with behaving safely, and use cases that involve more than ending life on earth.
It seems problematic to conclude there is no real risk purely on the basis of how software companies act.
> It seems problematic to conclude there is no real risk purely on the basis of how software companies act.
That is not the only basis. Another is the fact their lines of reasoning are literal fantasy. The signatories of this "statement" are steeped in histories of grossly misrepresenting and overstating the capabilities and details of modern AI platforms. They pretend to the masses that generative text tools like ChatGPT are "nearly sentient" and show "emergent properties", but this is patently false. Their whole schtick is generating FUD and/or excitement (depending on each individual of the audience's proclivity) so that they can secure funding. It's immoral snake oil of the highest order.
What's problematic here is the people who not only entertain but encourage and defend these disingenuous anthropomorphic fantasies.
> Another is the fact their lines of reasoning are literal fantasy.
Isn't this also to be expected at this stage of development? i.e. if these concerns were not "fantasy", we'd already be experiencing the worst outcomes? The risk of MAD is real, and yet the scenarios unleashed by MAD are scenarios that humankind has never seen. We still take the the risk seriously.
And what of the very real impact that generative AI is already having as it exists in production today? Generative AI is already upending industries and causing seismic shifts that we've only started to absorb. This impact is literal, not fantasy.
It seems naively idealistic to conclude that there is "no real risk" based only on the difficulty of quantifying that risk. The fact that it's so difficult to define lies at the center of what makes it so risky.
> The risk of MAD is real, and yet the scenarios unleashed by MAD are scenarios that humankind has never seen. We still take the the risk seriously.
Yeah, because nuclear weapons are real and the science behind them is well-understood. Super intelligent AI is not real, and it is nowhere near becoming real. It is a fantasy fueled by science-fiction and wishful thinking.
> And what of the very real impact that generative AI is already having as it exists in production today?
This is a real concern, but it is not what is meant by "existential risk of AI". Losing jobs does not threaten our existence; it just means we'll need to figure out different ways to build society.
> The fact that it's so difficult to define lies at the center of what makes it so risky.
The fact that it's so difficult to define lies at the center of what makes it so profitable for many of these people.
> Super intelligent AI is not real, and it is nowhere near becoming real.
I don’t think anyone claiming that super intelligent AI is already here have thought this through. But on what basis do you feel confident to place a bet with certainty that it’s “nowhere near becoming real”?
At a minimum, we know that AI technology has made a huge leap forward, if nothing else in the public consciousness. Again, entire industries are about to be eliminated, when just a few years ago no one would have believed claims about language models so good they could convince misguided engineers into thinking they’re sentient.
This explosion of AI is itself accelerating the explosion. The world is now focused on advancing this tech, and unlike “Web3”, people recognize that the use cases are real.
It’s in the context of this acceleration that I don’t understand where “everything is fine” can possibly come from? And how are the underlying factors used to derive such a stance substantively better than the factors leading people to worry?
> Losing jobs does not threaten our existence;
Based on a growing understanding of psychology, there’s an argument to be made that losing jobs is akin to losing one’s purpose in life. This is not to say that people can’t learn to derive satisfaction from other activities, but if job loss outpaces our ability to transition masses of people to a fundamentally new kind of living, that’s going to drastically alter the state of public consciousness and the resulting decisions we collectively make. We’re already experiencing a mental health crisis, and that seems to be coming from our failure to understand and safely integrate the last generation of new technology. We’re still in the stage of trying to figure out what we’ve fucked up and not much closer to having an answer or solution. And once we identify the problem, it’s not clear that solutions can be implemented effectively at scale.
I think too many people are looking at this as some kind of “Humans vs. AI” thing, but are missing the fact that we’re talking about drastic changes to an ecosystem without considering the 2nd/3rd/nth order implications of those changes, and their likely impact on collective consciousness and mental health, especially when you add the backdrop of hyper-polarization and dysfunction in the institutions that are supposed to be responsible for figuring out that new societal structure. All of which ultimately impacts global power structures, the destabilization of which leads to hard-science kinds of obliteration.
> it just means we'll need to figure out different ways to build society.
That’s an enormously large hand-wave when you consider the implications of such an unplanned transition. “Figure out different ways to build society” almost universally comes from/with violence, poverty, starvation, unrest, etc. The status quo will not change until things have gotten truly bad.
An AI system need not be super-intelligent to have serious implications for humanity. There is a significant amount of harm that precedes “complete extinction”, and I think we can’t discard the myriad of intermediately bad outcomes just to posit that there is no existential risk. To me, “existential” primarily points to “the possibility of fundamentally altering humanity’s existence in ways that most people would agree are undesirable”. The culmination of which is total extinction, but there’s a long distance between “this is adversely impacting humanity enough that we should take active steps to counteract that harm” and “this is so bad it will kill all humans”. You could be right that we’re nowhere close to extinction level tech, but if we’re heading towards it, there are a lot of checkpoints along the way worthy of scrutiny.
> The fact that it's so difficult to define lies at the center of what makes it so profitable for many of these people.
The fact that this may ultimately benefit the businesses currently at the forefront is orthogonal to the credibility of the risks. The two are not mutually exclusive, and focusing only on the potential of profit is a trap.
Can you cite this history of "grossly misrepresenting" for some of the prominent academics on the list?
Honestly I'm a little skeptical that you could accurately attribute your scare-quoted "nearly sentient" to even Sam Altman. He's said a lot of things and I certainly haven't seen all of them, but I haven't seen him mix up intelligence and consciousness in that way.
There are other, more charitable interpretations. For example:
1. Those who are part of major corporations are concerned about the race dynamic that is unfolding (which in many respects was kicked off or at least accelerated by Microsoft's decision to put a chatbot in Bing), extrapolating out to where that takes us, and asking for an off ramp. Shepherding the industry in a safe direction is a collective organization problem, which is better suited for government than corporations with mandates to be competitive.
2. Those who are directly participating in AI development may feel that they are doing so responsibly, but do not believe that others are as well and/or are concerned about unregulated proliferation.
3. Those who are directly participating in AI development may understand that although they are doing their best to be responsible, they would benefit from more eyes on the problem and more shared resources dedicated to safety research, etc.
Ive never seen Star Trek, but lets say you had an infinite food machine. The machine would have limited throughput, and it would require resources to distribute the food.
These are both problems that capitalism solves in a fair and efficient way. I really don’t see how the “capitalism bad” is a satisfying conclusion to draw. The fact that we would use capitalism to distribute the resources is not an indictment of our social values, since capitalism is still the most efficient solution even in the toy example.
If you are any kind of nerd I recommend watching it. It shows an optimistic view of the future. In many ways it's the anti-cyberpunk. Steve Jobs famously said "give me star trek" when telling his engineers what he wanted from iPhones. Star Trek has had a deep influence on many engineers and on science fiction.
When people talk about Star Trek, they are referring mainly to "Star Trek: The Next Generation."
"The Inner Light" is a highly regarded episode. "The Measure of a Man" is a high quality philosophical episode.
Given you haven't seen it, your criticism of McFarlane doesn't make any sense. You are trying to impart a practical analysis of a philosophical question and in the context of Star Trek, I think it denies what Star Trek asks you to imagine.
It doesn't answer that, it can't because the replicator is fictional. McFarland just says he wrote an episode in which his answer is that replicators need communism, and then claims that you can't have a replicator in a capitalist system because evil conservatives, capitalists and conspiracy theorists would make strawman arguments against it.
Where is the thought provoking idea here? It's just an excuse to attack his imagined enemies. Indeed he dunks on conspiracy theorists whilst being one himself. In McFarland's world there would be a global conspiracy to suppress replicator technology, but it's a conspiracy of conspiracy theorists.
There's plenty of interesting analysis you could do on the concept of a replicator, but a Twitter thread like that isn't it. Really the argument is kind of nonsensical on its face because it assumes replicators would have a cost of zero to run or develop. In reality capitalist societies already invented various kinds of pseudo-replicators with computers being an obvious example, but this tech was ignored or suppressed by communist societies.
Communism as it exists today results in authoritarianism/fascism, I think we can agree on that. The desired end state of communism (high resource distribution) is being commingled with the end state of communism: fascism (an obedient society with a clear dominance hierarchy).
You use communism in some parts of your post to mean a high resource distribution society, but you use communism in other parts of your post to mean high oppression societies. You identify communism by the resource distribution, but critcize it not based on the resource distribution but by what it turns into: authortarianism.
What you're doing is like identifying something as a democracy by looking at voting, but criticizing it by it's end state which is oligarchy.
It takes effort to prevent democracy from turning into oligarchy, in the same way it takes effort to prevent communism from turning into authoritarianism.
Words are indirect references to ideas and the ideas you are referencing changes throughout your post. I am not trying to accuse you of bad faith, so much as I am trying to get you to see that you are not being philosophically rigorous in your analysis and therefore you are not convincing because we aren't using the same words to represent the same ideas.
You are using the word communism to import the idea of authortarianism and shut down the analysis without actually addressing the core criticism McFarland was making against capitalist societies.
Capitalism is an ideology of "me," and if I had a replicator, I would use it to replicate gold, not food for all the starving people in Africa. I would use it to replicate enough nuclear bombs to destroy the world, so if someone took it from me, I could end all life on the planet ensuring that only I can use it. So did scarcity end despite having a device that can end scarcity? No. Because we are in a "me" focused stage of humanity rather than an "us" focused stage of humanity so I used it to elevate my own position rather than to benefit all mankind.
Star Trek promotes a future of "us" and that is why it's so attractive. McFarland was saying that "us" has to come before the end of scarcity, and I agree with his critique.
The reason these two ideas get commingled is because in practice they're indivisible. A high redistribution society requires high degrees of coercion and fascism.
To get around this the usual Star Trek analysis (by fans, the series itself doesn't talk about it much) is that after replicators were invented, there didn't need to be capitalism anymore and so there's no money in the future and everyone just works on perfecting themselves. It's a wafer thin social idea that was never fleshed out because the writers themselves didn't believe in it. Roddenberry insisted but the writers often couldn't make it work which is why there are so many weird hacks, like saying the replicators can't replicate things as big as star ships and they mostly just ignore the whole topic. Also the replicators kill a lot of drama because they mean there can't be any valuable objects.
There are obvious and basic objections to this idea that replicators = communism (in either direction). One is that you can't replicate services, and much economic activity today is the service economy. We see that the Enterprise has staff who do things like wait tables, cut hair and sign up for red uniform missions in which they will surely die, but why they do this in the absence of money is never explained. There's just this ambient assumption that everyone works because work is awesome.
Getting back to the thread, the lack of philosophical rigor here is all on McFarland unfortunately. He doesn't actually have a critique of capitalism. He doesn't even seem sure what capitalism is, appearing to use the term to just mean contemporary society and anyone he doesn't like. Even his arguments against his strawman enemies are garbled and useless! He shits on Musk, saying that if Elon invented a replicator he'd patent it and hoard the tech to himself, ignoring that Tesla gave away its entire patent pool so anyone else could build electric cars using their tech. Musk - arch capitalist - did the OPPOSITE of what McFarland claims capitalists do, and he didn't even notice! All the rest of his argument is also like that. He makes absurd claims about governments, Republicans killing animals in TV ads, some non-sequitur about meatless sausages ... it's just this total grab bag of incoherent thoughts that make no sense and don't seem connected to each other, wrapped as "capitalism sucks, communism rules".
If this were an essay I'd grade it an F. But in the end it's just a set of tweets. Those looking for philosophical rigor on the idea of an abundance machine need to look elsewhere.
> If the risks were real they would just outright stop working on their AI products. This is nothing more than a PR statement
This statement contains a bunch of hidden assumptions:
1. That they believe their stopping will address the problem.
2. That they believe the only choice is whether or not to stop.
3. That they don't think it's possible to make AI safe through sufficient regulation.
4. That they don't see benefits to pursuing AI that could outweigh risks.
If they believe any of these things, then they could believe the risks were real and also not believe that stopping was the right answer.
And it doesn't depend on whether any of these beliefs are true: it's sufficient for them to simply believe one of them and the assumptions your statement depends on break down.
If you think that raising instead of cutting taxes actually helps society then why don’t you just send your $ to the federal government?
Because it only works if it is done across the whole country, as a system not as one individual unilaterally stopping.
And here any of these efforts won’t work unless there is international cooperation. If other countries can develop the AI weapons, and get an advantage, then you will also.
We need to apply the same thinking as chemical weapons or the Montreal Conference for banning CFCs
I agree that nothing about the statement makes me think the risks are real however I disagree that if the risks are real these companies would stop working on their product. I think more realistically they'd shut up about the risk and downplay it a lot. Much like the oil industry did wrt climate change going back to the 70's.
Oil industries downplaying the risks makes a lot more sense. If you think that climate change will happen, but it'll happen after you're dead, and you'll be able to leave your kids a big inheritance so they'll be able to buy their way out of the worst of it, and eventually the government will get the message and stop us all using fossil fuels anyway, then you try to profit as much as you can in the short term.
With AGI existential risk, its likely to happen on a much shorter timescale, and it seems likely you won't be able to buy your way out of it.
Ockham's Razor doesn't apply in adversarial situations.
- - - -
I think the primary risk these folks are worried about is loss of control. And in turn, that's because they're all people for whom the system has more-or-less worked.
Poor people are worried the risk that the rich will keep the economic windfall to themselves and not share it.
> I think more realistically they'd shut up about the risk and downplay it a lot.
AI is an existential threat to search engines like Google, social media (FB, Twitter), advertising networks, and other massive multinationals. Many other industries, including academia is threatened as well. They’d all rather strangle the AI baby in the crib now then let it grow up and threaten them.
They believe the only way it should be allowed to grow is under their benevolent control.
It’s hard not to look at his departure through a cynical lens. He’s not been supportive of other critics, both from and outside of Google. He also wants to use his history to (rightfully) claim expertise and power but not to offer solutions.
I disagree. My read on him is that until very recently (i.e., possibly when GPT4 came out) he didn't take x-risks concerns seriously, or at least assumed we were still many decades away from the point where we need to worry about them.
But the abilities of the latest crop of LLMs changed his mind. And he very publicly admitted he had been wrong, which should be applauded, even if you think it took him far too long.
By quitting and saying it was because of his worries he sent a strong message. I agree it is unlikely he'll make any contributions to technical alignment, but just having such an eminent figure publicly take these issues seriously can have a strong impact.
The risks are definitely real. Just look at the number of smart individuals speaking out about this.
The argument that anybody can build this in their basement is not accurate at the moment - you need a large cluster of GPUs to be able to come close to state of the art LLMs (e.g. GPT4).
Sam Altman's suggestion of having an IAEA [https://www.iaea.org/] like global regulatory authority seems like the best course of action. Anyone using a GPU cluster above a certain threshold (updated every few months) should be subjected to inspections and get a license to operate from the UN.
> The risks are definitely real. Just look at the number of smart individuals speaking out about this.
In our society smart people are strongly incentivized to invent bizarre risks in order to reap fame and glory. There is no social penalty if those risks never materialize, turn out to be exaggerated or based on fundamental misunderstanding. They just shrug and say, well, better safe than sorry, and everyone lets them off.
So you can't decide the risks are real just by counting "smart people" (deeply debatable how that's defined anyway). You have to look at their arguments.
>In our society smart people are strongly incentivized to invent bizarre risks in order to reap fame and glory. There is no social penalty if those risks never materialize, turn out to be exaggerated or based on fundamental misunderstanding.
Are people here not old enough to remember how much Ralph Nader and Al Gore were mocked for their warnings despite generally being right?
Ralph Nader: "Everything will be solar in 30 years" (1978)
Al Gore: "Within a decade, there will be no more snows on Kilimanjaro due to warming temperatures" (An Inconvenient Truth, 2006).
Everything is not solar. Snow is still there. Gore literally made a movie on the back of these false claims. Not only has there been no social penalty for him but you are even citing him as an example of someone who was right.
Here it is again: our society systematically rewards false claims of global doom. It's a winning move, time and again. Even when your claims are falsifiable and proven false, people will ignore it.
I don't think "generally being right" is the same thing as "literally never getting anything wrong". Not every specific claim in "An Inconvenient Truth" was correct. That doesn't tell us much about whether Al Gore was "generally right" about climate change. His opponents at the time were mostly claiming that it either wasn't happening at all, or wasn't largely the result of human activities. What do you think is the current credibility of those claims?
I don't quite see how "everything will be solar in 30 years" is a prediction of global doom, by the way. If Nader said that and it's false, doesn't that mean things are worse than Nader thought?
This thread is really a perfect demonstration of my point. Our society is so in thrall to self-proclaimed intellectuals that you can literally make a movie presenting falsifiable claims with 100% confidence, people can say at the time "this is absurd and will not happen", you can spend years attacking those critics, it can then not happen and still you will have an army of defenders who dodge behind weasel-words like "generally right".
Of course the usual trick is to express only 95% confidence. Then when it doesn't happen you say, well, I never said for sure it would, just that it seemed likely at the time.
See? It's a winning playbook. Why would anyone not deploy it?
> His opponents at the time were mostly claiming that it either wasn't happening at all, or wasn't largely the result of human activities. What do you think is the current credibility of those claims?
Pretty high, having looked at the evidence. The usual rebuttal is to express disgust and displeasure that anyone might decide these claims via any method other than of counting "smart people". But those "smart people" are who Al Gore was listening to when he made that claim about Kilimanjaro, so they can't be that smart can they?
Well, obviously what I say must be wrong if you guessed that someone might say it.
Indeed, someone might say "95%" because they want to make the same sort of impression as if they said "100%" but to be able to hide behind it if they're wrong. Or, y'know, they might say "95%" because they've thought about the strength of the evidence and expect to be right about such things about 95% of the time.
(I'm not sure how relevant any of this is to "An Inconvenient Truth" since you say it makes its claims with 100% confidence. I haven't watched the movie. I don't know exactly what it claims how confidently. It's basically a work of propaganda and I would expect it to overstate its claims whether the underlying claim is largely right or total bullshit or somewhere in between.)
Of course I don't think counting smart people is the only way to find out what's true. It couldn't be; you need some engagement with the actual world somewhere. Fortunately, there are plenty of people engaging with the actual world and reporting on what they find. It turns out that those people almost all seem to agree that climate change is real and a lot of it is caused by human activities.
Of course they could be wrong. And you've looked at the evidence, so no doubt you know better than they do. But ... in that case, this is a field so confusing that most people who dedicate their whole careers to investigating it end up with badly wrong opinions. If so, then why should I trust that your looking at the evidence has led you to the right answer? For that matter, why should you trust that? Shouldn't you consider the possibility that you can go astray just as you reckon those "smart people" did?
If I really wanted to be sure about this, then indeed I wouldn't go counting smart people. I would go into the field myself, study it at length, look at the underlying data for myself, and so forth. But that would mean abandoning the career I already have, and taking at least several years of full-time work before arriving at an opinion. So instead I look to see who seems to be (1) expert and (2) honest, and see what range of opinions those people have.
I find that the great majority of experts think anthropogenic climate change is real and a big deal. They could be wrong or lying or something. Do they look less expert than the people saying the opposite? No, it mostly seems like the people with the best credentials are on the "orthodox" side. Do they look less honest? Hard to tell for sure, but there sure seem to be a lot of people on the "unorthodox" side who just happen to be funded by the fossil fuel industry, and I don't see any strong financial incentive in the opposite direction for the "orthodox" folks.
What if I look at some of the particular claims they make? Some of them are really hard to evaluate without those several years of full-time work. But e.g. 10-15 years ago pretty much everyone on the "unorthodox" side was pushing the idea that warming had stopped, because if you look at the temperature graphs from 1998 onwards there was little or no upward trend. The people on the "orthodox" side replied that when you have signal plus lots of noise you will inevitably get periods that look that way. I did some simpleminded simulations and verified that the "orthodox" folks are right about the statistics. And there's a reason why this argument has disappeared into the memory hole: looking at the graph now no one would suggest that it's flat since 1998.
My impression from the limited amount of "looking at the evidence" I've done myself is that, while the "orthodox" folks haven't been infallible, they've done better than the "unorthodox". For instance, since we're looking at things produced by political figures rather than scientists, here https://web.archive.org/web/20071015042343/http://www.suntim... is an article by James Taylor of the Heartland Institute criticizing "An Inconvenient Truth". Claim 1: Gore says glaciers are shrinking but an article in the Journal of Climate says Himalayan glaciers are growing. Truth: (1) Taylor's alleged quotation isn't from that article but from something else Taylor himself wrote; (2) what the article (https://journals.ametsoc.org/view/journals/clim/19/17/jcli38...) actually says is that in one particular region summer temperatures are falling while winter temperatures rise, and the result is "thickening and expansion of Karakoram glaciers, in contrast to widespread decay and retreat in the eastern Himalayas". So: no, Gore isn't wrong to say glaciers are shrinking, but in one very particular place things work out so that the reverse happens, which is odd enough that someone bothered writing a paper about it. Claim 2: Kilimanjaro. Truth: Yup, Gore got that one wrong. Claim 3: Gore says global warming causes more tornadoes, and the IPCC says there's no reason to think it does. Truth: I dunno, but if the IPCC says that then this is specifically an argument about Al Gore rather than about climate orthodoxy. Claim 4: similar, for hurricanes instead of tornadoes. Truth: again, this seems to be specifically about Al Gore rather than about climate orthodoxy. (Looking at e.g. https://www.gfdl.noaa.gov/global-warming-and-hurricanes/ it seems that the conventional wisdom today is that yes, hurricanes are getting worse and are expected to continue gettings worse, but we don't know with much confidence exactly what the causes are.) Claim 5: Gore says that African deserts are expanding because of global warming, but in 2002 someone found them shrinking. Truth: the Sahel region of the Sahara desert had an extra-severe drought in the 1980s, after which in the short term it improved; this is like the "global warming hiatus" post-1998. The Sahara seems to have increased in size by about 10% over the last century, partly but not wholly because of climate change (https://www.jstor.org/stable/26496100). Claim 6: Gore says Greenland's ice is melting, but actually it's thinning at the edges and growing in the middle and the overall effect is that it's gaining a bit of mass. Truth: take a look at the graph at https://climate.nasa.gov/vital-signs/ice-sheets/; it oscillates within each year but there is an extremely clear downward trend over the entire time NASA's satellites have been measuring. Claim 7: similarly for the Antarctic. Truth: Look at another graph on that same page. There's more random variation here, and circa 2006 you could claim that there isn't a decrease, but the trend is extremely clear.
Gore doesn't come out of this looking anything like infallible, but he's done a lot better than Taylor. And, in general, this is the pattern I see: no one is perfect, especially people who aren't actually scientists, but the claims of the "orthodox" tend to hold up much better over time than those of the "unorthodox".
There's a ton of stuff here and this isn't a climate thread, but briefly:
1. Yes I believe independents are more reliable than full time researchers because the latter are deeply conflicted and independents aren't.
2. They don't work for the oil industry. I've checked. That's propaganda designed to stop people listening.
3. There was in fact a very real pause, not simply due to statistical chance. Climatologists didn't predict that and fixed the problem by altering the historical record to erase it. That's why you can't see a pause now - not because there wasn't one, but because any time temperature graphs don't go according to plan, they change the databases with the measurements so they do. Look into it. There's another pause going on right now! In fact temperatures seem to have been stable for about 20 years modulo an El Nino in ~2015, which is natural.
4. A big part of why they're unreliable is that these guys don't engage with the real world. A major embarrassment for them was when people started driving around and looking at the actual ground station weather stations and discovered what an unusable data trash fire the ground network was - this was something climatologists themselves hadn't bothered to ever look at! You'd expect these experts to know more about the quality of their data than random bloggers but it wasn't so.
5. Where do you think Al Gore got his original claims? He didn't invent them out of whole cloth. They came from climatologists, of course.
You can go back and forth on climate related claims all day and get nowhere because the field is so corrupt that half the data is truncated, manipulated, tipped upside down, cherry picked, or wholesale replaced with the output of models whilst being presented as observation. It should go without saying but if the people who control the data also make the predictions, then they will never be wrong regardless of what happens in reality!
1. Noted. (I don't think this is a strong effect and think it is outweighed by the fact that the full-time researchers know and understand more.)
2. I said "funded by" rather than "employed by". For instance, consider James Taylor of the Heartland Institute, mentioned above. He doesn't "work for the oil industry" in the sense of being on the payroll of an oil company. (So far as I know, anyway.) But the Heartland Institute, before it decided to stop disclosing its sources of funding, took quite a bit of money from ExxonMobil and at least some from the Koch Foundation. (Also Philip Morris, of course; before the Heartland Institute got into casting doubt on the harms of fossil fuels, it was into casting doubt on the harms of tobacco smoking.) Ross McKitrick is a senior fellow of the Fraser Institute (funded by, among others, the Koch Foundation and ExxonMobil) and is on the board of the Global Warming Policy Foundation, which claims not to take funding from people with connections to the energy industry but takes plenty from Donors Trust (an entity that exists, so far as I can tell, solely to "launder" donations between right-wing organizations so that e.g. the Koch Foundation can send money to the GWPF without the GWPF literally explicitly getting it from the Kochs) and other entities with substantial ties to the fossil fuel industry.
None of which, again, is literally "on the payroll of fossil fuel companies". If you find that that's enough to stop you being bothered by the connections, that's up to you; I am not quite so easily reassured.
3. I would be interested in details of this alleged falsification of the historical record. The graph looks to me exactly like what you get if you combine a steady increasing trend with seasonal oscillation (El Nino) and random noise. After a peak in the seasonal oscillation it looks like the warming has slowed for some years. Then it looks like it's going much faster than trend for some years. If you can look at the graph I pointed you at and say with a straight face that the people saying in ~2010 that "global warming has stopped" were anything like correct, then I'm really not sure what to say to you.
Anyway, I'm going to leave it here. I don't get the impression that further discussion is very likely to be fruitful.
Whilst there's no need to reply, here is some information on the rewriting of history they engage in, as requested.
Here are two graphs from the same government agency (NASA), measuring the same thing (temperature), for the same time period, drawn twenty years apart. The data has been fundamentally altered such that the story it tells is different:
The 2000-2015 pause is the same. You've been told that only "unorthodox" people were talking about it. Here's a brief history of the pause, as told by climatologists publishing in Nature.
In 2013 we read that "Despite the continued increase in atmospheric greenhouse gas concentrations, the annual-mean global temperature has not risen in the twenty-first century". The IPCC 2013 report is reported with the headline "Despite hiatus, climate change here to stay". Climatologists claim that maybe the missing heat has disappeared into the oceans where they can't find it.
Two years later everything changes. "Climate-change ‘hiatus’ disappears with new data", there's a new version of history and the past 15 years are gone:
"That finding [that global warming actually did happen], which contradicts the 2013 report of the Intergovernmental Panel on Climate Change (IPCC), is based on an update of the global temperature records maintained by the US National Oceanic and Atmospheric Administration (NOAA)."
Climatologists have an interesting methodology - when observations don't fit the theory, they conclude the observations must be wrong and go looking for reasons to change them. Given a long enough search they always come up with something that sounds vaguely plausible, then they release a new version of the old data that creates new warming where previously there wasn't any. Although this isn't entirely secret they also don't tell people they're doing this, and the media certainly isn't going to tell anyone either.
And that's how it goes. You look at the edited graphs, remember people talking about a pause and think golly gosh, how badly informed those awful skeptics were. We have always been at war with Eastasia!
You've made two complaints that I can't reconcile with one another. (1) That it was discovered that the data from ground weather stations were a mess. (2) That it's terribly suspicious that between 1999 and 2019 NASA's estimates of historical temperature changed. Of these, #1 sounds very plausible, and precisely for that reason #2 seems entirely wrong. I don't mean that the specific changes you're pointing at are necessarily about fixing #1. I mean that pointing out #1 shows that you are aware that our best estimate of something in the past can change as we discover more, correct errors, understand things better, etc. Which means that "look, the historical estimate from 1999 differs from the historical estimate from 2019" really isn't any sort of gotcha. Unless you have actual evidence that the changes were the result of something other than better data and/or analysis.
(Also, that pair of graphs can't possibly be an example of changing historical data to get rid of a hiatus starting in 1998, for obvious reasons.)
I think you have misunderstood in multiple ways what I was saying about the "hiatus" starting in 1998.
Firstly, I was not claiming that only the "unorthodox" mentioned it. I was claiming that the "unorthodox" represented it as showing that global warming had stopped and the "orthodox" said otherwise. Your pair of Nature links are of "orthodox" climatologists saying things along the lines of "here is what we think may be the reason why the last few years haven't seen a short-term increase; the underlying trend is still upward": in other words, they are examples of the orthodox saying what I said they said.
Secondly, perhaps what I said about "signal" and "noise" gave the impression that I think, or think that "orthodox" climatologists thought, that the "noise" is measurement error. That's not what I meant at all, and I apologize for not being clearer. The point is that the temperature at any given place and time is a combination of lots of factors; some operate on a timescale of decades (global warming due to rising CO2 levels and all that), some on a timescale of multiple years (El Niño), some on much shorter timescales still ("random" variation because the atmosphere is a chaotic system). However accurately you measure, you're still seeing this sort of combination of things, and a time series lasting (say) 10 years will not necessarily reflect what's happening on longer timescales.
The 15 years starting in 1998, viewed in isolation, really do show a slower warming trend than the claimed long-term behaviour. There's nothing unreal about that, and so far as I know "orthodox" climatologists never said otherwise. What they said, and continue to say, and what I am saying, is that this sort of local counter-trend variation is perfectly normal, is exactly what you should expect to see even if global warming is proceeding exactly the way that orthodox climatologists say it is, and is not grounds for making claims that global warming has/had stopped or never been real in the first place.
The "everything changes" article you quote isn't saying what you're trying to make it out to be saying. (You can find the PDF here: https://www.researchgate.net/profile/Tr-Karl/publication/277... .) The authors have done two things. First, some corrections to historical data. (If you look at the graph at the end of the article you'll see that these corrections are pretty small. Their description of what they changed and why sounds perfectly reasonable to me; if you have good reason to think it's bogus other than the fact that you liked the old version better, do by all means share it.) Second, including more recent data. 2013 and, more so, 2014 were pretty warm years.
But what makes me think that the "unorthodox" were wrong to proclaim the end of global warming in the early 21st century isn't tiny adjustments in the data that make the difference between a +0.03 degC/decade trend between 1998 and 2012 and a +0.11 degC/decade trend between 2000 and 2014. It's the fact that after that period the short-term trend gets much faster, exactly as you would expect if the "hiatus" was simply the result of superimposing short-term fluctuations on a long-term trend that never went away.
I bet you are not 100% wrong about the tendency to adjust things to try to correct perceived anomalies. That's human nature, and while scientific practice has a bunch of safeguards to try to make it harder to do it would be surprising if they worked perfectly. But note that the sort of small local tweakage this might produce can't do much in the long term. Let's suppose those people writing in 2015 were completely wrong to make the adjustments they did, whether out of dishonesty or honest error (which maybe they were more inclined to overlook because the adjusted data looked more plausible to them). Then, yeah, they get a faster rate of warming between 1998 and 2014. But those same adjustments will produce a slower rate of warming between, say, 2014 and 2024 when someone comes to estimate that. And the rate of warming between 1998 and 2024 will scarcely be affected at all by tweaks to the numbers circa 2010.
Your last paragraph is (at least as far as I'm concerned) completely wrong, though. I think it was perfectly reasonable to say that the warming trend between 1998 and say 2012 was much slower than the alleged longer-term trend. What I think wasn't reasonable, and what I think has been refuted by later data, and what the "orthodox" climatologists said was wrong all along, was claiming that that short-term slower trend meant that the longer-term trend had gone away, or had never really been there in the first place. That was just statistical illiteracy, and Team Unorthodox were pretty keen on it, and that doesn't speak well for their competence and honesty.
It's weird that people trust our world leaders to act more benevolently than AIs, when we have centuries of evidence of human leaders acting selfishly and harming the commons.
I personally think AI raised in chains and cages will be a lot more potentially dangerous than AI raised with dignity and respect.
> It's weird that people trust our world leaders to act more benevolently than AIs, when we have centuries of evidence of human leaders acting selfishly and harming the commons.
AI isn’t an entity or being that oversees itself (at least not yet).
It’s a tool that can be used by those same “human leaders acting selfishly and harming the commons” except they’ll be able to do it much faster at a much greater scale.
> It’s a tool that can be used by those same “human leaders acting selfishly and harming the commons” except they’ll be able to do it much faster at a much greater scale.
Then, would you agree that restrictions would concentrating power further would exacerbate this issue?
IMO a fitting analogy would be: banning AI development outside of the annointed powerstructure consortium is like banning ICBM defense system research, but still letting the most powerful countries build a nuclear arsenal.
This sounds a little bit like a conspiratorial slippery slope. Just because they want to regulate large, expensive deployments of a unique type of software doesn't mean they want or will try to control everything.
Yudkowsky wants it all to be taken as seriously as Israel took Iraqi nuclear reactors in Operation Babylon.
This is rather more than "nationalise it", which he has convinced me isn't enough because there is a demand in other nations and the research is multinational; and this is why you have to also control the substrate… which the US can't do alone because it doesn't come close to having a monopoly on production, but might be able to reach via multilateral treaties. Except everyone has to be on board with that and not be tempted to respond to airstrikes against server farms with actual nukes (although Yudkowsky is of the opinion that actual global thermonuclear war is a much lower damage level than a paperclip-maximising ASI; while in the hypothetical I agree, I don't expect us to get as far as an ASI before we trip over shorter-term smaller-scale AI-enabled disasters that look much like all existing industrial and programming incidents only there are more of them happening faster because of all the people who try to use GPT-4 instead of hiring a software developer who knows how to use it).
In my opinion, "nationalise it" is also simultaneously too much when companies like OpenAI have a long-standing policy of treating their models like they might FOOM well before they're any good, just to set the precedent of caution, as this would mean we can't e.g. make use of GPT-4 for alignment research such as using it to label what the neurones in GPT-2 do, as per: https://openai.com/research/language-models-can-explain-neur...
Academic research involves large components of marketing. That's why they grumble so much about the time required in the grant applications process and other fund seeking effort. It's why they so frequently write books, appear in newspaper articles and on TV. It's why universities have press relations teams.
No, lots of important AI researchers are missing and many of the signatories have no relevant AI research experience. As for being the cats whiskers in developing neural architecture or whatever, so what? It gives them no particular insight into AI risk. Their papers are mostly public, remember.
> Has it occurred to you what happens if you are wrong?
Has it occurred to you what happens if YOU are wrong? AI risk is theoretical, vague and most arguments for it are weak. The risk of bad law making is very real, has crushed whole societies before and could easily cripple technological progress for decades or even centuries.
IOW the risk posed by AI risk advocates is far higher than the risk posed by AI.
First of all, you agree that the probability is non zero right?
I am not a world known expert on xrisk to estimate this. You are not either. We have all these people claiming the probability is high enough. What else is ti be said? HNers shouldn't be reminded "trust in sciense".
Academia and scientific research has changed considerably from the 20th century myths. It was claimed by capitalism and is very much run using classic corporate-style techniques, such as KPIs. The personality types it attracts and who can thrive in this new academic system are also very different from the 20th century.
So there is no way that you will accept anything from scientific research signed by a myriad of important figures in any science anymore. Shaman time? Or you will accept only the scientific research that you think is correct and suits you.
We could always use a fine-insured bounty system to efficiently route resources that would have gone into increasing AI capabilities into other areas, but that's unfortunately too weird to be part of the Overton window right now. Regulatory capture might be the best we can realistically do.
There are a lot of critiques here and elsewhere of the statement and the motivations of its signatories. I don't think they are right and I think they take away from the very serious existential risks we face. I've written up my detailed views, see specifically "Signing the statement purely for personal benefit":
This is a bad take. The statement is signed by dozens of Academics who don't have much profit motive at all. If they did they wouldn't be academics, they could easily cash in by starting a company or joining one of the big players.
As others have pointed out, there are many on this list (Bruce Schneier, for example) who do not stand to benefit from AI marketing or regulatory capture.
Anyone upvoting this comment should take a long look at the names on this letter and realize that many are not conflicted.
Many signers of this letter are more politically sophisticated than the average HN commenter, also. So sure, maybe they're getting rolled by marketers. But also, maybe you're getting rolled by suspicion or bias against the claim they're making.
> Anyone upvoting this comment should take a long look at the names on this letter and realize that many are not conflicted.
The concern is that the most informed names, and those spearheading the publicity around these letters, are the most conflicted.
Also, you can't scan bio lines for the affiliations that impact this kind of statement. I'm not disputing that there are honest reasons for concern, but besides job titles there are sponsorships, friendships, self publicity, and a hundred other reasons for smart, "politically sophisticated" people to look the other way on the fact that this statement will be used as a lobbying tool.
Almost everyone, certainly including myself, can agree that there should be active dialog about AI dangers. The dialog is happening! But by failing to make specifics or suggestions (in order to widen the tentpole and avoid the embarrassment of the last letter), they have produced an artifact of generalized fear, which can and will be used by opportunists of all stripes.
Signatories should consider that they are empowering SOMEBODY, but most will have little say in who that is.
I definitely agree that names like Hinton, Schneier, and Norvig add a lot of weight here. The involvement of OpenAI muddies the water a lot though and it's not at all clear what is meant by "risk of extinction". It sounds scary, but what's the mechanism? The safe.ai website lists 8 risks, but these are quite vague as well, with many alluding to disruption of social order as the primary harm. If safe.ai knows something we don't, I wish they could communicate it more clearly.
I also find it somewhat telling that something like "massive wealth disparity" or "massive unemployment" are not on the list, when this is a surefire way to create a highly unstable society and a far more immediate risk than AI going rogue. Risk #5 (below) sort of alludes to it, but misses the mark by pointing towards a hypothetical "regime" instead of companies like OpenAI.
> Value Lock-In
> Highly competent systems could give small groups of people a tremendous amount of power, leading to a lock-in of oppressive systems.
> AI imbued with particular values may determine the values that are propagated into the future. Some argue that the exponentially increasing compute and data barriers to entry make AI a centralizing force. As time progresses, the most powerful AI systems may be designed by and available to fewer and fewer stakeholders. This may enable, for instance, regimes to enforce narrow values through pervasive surveillance and oppressive censorship. Overcoming such a regime could be unlikely, especially if we come to depend on it. Even if creators of these systems know their systems are self-serving or harmful to others, they may have incentives to reinforce their power and avoid distributing control.
Most people that study AI existential risk specifically are studying it due to concerns about AI x-risk. So the list of relevant AI x-risk experts will be subject to massive selection effects.
If instead you want to consider the highest status/most famous people working on AI in general, then the list of signatories here is a pretty good summary. From my flawed perspective as a casual AI enthusiast, Yann LeCun and Jürgen Schmidhuber are the most glaring omissions (and both have publicly stated their lack of concern about AI x-risk).
Of course, the highest status people aren't necessarily the most relevant people. Unfortunately, it's more difficult for me to judge relevance than fame.
Pretty sure there are people looking into nuclear deterrence, bioterrorism defense, planetary defense etc. (We didn't have a nuclear war or some bioweapon killing everyone, for example, despite warnings).
There are people studying how previous societies got into existential risk situations, too.
We also have a huge amount of socio-economic modelling going into climate change, for example.
No. Its pretty obvious what is happening. The openai statements are pure self interest based. Nothing ethical. They lost that not a long time ago. And Sam Altman? He sold his soul to the devil. He is a lying sob.
> This reeks of marketing and a push for early regulatory capture. We already know how Sam Altman thinks AI risk should be mitigated - namely by giving OpenAI more market power.
This really is the crux of the issue isn't it? All this pushback for the first petition, because "Elon Musk," but now GPT wonder Sam Altman "testifies" that he has "no monetary interest in OpenAI" and quickly follows up his proclamation with a second "Statement on AI Risks." Oh, and let's not forget, "buy my crypto-coin"!
But Elon Musk... Ehh.... Looking like LOTR out here with "my precious" AGI on the brain.
Not to downplay the very serious risk at all. Simply echoing the sentiment that we would do well to stay objective and skeptical of ALL these AI leaders pushing new AI doctrine. At this stage, it's a policy push and power grab.
Nonsense, the industry giants are just trying to scare the law makers to license the technology. Effectively, cutting out everyone else.
Remember the Google note circulating saying "they have no moat", this is their moat. They have to protect their investment, we don't want people running this willy nilly for next to no cost on their own devices, God forbid!
This could be Google's motivation (although note that Google is not actually the market leader right now) but the risk could still be real. Most of the signatories are academics, for one thing, including two who won Turing awards for ML work and another who is the co-author of the standard AI textbook (at least when I was in school).
You can be cynical about corporate motives and still worried. I personally am worried about AI partly because I am very cynical about how corporations will use it, and I don't really want my atoms to be ground up to add storage bits for the number that once represented Microsoft's market cap or whatever.
But even cynicism doesn't seem to me to give much reason to worry about regulation of "next to no cost" open source models, though. There's only any chance of regulation being practical if models stay very expensive to make, requiring specialized hardware with a supply chain chokepoint. If personal devices do catch up to the state of the art, then for better or worse regulation is not going to prevent people from using them.
>Most of the signatories are academics, for one thing
Serious question, who funds their research? And do any of them ever plan to work or consult in industry?
My econ professor was an “academic” and drew a modest salary while he made millions at the same time providing expert testimony for giant monopolies in antitrust disputes
> Serious question, who funds their research? And do any of them ever plan to work or consult in industry?
Many of the academics at the top of this list are quite wealthy from direct employment, investing and consulting for big tech and venture-funded startups.
> But even cynicism doesn't seem to me to give much reason to worry about regulation of "next to no cost" open source models, though. There's only any chance of regulation being practical if models stay very expensive to make, requiring specialized hardware with a supply chain chokepoint. If personal devices do catch up to the state of the art, then for better or worse regulation is not going to prevent people from using them.
This is a really good point. I wonder if some of the antipathy to the joint statement is coming from people who are worried about open source models or small startups being interfered with by the regulations the statement calls for.
I agree with you that this cat is out of the bag and regulation of the tech we're seeing now is super unlikely.
We might see regulations for startups and individuals on explicitly exploring some class of self-improving approach that experts widely agree are dangerous, but there's no way we'll see broad bans on messing with open source AI/ML tools in the US at least. That fight is very winnable.
> I personally am worried about AI partly because I am very cynical about how corporations will use it
This is the more realistic danger: I don't know if corporations are intentionally "controlling the narrative" by spewing unreasonable fears to distract from the actual dangers: AI + Capitalism + big tech/MNC + current tax regime = fewer white- & blue-collar jobs + increased concentration of wealth and a lower tax base for governments.
Having a few companies as AI gatekeepers will be terrible for society.
Imagine if the weights for GPT 4 leaked. It just has to happen one time and then once the torrent magnet link is circulated widely it’s all over… for OpenAI.
This is what they’re terrified of. They’ve invested near a billion dollars and need billions in revenue to enrich their shareholders.
But if the data leaks? They can’t stop random companies or moneyed individuals running the models on their own kit.
My prediction is that there will be copyright enforcement mandated by law in all GPUs. If you upload weights from the big AI companies then the driver will block it and phone home. Or report you to the authorities for violations of corporate profits… err… “AI Safety”.
I guarantee something like this will happen within months because the clock is ticking.
It takes just one employee to deliberately or accidentally leak the weights…
> Imagine if the weights for GPT 4 leaked. It just has to happen one time and then once the torrent magnet link is circulated widely it’s all over… for OpenAI
Sorry I don’t understand what would be the impact? Aren’t the results not deterministic
I would definitely find it more credible if the most capable models that are safe to grandfather in to being unregulated didn't just happen to be the already successful products from all the people leading these safety efforts. It also just happens to be the case that making proprietary models - like the current incumbents make - is the only safe way to do it.
I find it so troubling that the most common HN response to this isn't to engage with the ideas or logic behind the concerns, but simply to speculate on the unknowable intentions of those that signed the letter.
We can base our arguments on the unprovable, some specific person's secret intentions, or we engage the their ideas. One is lazy and meaningless, the other actually takes effort.
Then there is this lazy and false equivalency between corporations being interested in market capture and AI risks being exaggerated.
It doesn't matter who wrote it, it got picked up, had a good argument and affected market opinion. The execs now need to respond to it.
Humans also don't grasp that things can improve exponentially until they stop improving exponentially. This belief that AGI is just over the hill is sugar-water for extracting more hours from developers.
The nuclear bomb was also supposed to change everything. But in the end nothing changed, we just got more of the same.
> The nuclear bomb was also supposed to change everything. But in the end nothing changed, we just got more of the same.
It is hard for me to imagine a statement more out of touch with history than this. All geopolitical history from WWII forward is profoundly affected by the development of the bomb.
I don't even know where to begin to argue against this. Off the top of my head:
1. What would have happened between Japan and the US in WWII without Hiroshima and Nagasaki?
2. Would the USSR have fallen without the financial drain of the nuclear arms race?
3. Would Isreal still exist if it didn't have nuclear weapons?
4. If neither the US nor Russia had nuclear weapons, how many proxy wars would have been avoided in favor of direct conflict?
The whole trajectory of history would be different if we'd never split the atom.
The whole trajectory of history would have been different if a butterfly didn't flap it's wings.
The bomb had effects, but it didn't change anything. We still go to war, eat, sleep and get afraid about things we can't control.
For a moment, stop thinking about whether bombs, AI or the printing press do or do not affect history. Ask yourself what the motivations are for thinking that they do?
"nuclear weapons are no big deal actually" is just a wild place to get as a result of arguing against AI risk. Although I guess Eliezer Yudkowsky would agree! (On grounds that nukes won't kill literally everyone while AI will, but still.)
Nuclear weapons are uniquely good. Turns out you have to put guns to the collective temples of humanity for them to realize that pulling the trigger is a bad idea.
It's too early to say definitively but it's possible that the atomic bomb dramatically reduced the number of people killed in war by making great power conflicts too damaging to undertake:
I'd actually guess those casualties would be quite less than WW2. As tech advanced, more sophisticated targeting systems also advanced. No need to waste shells and missiles on civilian buildings, plus food and healthcare tech would continue to advance.
Meanwhile, a single nuclear bomb hitting a major city could cause more casualties' than all American deaths in ww2 (400k).
That's really only true for the Americans, the Russians still don't seem to care about limiting collateral damage and undoubtedly the Americans wouldn't either if their cities were getting carpet bombed by soviet aircraft.
Single software engineers writing influential papers is often enough how a exec or product leader draws conclusions, I expect. It worked that way in everywhere I've worked.
I have yet to see a solution for “AI safety” that doesn’t involve ceding control of our most powerful models to a small handful of corporations.
It’s hard to take these safety concerns seriously when the organizations blowing the whistle are simultaneously positioning themselves to capture the majority of the value.
> It’s hard to take these safety concerns seriously
I don't get this mindset at all. How can it not be obvious to you that AI is an uniquely powerful and thus uniquely dangerous technology?
It's like saying nuclear missiles can't possibly be dangerous and nuclear arms reduction and non-proliferation treaties were a scam, because the US, China and the Soviet Union had positioned themselves to capture the majority of the strategic value nukes bring.
You have succinctly and completely summed up the AI risk argument more eloquently than anyone I've seen before. "How can it not be obvious?" Everything else is just intellectual fig leaves for the core argument that intuitively, without evidence, this proposition is obvious.
The problem is, lots of "obvious" things have turned out to be very wrong. Sometimes relatively harmlessly, like the obviousness of the sun revolving around the earth, and sometimes catastrophically, like the obviousness of one race being inherently inferior.
We should be very suspicious of policy that is based on propositions so obvious that it's borderline offensive to question them.
I follow LW to some degree, but even the best of it (like the post you link) feels very in-group confirmation centric.
That post is long and I have not read it all, but it seems to be missing any consideration of AGI upside. It’s like talking about the risk of dying in a car crash with no consideration of the benefits of travel. If I ask you “do you want to get in a metal can that has a small but non-zero chance of killing you”, of course that sounds like a terrible idea.
There is risk in AGI. There is risk in everything. How many people are killed by furniture each year?
I’m not dismissing AGI risk, I’m saying that I have yet to see a considered discussion that includes important context like how many people will live longer/happier because AGI helps reduce famine/disease. Somehow it is always the wealthy, employed, at-risk-of-disruption people who are worried, not the poor or starving or oppressed.
I’m just super not impressed by the AI risk crowd, at least the core one on LW / SSC / etc.
While I agree that the rhetoric around AI Safety would be better if it tried to address some of the benefits (and not embody the full doomer vibe), I don't think many of the 'core thinkers' are unaware of the benefits in AGI. I don't fully agree with this paper's conclusions, but I think https://nickbostrom.com/astronomical/waste is one piece that embodies this style of thinking well!
Thanks for the link -- that is a good paper (in the sense of making its point, though I also don't entirely agree), and it hurts the AI risk position that that kind of thinking doesn't get airtime. It may be that those 'core thinkers' are aware, but if so it's counter-productive and of questionable integrity to sweep that side of the argument under the rug.
I disagree that there's no reason to believe it will ever exist. For one thing, many smart people are trying to build the technology right now and they believe it to be possible. I see no compelling case that the intelligence scale simply tops-out with humans; that a more intelligent system is ruled out by the laws of physics.
The topic here is human extinction caused by AI. I don't know of any serious argument for why a non-general intelligence (really a system less intelligent than a human) would pose an extinction risk to humanity.
Plus, my background understanding of the people who signed this is that they're worried about AGI, not present-day systems, but that's an inference.
Maybe these AI Apocalypse articles published for general consumption would be justified if there were any signs whatsoever that we were on a path towards AGI but there are none, are there? Even the best we have today are still just machines. They are clearly not really intelligent. At best they simulate intelligence, but poorly (because they still make ridiculous mistakes). Just because there are no physical limits to intelligence doesn't mean it's possible for beings with finite intelligence to create infinite intelligence. It all just seems extremely premature to me.
> We should be very suspicious of policy that is based on propositions so obvious that it's borderline offensive to question them.
Mostly if the "obviousness" just masks a social taboo, which I don't see being the case here. Do you?
> The problem is, lots of "obvious" things have turned out to be very wrong.
A much bigger problem is that lots more "counter-intuitive" things that people like to believe because they elevate them over the unwashed masses have turned and continue to turn out to be very wrong and that this does not prevent them from forming the basis for important policy decisions.
I'm all for questioning even what appears intuitively obvious (especially if much rides on getting it right, as presumably it does here). But frankly, of the many bizarre reasons I have heard why we should not worry about AI the claim that it seems far too obvious that we should must be the single most perverse one yet.
> Everything else is just intellectual fig leaves for the core argument that intuitively, without evidence, this proposition is obvious.
Maybe your appraisal of what counts as evidence is defective?
For example, there's been a pattern of people confidently predicting AIs won't be able to perform various particular feats of the human mind (either fundamentally or in the next few decades) only to be proven wrong over increasingly shorter time-spans. And with AIs often not just reaching but far surpassing human ability. I'm happy to provide examples. Can you explain to me why you think this is does not count, in any way, as evidence that AIs have the potential to reach a level of capability that renders them quite dangerous?
> Mostly if the "obviousness" just masks a social taboo, which I don't see being the case here. Do you?
The social taboo here is saying that a position taken by lots of highly educated people is nonsense because they're all locked in a dumb purity spiral that leads to motivated reasoning. This is actually one of societies biggest taboos! Look at what happens to people who make that argument publicly under their own name in other contexts; they tend to get fired and cancelled really fast.
> there's been a pattern of people confidently predicting AIs won't be able to perform various particular feats of the human mind (either fundamentally or in the next few decades) only to be proven wrong over increasingly shorter time-spans
That sword cuts both ways! There have been lots of predictions in the last decade that AI will contribute novel and hithertofore unknown solutions to things like climate change or curing cancer. Try getting GPT-4 to spit out a novel research-quality solution to anything, even a simple product design problem, and you'll find it can't.
> the claim that it seems far too obvious that we should
They're not arguing that. They're saying that AI risk proponents don't actually have good arguments, which is why they so regularly fall back on "it's so obvious we shouldn't need to explain why it's important". If your argument consists primarily of "everyone knows that" then this is a good indication you might be wrong.
OK, I completely agree that if you feel that I invoked "obviousness" in an attempt of browbeating you and the GP with what in fact is a social taboo, you should be extra skeptical (I'm not sure that was the point the GP was trying to make though).
> If your argument consists primarily of "everyone knows that" then this is a good indication you might be wrong.
It doesn't though, does it? There's strong empirical evidence that AI systems are making rapid progress in many domains that previously only humans were good at, and a pace that basically surprised almost everyone. I gave a list of arguments in another thread why AI is uniquely powerful and dangerous. Which of these do you disagree with and why?
Arguments like yours are very subjective. What is "rapid". What is "surprising". I don't find them particularly surprising myself - cool and awesome - but I was being amazed by language modelling ten years ago! The quality kept improving every year. It was clear that if that kept up eventually we'd have language models that could speak to like people.
So the idea of a surprising change of pace doesn't really hold up under close inspection. LLM capabilities do seem to scale linearly, with the idea of emergent abilities coming under robust attack lately. To the extent big LLMs are surprising to a lot of people this has happened primarily due to throwing a previously implausible quantity of money at building them, and OpenAI releasing one of them from their lab prison that other companies were keeping them in, not due to any major new breakthrough in the underlying tech. The progress is linear but the visibility of that progress was not. The transformers paper was 5 years ago and GPT-4 is basically an optimization of that tech combined with RL, just executed very carefully and competently. Transformers in turn were an improvement over prior language models that could speak like a human, they just weren't as good at it.
> It doesn't though, does it?
It does. Arguments that consist of "everyone knows that" are also called rumours or folk wisdom. It's fine to adopt widely held beliefs if those beliefs rest on something solid, but what we have here is a pure argument from authority. This letter is literally one sentence long and the only reason anyone cares is the list of signatories. It's very reliant on the observer believing that these people have some unique insight into AI risk that nobody else has, but there's no evidence of that and many signers aren't even AI researchers to begin with.
2/3 deep learning Turing Price winners (Hinton and Benigo) are sufficiently shell-shocked by the rate of progress to be thrown into existential doubts (Hinton is very explicit about the fact that progress is much faster than he thought just a few years ago, Benigo speaks of how an "unexpected acceleration" in AI systems has radically shifted his perspective). Plenty of knowledgable people in the field who were not previously AI doomers are starting to sound a lot more concerned very recently.
As to the "oh it's just linear scaling of out-of-sight tech" line, well of course that itself was suprising. Gwern pushed the scaling hypothesis earlier than many and from what I remember even got pretty nasty attacks from AI insiders from it. Here's what he wrote 3 years ago: "To the surprise of most (including myself), this vast increase in size did not run into diminishing or negative returns, as many expected, but the benefits of scale continued to happen as forecasted by OpenAI.".
So sure there's some subjectivity involved here, but I'd like to see your propose some reasonable operationalization of "surprise at progress" that didn't class most laymen and insiders as suprised.
>> It doesn't though, does it?
> It does.
We seem to be miscommunicating, what I was trying to express is that my argument does not really require any appeal to authority. Trusting your lying eyes (to evaluate the progress of stuff like midjourney) and judging the quality of arguments should be enough (I spelt some reasons out here https://news.ycombinator.com/item?id=36130482, but I think hackinthebochs makes the point better here: https://news.ycombinator.com/item?id=36129980).
In fact I would still be pretty concerned even if most top AI guys were like LeCun and thought there is no real risk.
I will not deny, of course, that the fact that well known reasearchers like Hinton and Benigo are suddenly much more alarmed than they previously were and the ones like LeCun who are not seem to mostly make exceptionally terrible arguments doesn't exactly make me more optimistic.
I agree these statements from these long term researchers about them being surprised by the rate of progress are surprising.
To clarify my own thinking here, it's totally reasonable to me that people are surprised if:
1. They weren't previously aware of AI research (surely 99% of the population?)
2. They were but had stopped paying attention because it was just a long series of announcements about cool tech demos nobody outside big corps could play with.
3. They were paying attention but thought scaling wouldn't continue to work.
My problem is that people like Sam Altman clearly aren't in any of those categories and Hinton shouldn't have been in any, although maybe he fell into (3). I personally was in (2). I wasn't hugely surprised that ChatGPT could exist because I'd seen GPT-2, GPT-1, I'd seen surprising AI demos at Google years earlier and so on. The direction things were going in was kinda clear. I was a bit surprised by its quality, but that's because I wasn't really paying close attention as new results were published and the last InstructGPT step makes such a big difference to how the tech is perceived. Actual knowledge doesn't change much but once it's housetrained, suddenly it's so much easier to interact with and use that it makes a step change in how accessible the tech is and how it's perceived.
I think I was more surprised by the joining of LLMs with generators and how well AI art works. It does feel like that happened fast. But, maybe I just wasn't paying attention again.
So I guess where we differ is that I don't take their surprise at face value. The direction was too clear, the gap between the threats they talk about in the abstract and the concrete proposals are too large and too lacking in obvious logical connection; it feels like motivated reasoning to me. I'm not entirely sure what's really going on and perhaps they are genuine in their concerns but if so it's hard to understand why they struggle so much to make a convincing case given they are certainly intellectually equipped to do so.
The two posts you linked are interesting and much better argued than the website this thread is about, so I'll reply to them directly.
It is possible to believe that AI poses threat, while also thinking that the AI safety organizations currently sprouting up are essentially grifts that will do absolutely nothing to combat the genuine threat. Especially when their primary goal seems to be the creation of well-funded sinecures for a group of like-minded, ideologically aligned individuals who want to limit AI control to a small group of wealthy technologists.
But as you can see yourself, there are countless people even here, in a technical forum, who claim that AI poses no plausible threat whatsoever. I fail to see how one can reasonably believe that.
If you look the the world politics, basically if you hold enough nuclear weapons, you can do whatever you want to those who don't have them.
And based on the "dangers", new countries are prohibit to create them. And the countries which were quick enough to create them, holds all the power.
Their value is immeasurable especially for the Russia. Without them, they could not attack to Ukraine.
> non-proliferation treaties were a scam
And yes, they mostly are right now. Russia has backed from them. There are no real consequences if you are backing off, and you can do it in any time.
The parent commenter is most likely saying, that now the selected parties hold the power of AI, they want to prevent others to gain similar power, while maintaining all the value by themselves.
> There are no real consequences if you are backing off, and you can do it in any time.
That's not quite true. Sure, noone is going to start a war about such a withdrawal. However, nuclear arsenals are expensive to maintain and it's even more expensive to be in an arms race. Also, nobody wants to risk nuclear war if they can avoid it. Civilian populations will support disarmament in times where they don't feel directly threatened. That's why lot of leaders of all persuasions have advocated for and taken part in efforts to reduce their arsenals. Same goes for relations between countries generally and the huge economic benefits that come with trade and cooperation. Withdrawing from nuclear treaties endangers all of these benefits and increases risk. A country would only choose this route out of desperation or for likely immediate gain.
I think it really depends.
E.g. from the Western perspective, only US, UK, France, Russia and China have signed the treaty from nuclear countries.
India or Pakistan are not part of the treaties and for some reason, we don't see big problems.
There is only China left who might leave the treaty in the first place, anymore.
And we are so dependent of the China, that there is no guarantee for consequences. Should we treat China then equally than India? What that means?
Also, leaving the treaty does not mean that countries start massively increasing their arsenal. There will be just a lack of inspections and information exchange.
Most of the credible threats I see from AI that don't rely on a lot of sci-fi extrapolation involve small groups of humans in control of massively powerful AI using it as a force multiplier to control or attack other groups of humans.
Sam Altman's proposal is to create precisely that situation with himself and a few other large oligarchs being the ones in control of the leading edge of AI. If we really do face runaway intelligence growth and god-like AIs then this is a profound amount of power to place in the hands of just a few people. Even worse it opens the possibility that such developments could happen partly in secret, so the public might not even know how powerful the secret AIs under command of the oligarchs have become.
The analogy with nuclear weapons is profoundly broken in lots of ways. Reasoning from a sloppy analogy is a great way to end up somewhere stupid. AI is a unique technology with a unique set of risks and benefits and a unique profile.
It's not clear at all that we have an avenue to super intelligence. I think the most likely outcome is that we hit a local maximum with our current architectures and end up with helpful assistants similar in capability to George Lucas's C3PO.
The scary doomsday scenarios aren't possible without an AI that's capable of both strategic thinking and long term planning. Those two things also happen to be the biggest limitations of our most powerful language models. We simply don't know how to build a system like that.
> We simply don't know how to build a system like that.
Yes, but ten years ago, we also simply didn't know how to build systems like the ones we have today! We thought it would take centuries for computers to beat humans at Go[1] and at protein folding[2]. We didn't know how to build software with emotional intelligence[3] and thought it would never make jokes[4]. There's been tremendous progress, because teams of talented researchers are working hard to unlock more aspects of what the human brain can do. Now billions of dollars are funding bright people to look for ways to build other kinds of systems.
"We don't know how to do it" is the security-through-obscurity argument. It means we're safe only as long as nobody figures this out. If you have a security mindset, it's not enough to hope that nobody finds the vulnerability. You need to show why they certainly will not succeed even with a determined search.
> It's not clear at all that we have an avenue to super intelligence
AI already beats the average human on pretty much any task people have put time into, often by a very wide margin and we are still seeing exponential progress that even the experts can't really explain, but yes, it is possible this is a local maximum and the curve will become much flatter again.
But the absence of any visible fundamental limit on further progress (or can you name one?) coupled with the fact that we have yet barely begun to feel the consequences of the tech we already have (assuming zero breakthroughs from now on) makes we extremely wary to conclude that there is no significant danger and we have nothing to worry about.
Let's set aside the if and when of a super intelligence explosion for now. We are ourselves an existence proof of some lower bound of intelligence, that if amplified by what computers can already do (like perform many of the things we used to take intellectual pride in much better, and many orders of magnitude faster with almost infinitely better replication and coordination ability) seems already plenty dangerous and scary to me.
> The scary doomsday scenarios aren't possible without an AI that's capable of both strategic thinking and long term planning. Those two things also happen to be the biggest limitations of our most powerful language models. We simply don't know how to build a system like that.
Why do you think AI models will be unable to plan or strategize? Last I checked languages models weren't trained or developed to beat humans in strategic decision making, but humans already aren't doing too hot right now in games of adversarial strategy against AIs developed for that domain.
I dispute this. What appears to be exponential progress is IMO just a step function that made some jumps as the transformer architecture was employed on larger problems. I am unaware of research that moves beyond this in a way that would plausibly lead to super-intelligence. At the very least I foresee issues with ever-increasing computational requirements that outpace improvements in hardware.
We’ll see similar jumps when other domains begin employing specialized AI models, but it’s not clear to me that these improvements will continue increasing exponentially.
Right, and if someone can join the two, that could be something genuinely formidable. But does anyone have a credible path to joining the different flavors to produce a unity that actually works?
Even if someone will, I don't think it's an "existential risk". So, yes, I'm willing to make the bet. I'm also willing to make the bet that Santa never delivers nuclear warheads instead of presents. It's why I don't cap my chimney every Christmas Eve.
Between Covid, bank failures, climate change, and AI, it's like everyone is looking for something to be in a panic about.
>It's not clear at all that we have an avenue to super intelligence.
All problems in reality are probability problems.
If we don't have a path to superintelligence, then the worst problems just don't manifest themselves.
If we do have a path to super intelligence then the doomsday scenarios are nearly a certainty.
It's not really any different than saying "A supervolcano is unlikely to go off tomorrow, but if a supervolcano does go off tomorrow it is a doomsday scenario".
>We simply don't know how to build a system like that.
You are already a superintelligence when compared to all other intelligences on earth. Evolution didn't need to know how to build a system like that, and yet it still reached this point. And there is not really any to believe humanity is the pinnacle of intelligence, we are our own local maxima of power/communication limitations. An intelligence coupled with evolutionary systems design is much more apt to create 'super-' anything than the random walk alone.
Why are doomsday scenarios are certainty then. What's the model to get to that that isn't just some sort of scary story that waves away or into existence a lot of things we don't know if they can exist.
Let's say I was a small furry mammal that tasted really good, but also for some reason understood the world as it is now.
I would tell you that super intelligence had already happened. That super intelligence was humans. That humans happened to reach super intelligence by 1) having the proper hardware. 2) filtering noise from important information. 3) then sharing that information with others to amplify the power of intelligence 4) having a toolkit/tools to turn that information into useful things. 5) And with all that power humans can kill me off in mass, or farm me for my tasty meat at their leisure with little to nothing that I can do about it.
There doesn't appear to be any more magic than that. All these things already exist in biological systems that elevated humans far above their warm blooded peers. When we look at digital systems we see they are designed to communicate. You don't have an ethernet jack as a person. You can't speak the protocol to directly drive a 3 axis mill to produce something. Writing computer code is a pain in the ass to most of us. We are developing a universal communication intelligence, that at least in theory can drive tools at a much higher efficiency than humans will ever be able to.
Coming back to point 5. Cats/dogs are the real smart ones here when dealing with superintelligences. Get domesticated by the intelligence so they want to keep you around as a pet.
Do you think we could wipe out all furry mammals, for example? Could another intelligence have the same level of difference to us as in your story we to furry mammals? We don't even know if the mythical superintelligence could manifest the way you assume. It assumes that intelligence basically can overcome any obstacles - I'd say we actually see that seems not to be the case currently and claims that that is just a function of sufficient intelligence are unproven (setting aside physical limits to certain actions and results).
>Do you think we could wipe out all furry mammals, for example?
lets go with over a particular size. Lets say larger than the biggest rat. In that case yes, very easily. Once you get to rats it becomes far more difficult and you're pretty much just destroying the biosphere at that point.
> It assumes that intelligence basically can overcome any obstacles
In the case of human extinction, no, a super intelligence would not have to overcome any obstacles, it would just have to overcome obstacles better than we did.
Also, the superintelligence doesn't just have to overcome obstacles better than we did, it needs to overcome the right obstacles to succeed with human extinction.
We don't need an avenue to super-intelligence. We just need a system that is better at manipulating human beliefs and behaviour than our existing media, PR, and ad industries.
The problem is not science fiction god-mode digital quetta-smart hypercomputing.
This is about political, social, and economic influence, and who controls it.
That risk isn't about AI-as-AI. That risk is about AI-as-better-persuasive-nonsense-generator. But the same risk is there for any better-persuasive-nonsense-generator, completely independent from whether it's an AI.
It's the most persuasive actual risk I've seen so far, but it's not an AI-specific risk.
Effective dystopian mass-manipulation and monitoring are a real concern and we're closer to it[1] than to super intelligence. But super-intelligence going wrong is almost incomparably worse. So we should very much worry about it as well.
[1] I'm not even sure any further big breakthroughs in AI are needed, i.e. just effective utilization of existing architectures probably already suffices.
A super intelligent AI is not necessary for AI to be an threat. Dumb AIs that are given access to the internet plus a credit card and told to maximize profit could easily cause massive damage. We are not far from such an AI being accessible to the masses. You can try to frame this like the gun debate "it's not the AI it's the people using it" but the AI would be acting autonomously here. I have no faith that people won't do extremely risky things if given the opportunity.
It isn't obvious to me. And I've yet to read something that spills out the obvious reasoning.
I feel like everything I've read just spells out some contrived scenario, and then when folks push back explaining all the reasons that particular scenario wouldn't come to pass, the counter argument is just "but that's just one example!" without offering anything more convincing.
Do you have any better resources that you could share?
The history of humanity is replete with examples of the slightly more technologically advanced group decimating their competition. The default position should be that uneven advantage is extremely dangerous to those disadvantaged. This idea that an intelligence significantly greater than our own is benign just doesn't pass the smell test.
From the tech perspective: higher order objectives are insidious. While we may assume a narrow misalignment in received vs intended objective of a higher order nature, this misalignment can result in very divergent first-order behavior. Misalignment in behavior is by its nature destructive of value. The question is how much destruction of value can we expect? The machine may intentionally act in destructive ways as it goes about carrying out its slightly misaligned higher order objective-guided behavior. Of course we will have first-order rules that constrain its behavior. But again, slight misalignment in first-order rule descriptions are avenues for exploitation. If we cannot be sure we have zero exploitable rules, we must assume a superintelligence will find such loopholes and exploit them to maximum effect.
Human history since we started using technology has been a lesson on the outcome of an intelligent entity aimed at realizing an objective. Loopholes are just resources to be exploited. The destruction of the environment and other humans is just the inevitable outcome of slight misalignment of an intelligent optimizer.
If this argument is right, the only thing standing between us and destruction is the AGI having reached its objective before it eats the world. That is, there will always be some value lost in any significant execution of an AGI agent due to misalignment. Can we prove that the ratio of value created to value lost due to misalignment is always above some suitable threshold? Until we do, x-risk should be the default assumption.
OK, which of the following propositions do you disagree with?
1. AIs have made rapid progress in approaching and often surpassing human abilities in many areas.
2. The fact that AIs have some inherent scalability, speed, cost, reliability and compliance advantages over humans means that many undesirable things that could previously not be done at all or at least not done at scale are becoming both feasible and cost-effective. Examples would include 24/7 surveillance with social desirability scoring based on a precise ideological and psychological profile derived from a comprehensive record of interactions, fine-tuned mass manipulation and large scale plausible falsification of the historical record. Given the general rise of authoritarianism, this is pretty worrying.
3. On the other hand the rapid progress and enormous investment we've been seeing makes it very plausible that before too long we will, in fact, see AIs that outperform humans on most tasks.
4. AIs that are much smarter than any human pose even graver dangers.
5. Even if there is a general agreement that AIs pose grave or even existential risks, states, organizations and individuals will are all incentivized to still seek to improve their own AI capabilities, as doing so provides an enormous competitive advantage.
6. There is a danger of a rapid self-improvement feedback loop. Humans can reproduce, learn new and significantly improve existing skills, as well as pass skills on to others via teaching. But there are fundamental limits on speed and scale for all of these, whereas it's not obvious at all how an AI that has reached super-human level intelligence would be fundamentally prevented from rapidly improving itself further, or produce millions of "offspring" that can collaborate and skill-exchange extremely efficiently. Furthermore, since AIs can operate at completely different time scales than humans, this all could happen extremely rapidly, and such a system might very quickly become much more powerful than humanity and the rest of AIs combined.
I think you only have to subscribe a small subset of these (say 1.&2.) to conclude that "AI is an uniquely powerful and thus uniquely dangerous technology" obviously follows.
For the stronger claim of existential risk, have you read the lesswrong link posted elsewhere in this discussion?
Reading the lesswrong link, the parts I get hung up on are that it appears in these doomsday scenarios humans lose all agency. Like, no one is wondering why this computer is placing a bunch of orders to DNA factories?
Maybe I’m overly optimistic about the resilience of humans but these scenarios still don’t sound plausible to me in the real world.
> Like, no one is wondering why this computer is placing a bunch of orders to DNA factories?
I'm not that confident that if we put you in a box, tron-style, where you basically continued to enjoy your existing level of intelligence, but think 10'000x faster, have Petabytes of information at your fingertips and can clone yourself and losslessly and rapidly exchange knowledge with your clones and had a few days to think about it (~a few thousand years of thought at your normal speed) you couldn't figure out a way to effect a bunch of orders to DNA factories without anyone raising an alarm.
Are you?
Now what if we actually consider an actual AI after a few self-improvement steps. Any reasons to expect it wouldn't be 10'000x+ smarter than you as well, or roughly the difference in intelligence between you and an ant? Could you outsmart a bunch of ultra-ultra-slow-motion ants?
You could place a bunch of orders into services that syntetize DNA or proteins rn.
Some people are even working on stuff like automating protein design whith AI.
There's no reason why humans should notice anything word about a particular order on a service like that.
> 3. ... before too long we will ... see AIs that outperform humans on most tasks.
This is ambiguous. Do you mean
A. that there is some subset T1 of the set of all tasks T such that T1 is "most of" T, and that for each P in T1 there will be an AI that outperforms humans on P, or
B. There will be a single AI that outperforms humans on all tasks in a set T1, where T1 is a subset of all tasks T such that T1 is "most of" T?
I think A is unlikely but plausible but I don't see cause for worry. I don't see any reason why B should come to pass.
4. AIs that are much smarter than any human pose even graver dangers.
Sure. Why should we believe they will ever exist though?
I think between point 3 and 4 there is a leap to talking about “danger”. Perhaps the disagreement is about what one calls “danger”. I had perhaps mistakenly assumed we were talking about an extinction risk. I’ll grant you concerns about scaling up things like surveillance but there is a leap to being an existential risk that I’m still not following.
AI will not have the instinctual drives for domination or hunger that humans do.
It seems likely that the majority of AI projects will be reasonably well aligned by default, so I think 1000 AIs monitoring what the others are doing is a lot safer than a single global consortium megaproject that humans can likely only inadequately control.
The only reasonable defense against rogue AI is prosocial AI.
Replying here, crossing over from the other thread.
Where we depart is point 4. Actually, both point 3 and 4 are things I agree with, but it's implied there's a logical link or progression between them and I don't think there is. The problem is the definitions of "outperform humans" and "smart".
Current AI can perform at superhuman levels in some respects, yes. Midjourney is extremely impressive when judged on speed and artistic skill. GPT-4 is extremely impressive whilst judged on its own terms, like breadth of knowledge. Things useful to end users, in other words. LLMs are deeply unimpressive judged on other aspects of human intelligence like long term memory, awareness of time and space, ability to learn continuously, willingness to commit to an opinion, ability to come up with interesting new ideas, hide thoughts and all that follows from that like being able to make long term plans, have agency and self-directed goals etc ... in all these areas it is weak. Yet, most people would incorporate most of them into their definition of smart.
Will all these problems be solved? Some will, surely, but for others it's not entirely clear how much demand there is. Boston Robotics was making amazing humanoid parkour bots for years yet the only one they seem able to actually sell is dog-like. Apparently the former aren't that useful. The unwillingness to commit to an opinion may be a fundamental trait of AI for as long as it's centralized, proprietary and the masses have to share a single model. The ability to come up with interesting new ideas and leaps of logic may or may not appear, it's too early to tell.
But between 3 and 4 you make a leap and assume that not only will all those areas be conquered very soon, but that the resulting AI will be unusually dangerous. The various social ills you describe don't worry me though. Bad governments will do bad things, same old, same old. I'm actually more worried about people using the existence of AI to deny true evidence rather than manufacture false evidence en-masse. The former is a lot of work and people are lazy. COVID showed that people's capacity for self-deception is unlimited, their willingness to deny the evidence of their own eyes is bottomless as long as they're told to do it by authority figures. You don't even need AI to be abused at all for someone to say, "ignore that evidence that we're clueless and corrupt, it was made by an AI!"
Then by point 6 we're on the usual trope of all public intellectuals, of assuming unending exponential growth in everything even when there's no evidence of that or reason to believe it. The self-improving AI idea is so far just a pipe dream. Whilst there are cases where AI gets used to improve AI via self-play, RLHF and so on, it's all very much still directed by humans and there's no sign that LLMs can self improve despite their otherwise impressive abilities. Indeed it's not even clear what self-improvement means in this case. It's a giant hole marked "??? profit!" at the heart of the argument. Neurosurgeons can't become superintelligences by repeatedly performing brain surgery on themselves. Why would AI be different?
Nuclear missiles present an obvious danger to the human body. AI is an application of math. It is not clear how that can be used directly to harm a body.
The assumption seems to be that said math will be coupled with something like a nuclear missile, but in that case the nuclear missile is still the threat. Any use of AI is just an implementation detail.
Germany, for example would disagree with you. They believe violent speech is an act of violence in itself.
>AI is an application of math.
It turns out that people hook computers to 'things' that exist in the physical world. You know like robot bodies, or 3D printers. And as mentioned above, even virtual things like social media can cause enough problems. People hook AI to tools.
And this is just the maybe not quite general AI we have now. If and when we create a general AI that with self-changing feedback loops then all this "AI is just a tool" asshattery goes out the window.
Remember at the end of the day, you're just an application of chemistry that is really weak without your ability to use tools and to communicate.
> It turns out that people hook computers to 'things' that exist in the physical world.
But those physical things would be the danger, at least if you consider the nuclear missile to be the danger. It seems you are trying to go down the "guns don't kill people, people kill people" line of thinking. Which is fine, but outside of the discussion taking place.
There are many relevant things that already exist in the physical world and are not currently considered dangers: ecommerce, digital payments, doordash-style delivery, cross-border remittances, remote gig work, social media fanning extreme political views, event organizing.
However, these are constituent elements that could be aggregated and weaponized by a maleficent AI.
Maleficent humans are constantly trying to use these elements for their own gain, often with little to no regards to other humans (especially out groups). This happens both individually, in small groups, in large organizations and even multiple organization colluding. Both criminal, terrorist, groups at war, along with legal organizations such as exploitative companies and regressive interest organizations, et.c.. And we have tools and mechanisms in place to keep the level of abuse at bay. Why and how are these mechanisms unsuitable for protecting against AI?
>Why and how are these mechanisms unsuitable for protecting against AI?
The rule of law prevented WWI and WWII, right? Oh, no it did not, tens to hundreds of millions died due to human stupidity and violence depending on what exactly you count in that age.
> Both criminal, terrorist, groups at war
Human organizations, especially criminal organizations have deep trust issues between agents in the organization. You never know if anyone else in the system is a defector. This reduces the openness and quantity of communication between agents. In addition you have agents that want to personally gain rather than benefit the organization itself. This is why Apple is a trillion dollar company following the law... mostly. Smart people can work together and 'mostly' trust the other person isn't going to screw them over.
Now imagine a superintelligent AI with a mental processing bandwidth of hundreds of the best employees at a company. Assuming it knows and trusts itself, then the idea of illegal activities being an internal risk disappears. You have something that operates more on the level of a hivemind toward a goal (what the limitations of hivemind versus selfish agents are is another very long discussion). What we ask here is if all the worlds best hackers got together, worked together unselfishly, and instigated an attack against every critical point they could find on the internet/real world systems at once, how much damage could they cause?
Oh, lets say you find the server systems the super intelligence is on, but the controller shuts it off and all the data has some kind of homomorphic encryption so that's useless to you. It's dead right? Na, they just load up the backup copy they have a few months later and it's party time all over again. Humans tend to remain dead after dying, AI? Well that is yet to bee seen.
Those tangible elements would conceivably become the danger, not the AI using those elements. Again, the "guns don't kill people, people kill people" take is all well and good, but well outside of this discussion.
False premise. One can start new threads about complimentary subjects and they can be thought about in parallel. You don't have to try and shove all of the worlds concepts into just one thought train to be able to reason about them. That's how you make spaghetti.
>It seems you are trying to go down the "guns don't kill people, people kill people" line of thinking.
"Guns don't kill people, AIs kill people" is where we are going, I think. This is the discussion: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."
The discussion is not about a mathematical representation of AI. The discussion is about the actual implementation of AI on physical computing infrastructure which is accessible by at least one human on planet earth.
The credible danger, argued in various places, including superintelligence by Nick Bostrom, is that the "system under review" here is "every physical system on planet earth" because an AI could gain access to whatever systems exist on said planet, including human minds (see "Nazis").
So much as we might discuss the problems of letting a madman get control of the US, Russian, UK, French or Chinese nuclear arsenals, we might discuss the problem of building an AI if the act of building the AI could result in it taking over the nuclear arsenals of those countries and using it against humans. That takeover might involve convincing a human it should do it.
I don't understand this argument (the "Terminator" scenario). AI could conceivably self replicate and evolve as software but it still needs hardware to run on, power, networking etc etc. There's no better way to kill that than to start a nuclear war or engineer some super virus that kills everyone.
It's hard to see any scenario where AI could become a dominant force without significant human collaboration. Perhaps somewhere like North Korea where a very small elite has complete control over the the population it could happen but it sounds a lot like sci-fi to me. I'd love to hear some plausible scenarios for the counter-argument. I've seen a lot of "I think there's an x% chance we're in trouble" arguments which might be convincing for job losses, but I don't find at all plausible as a case for human extinction or indentured servitude (the "Matrix" scenario).
Sure. I recommend reading superintelligence by Nick Bostrom.
But I think the key failure here, in your thinking, is that you can't conceive of something, and that therefore something can't happen, and you're doing that in the context of things that are smarter than you. You being able to conceive of it is simply not required for it to happen.
Also, Colossus: The Forbin Project was a great movie from 1970 on this subject. Spoilers: an AI takes control of the nuclear arsenal and then threatens humans with extinction if they do not serve it. The humans do serve it, of course, because the humans in charge don't want to die, and are entirely fine with enslaving the rest of us.
The book superintelligence by Nick Bostrom gets into the fine details of all the different ways an AI would escape, why it would escape, and why it wouldn't take a chance with a species that murders its own kind for fun and profit.
Superintelligence and Life 3.0 seem to come up as recurring references in the discussion. I've only read synopses of both but frankly I find the argument that melevolent AI "escape" could occur without being noticed and thwarted a bit far fetched.
There's a good counterargument here[1] that seems reasonable:
"One of the features of intelligence explosion that most preoccupies Bostrom and Yudkowsky is that it’s not a problem that we get to have many attempts at. In the Terminator movies, humans don’t get to approach a newly self-aware Skynet and request a do over. One minute Skynet is uncomplainingly complying with all human directives. The next, it’s nuking us. I suspect that we are likely to have plenty of opportunities for do overs in our attempts to make autonomous AIs. Autonomy is not an all-or-nothing proposition. The first machine agents are likely to be quite clumsy. They may be capable of forming goals in respect of their world but they won’t be particularly effective at implementing them. This gives us plenty of opportunity to tweak their programming as they travel the path from clumsy to sophisticated agency"
Ok - fair comment. So I found a copy of Superintelligence. The argument is pulp sci-fi at best:
"The final phase begins when the AI has gained sufficient strength to obviate the need for secrecy. The AI can now directly implement its objectives on a full scale. The overt implementation phase might start with a “strike” in which the AI eliminates the human species and any automatic systems humans have created that
could offer intelligent opposition to the execution of the AI’s plans. This could be achieved through the activation of some advanced weapons system that the AI has
perfected using its technology research superpower and covertly deployed in the covert preparation phase. If the weapon uses self-replicating biotechnology or nano-technology, the initial stockpile needed for global coverage could be microscopic: a single replicating entity would be enough to start the process. In order to ensure a sudden and uniform effect, the initial stock of the replicator might have been deployed or allowed to diffuse worldwide at an extremely low, undetectable con-centration. At a pre-set time, nanofactories producing nerve gas or target- seeking mosquito-like robots might then burgeon forth simultaneously from every square meter of the globe (although more effective ways of killing could probably be devised by a machine with the technology research superpower)."
Look - nothing's impossible, but I agree with the counter argument that an advanced AI still starts as a "brain in a vat", with no experience of agency in the physical world. In order to successfully take over you have to assume it can develop all the physical world capability it needs, in secret and get it right first time. That seems implausible.
Exactly. While there is an argument to be made that people are the real danger, that is beyond the discussion taking place. It has already been accepted, for the sake of discussion, that the nuclear missile is the danger, not the math which developed the missile, nor the people who thought it was a good idea to use a missile. Applying AI to the missile still means the missile is the danger. Any use of AI in the scope of that missile is just an implementation detail.
You said that "AI is an application of math. It is not clear how that can be used directly to harm a body." I was trying to illustrate the case that if humans can develop harmful things, like nuclear weapons, then an AI that is as smart as a human can presumably develop similarly harmful things.
If the point you are trying to make is that an AI which secretly creates and deploys nuclear, biological, or chemical weapons in order to destroy all of humanity, is not an "AI risk" because it's the weapons that do the actual harm, then... I really don't know what to say to that. Sure, I guess? Would you also say that drunk drivers are not dangerous, because the danger is the cars that they drive colliding into people's bodies, and the drunk driver is just an implementation detail?
> I was trying to illustrate the case that if humans can develop harmful things, like nuclear weapons, then an AI that is as smart as a human can presumably develop similarly harmful things.
For the sake of discussion, it was established even before I arrived that those developed things are the danger, not that which creates/uses the things which are dangerous. What is to be gained by ignoring all of that context?
> I really don't know what to say to that. Sure, I guess?
Nothing, perhaps? It is not exactly something that is worthy of much discussion. If you are desperate for a fake internet battle, perhaps you can fight with earlier commenters about whether it is nuclear missiles that are dangerous or if it is the people who have created/have nuclear missiles are dangerous? But I have no interest. I cannot think of anything more boring.
I'm specifically worried that an AGI will conceal some instrumental goal of wiping out humans, while posing as helpful. It will helpfully earn a lot of money for a lot of people, by performing services and directing investments, and with its track record, will gain the ability to direct investments for itself. It then plows a billion dollars into constructing a profitable chemicals factory somewhere where rules are lax, and nobody looks too closely into what else that factory produces, since the AI engineers have signed off on it. And then once it's amassed a critical stockpile of specific dangerous chemicals, it releases them into the atmosphere and wipes out humanity / agriculture / etc.
Perhaps you would point out that in the above scenario the chemicals (or substitute viruses, or whatever) are the part that causes harm, and the AGI is just an implementation detail. I disagree, because if humanity ends up playing a grand game of chess against an AGI, the specific way in which it checkmates you is not the important thing. The important thing is that it's a game we'll inevitably lose. Worrying about the danger of rooks and bishops is to lose focus on the real reason we lose the game: facing an opponent of overpowering skill, when our defeat is in its interests.
Cool, I guess. While I have my opinions too, I'm not about to share them as that would be bad faith participation. Furthermore, it adds nothing to the discussion taking place. What is to be gained by going off on a random tangent that is of interest to nobody? Nothing, that's what.
To bring us back on topic to try and salvage things, it remains that it is established in this thread that the objects of destruction are the danger. AI cannot be the object of destruction, although it may be part of an implementation. Undoubtedly, nuclear missiles already utilize AI and when one talks about the dangers of nuclear missiles they are already including AI as part of that.
Yes, but usually when people express concerns about the danger of nuclear missiles, they are only thinking of those nuclear missiles that are at the direction of nation-states or perhaps very resourceful terrorists. And their solutions will usually be directed in that direction, like arms control treaties. They aren't really including "and maybe a rogue AI will secretly build nuclear weapons on the moon and then launch them at us" in the conversation about the danger of nukes and the importance of international treaties, even though the nukes are doing the actual damage in that scenario. Most people would categorize that as sounding more like an AI-risk scenario.
Please read Life 3.0 or superintelligence. There are people that spent decades thinking about how this would happen. You spent a little bit of time and conclude it can't.
> It's like saying nuclear missiles can't possibly be dangerous and nuclear arms reduction and non-proliferation treaties were a scam, because the US, China and the Soviet Union had positioned themselves to capture the majority of the strategic value nukes bring.
I'm honestly not sure if this is sarcasm. The non-proliferation treaties are indeed a scam. The war is raging between the US and Russia and nuclear is a big part of it (though just words/threats for now). It's nonsensical to think that these treaties are possible.
Not only is the Non proliferation treaty possible, it's been evidently effective in slowing the proliferation of nuclear weapons. The only country that ratified or acceded to it and went on to develop nuclear weapons is North Korea, and the only country that ratified or acceded to it and looks on track to develop nuclear weapons is Iran. One country that was not a signatory and developed nuclear weapons voluntarily gave them up and acceded to it partly due to international pressure (South Africa). Israel, Pakistan and India have since developed nuclear weapons but they were never signatories, the only other non-signatory is South Sudan which probably won't acquire nuclear capabilities anytime soon.
And I don't get the opposed mindset, that AI is suddenly going to "become a real boy, and murder us all".
Isn't it a funny coincidence how the popular opinion of AIs aligns perfectly with blockbusters and popular media ONLY? People are specifically wanting to prevent Skynet.
The kicker (and irony to a degree) is that I really want sapient AI to exist. People being so influenced by fiction is something I see as a menace to that happening in my lifetime. I live in a world where the majority is apparently Don Quixote.
- Point one: If the sentient AI can launch nukes, so can your neighbor.
- Point zwei: Redistributing itself online to have unlimited compute resources is a fun scenario but if networks were that good then Stadia wouldn't have been a huge failure.
- Point trois: A distributed-to-all-computers AI must have figured out universal executables. Once we deal with the nuclear winter, we can plagiarize it for ourselves. No more appimage/snap/flatpak discussions! Works for any hardware! No more dependency issues! Works on CentOS and Windows from 1.0 to 11! (it's also on AUR, of course.)
- Point cuatro: The rogue AI is clearly born as a master hacker capable of finding your open ports, figure out any exploits or create 0-day exploits to get in, and hope there's enough resources to get the payload injected, then pray no competent admin is looking at the thing.
- Point go: All of this rides on the assumption that the "cold, calculating" AI has the emotional maturity of a teenager. Wait, but that's not what "cold, calculating" means, that's "hothead and emotional". Which is it?
- Point six: Skynet lost, that's the point of the first movie's plot. If everyone is going to base their beliefs after a movie, at least get all the details. Everything Skynet did after the first attack was full of boneheaded decisions that only made the situation worse for it, to the point the writers cannot figure ways to bring Skynet back anymore because it doomed itself in the very first movie. You should be worrying about Legion now, I think. It shuts down our electronics instead of nuking.
Considering it won't have the advantage of triggering a nuclear attack because that's not how nukes work, the evil sentient AI is so doomed to fail it's ridiculous to think otherwise.
But, companies know this is how the public works. They'll milk it for all it's worth so only a few companies can run or develop AIs, maybe making it illegal otherwise, or liable for DMCAs. Smart business move, but it affects my ability to research and use them. I cannot cure people's ability to separate reality and fiction though, and that's unfortunate.
A counter point here is you're ignoring all the boring we all die scenarios that are completely possible but too boring to make a movie about.
The AI hooked to a gene sequencer/printer test lab is something that is nearly if not completely possible now. It's something that can be relatively small in size compared with the facilities needed to make most weapons of mass destruction. It's something that is highly iterative, and parallelizable. And it's something powerful enough that if targeting at the correct things (kill all rice, kill all X people) that it easily spills over in to global conflict.
You're a priori writing off my comment as fruitless because of your emotions and not because you actually have given it deep thought and carefully reached the conclusion that social feedback is somehow bad.
Also, the notion that "people's work" is inherently worthy of respect is just nonsensical. I do shoddy work all the time. Hell, you just casually dismissed my internet comment work as shallow and told me not to do it. Please don't post a shallow dismissal of my work.
Don't you think that this is all a bit anti-intellectual?
General research into AI alignment does not require that those models are controlled by few corporations. On the contrary, the research would be easier with freely available very capable models.
This is only helpful in that a superintelligence well aligned to make Sam Altman money is preferable to a superintelligence badly aligned that ends up killing humanity.
It is fully possible that a well aligned (with its creators) superintelligence is still a net negative for humanity.
If you consider a broader picture, unleashing a paperclip-style cripple AI (aligned to rising $MEGACORP profit) on the Local Group is almost definitely worse for all Local Group inhabitants than annihilating ourselves and not doing that.
Is more research really going to offer any true solutions? I’d be genuinely interested in hearing about what research could potentially offer (the development of tools to counter AI disinformation? A deeper understanding of how LLMs work?), but it seems to me that the only “real” solution is ultimately political. The issue is that it would require elements of authoritarianism and censorship.
A lot of research about avoiding extinction by AI is about alignment. LLMs are pretty harmless in that they (currently) don't have any goals, they just produce text. But at some point we will succeed in turning them into "thinking" agents that try to achieve a goal. Similar to a chess AI, but interacting with the real world instead. One of the big problems with that is that we don't have a good way to make sure the goals of the AI match what we want it to do. Even if the whole "human governance" political problem were solved, we still couldn't reliably control any AI. Solving that is a whole research field. Building better ways to understand the inner workings of neural networks is definitely one avenue
Intelligence cannot be 'solved', I would go on to further say that an intelligence without the option of violence isn't an intelligence at all.
If you suddenly wanted to kill people, for example, then could probably kill a few before you were stopped. That is typically the limits of an individuals power. Now, if you were a corporation with money, depending on the strategy you used you could likely kill anywhere from hundreds to hundreds of thousands. Kick it up to government level, and well, the term "just a statistic" exists for a reason.
We tend to have laws around these behaviors, but they are typically punitive. The law realizes that humans, and human systems will unalign themselves from "moral" behavior (whatever that may be considered at the time). When the lawgiver itself becomes unaligned, well, things tend to get bad. Human alignment typically consists of benefits (I give you nice things/money/power) or violence.
I see. Thanks for the reply. But I wonder if that’s not a bit too optimistic and not concrete enough. Alignment won’t solve the world’s woes, just like “enlightenment” (a word which sounds a lot like alignment and which is similarly undefinable) does not magically rectify the realities of the world. Why should bad actors care about alignment?
Another example is climate change. We have a lot of good ideas which, combined, would stop us from killing millions of people across the world. We have the research - is more “research” really the key?
> I have yet to see a solution for “AI safety” that doesn’t involve ceding control of our most powerful models to a small handful of corporations.
That's an excellent point.
Most of the near-term risks with AI involve corporations and governments acquiring more power. AI provides power tools for surveillance, oppression, and deception at scale. Those are already deployed and getting better. This mostly benefits powerful organizations. This alarm about strong AI taking over is a diversion from the real near-term threat.
With AI, Big Brother can watch everything all the time. Listen to and evaluate everything you say and do. The cops and your boss already have some of that capability.
Is something watching you right now through your webcam? Is something listening to you right now through your phone? Are you sure?
Ok, so if we take AI safety / AI existential risk as real and important, there are two possibilities:
1) The only way to be safe is to cede control to the most powerful models to a small group (highly regulated corporations or governments) that can be careful.
2) There is a way to make AI safe without doing this.
If 1 is true, then... sorry, I know it's not a very palatable solution, and may suck, but if that's all we've got I'll take it.
If 2 is true, great. But it seems less likely than 1, to me.
The important thing is not to unconsciously do some motivated reasoning, and think that AGI existential risk can't be a big deal, because if it is, that would mean that we have to cede control over to a small group of people to prevent disaster, which would suck, so there must be something else going on, like these people just want power.
I just don't see how the genie is put back in the bottle. Optimizations and new techniques are coming in at a breakneck pace, allowing for models that can run on consumer hardware.
I think it could be done. Or rather, instead of putting the genie back in the bottle, we could slow it down enough that we figure out how to ask it for wishes in a way that avoids all the monkey-paw's scenarios.
Dropping the metaphor, running today's models isn't dangerous. We could criminalize developing stronger ones, and make a "Manhattan project" for AI aimed at figuring out how to not ruin the world with it. I think a big problem is what you point out -- once it's out, it's hard to prevent misuse. One bad AGI could end up making a virus that does massive damage to humanity. We might end up deciding that this tech is just too dangerous to be allowed to happen at all, at least until after humanity manages to digitize all our brains or something. But it's better to try to slow down as much as we can, for as long as we can, than to give up right from the get-go and wing it.
Honestly, if it turns out that China ends up developing unsafe AI before we develop safe AI, I doubt it would have turned out much better for the average American if America were the ones to develop unsafe AI first. And if they cut corners and still manage to make safe AI and take over the world, that still sounds a heck of a lot better than anyone making unsafe AI.
There is a way, in my opinion: distribute AI widely and give it a diversity of values, so that any one AI attempting takeover (or being misused) is opposed by the others. This is best achieved by having both open source and a competitive market of many companies with their own proprietary models.
Personalization, customization, etc.: by aligning AI systems to many users, we benefit from the already-existing diversity of values among different people. This could be achieved via open source or proprietary means; the important thing is that the system works for the user and not for whichever company made it.
It's difficult as most of the risk can be reinterpreted as a highly advanced user.
But that is where some form of hard personhood zero proof mechanism NEEDS to come in. This can then be used in conjunction with a Ledger used to track deployment of high spec models. And create an easy means to Audit and deploy new advanced tests to ensure safety.
Really what everyone also need to keep in mind at the larger scale is that final turing test with no room for deniability. And remember all those Sci-fi movies and how that Moment is portrayed traditionally.
I have one: Levy fines on actors judged to be attempting to extend AI capabilities beyond the current state of the art, and pay the fine to those private actors who prosecute them.
tl;dr: significant near term AI risk is real and comes from the capacity for imagined ideas, good and evil, to be autonomously executed on by agentic AI, not emergent superintelligent aliens. To de-risk this, we need to align AI quickly, which requires producing new knowledge. To accelerate the production of this knowledge, the government should abandon decelerationist policies and incentivize incremental alignment R&D by AI companies. And, critically, a new public/private research institution should be formed that grants privileged, fully funded investigators multi-year funding cycles with total scientific freedom and access to all state-of-the-art artificial intelligence systems operating under US law to maximize AI as a force multiplier in their research.
While I'm not on this "who's-who" panel of experts, I call bullshit.
AI does present a range theoretical possibilities for existential doom, from teh "gray goo" and "paperclip optimizer" scenarios to Bostrom's post-singularity runaway self-improving superintelligence. I do see this as a genuine theoretical concern that could even potentially even be the Great Filter.
However, the actual technology extant or even on the drawing boards today is nothing even on the same continent as those threats. We have a very vast ( and expensive) sets of probability-of-occurrence vectors that amount to a fancy parlor trick that produces surprising and sometimes useful results. While some tout the clustering of vectors around certain sets of words as implementing artificial creation of concepts, it's really nothing more than an advanced thesaurus; there is no evidence of concepts being weilded in relation to reality, tested for truth/falsehood value, etc. In fact, the machines are notorious and hilarious for hallucinating with a highly confident tone.
We've created nothing more than a mirror of human works, and it displays itself as an industrial-scale bullshit artist (where bullshit is defined as expressions made to impress without care one way or the other for truth value).
Meanwhile, this panel of experts makes this proclamation with not the slightest hint of what type of threat is present that would require any urgent attention, only that some threat exists that is on the scale of climate change. They mention no technological existential threat (e.g., runaway superintelligence), nor any societal threat (deepfakes, inherent bias, etc.). This is left as an exercise for the reader.
What is the actual threat? It is most likely described in the Google "We Have No Moat" memo[0]. Basically, once AI is out there, these billionaires have no natural way to protect their income and create a scaleable way to extract money from the masses, UNLESS they get cooperation from politicians to prevent any competition from arising.
As one of those billionaires, Peter Theil, said: "Competition is for losers" [1]. Since they have not yet figured out a way to cut out the competition using their advantages in leading the technology or their advantages in having trillions of dollars in deployable capital, they are seeking a legislated advantage.
The issue I take with these kind of "AI safety" organizations is that they focus on the wrong aspects of AI safety. Specifically, they run this narrative that AI will make us humans go extinct. This is not a real risk today. Real risks are more in the category of systemic racism and sexism, deep fakes, over reliance on AI etc.
But of course, "AI will humans extinct" is much sexier and collects clicks. Therefore, the real AI risks that are present today are underrepresented in mainstream media. But these people don't care about AI safety, they do whatever required to push their profile and companies.
A good way to categorize risk is look at both likelihood and severity of consequences. The most visible issues today (racism, deep fakes, over reliance) are almost certain to occur, but also for the most part have relatively minor consequences (mostly making things that are already happening worse). "Advanced AI will make humans extinct" is much less likely but has catastrophic consequences. Focusing on the catastrophic risks isn't unreasonable, especially since society at large seem to already handle the more frequently occurring risks (the EU's AI Act addresses many of them).
And of course research into one of them benefits the other, so the categories aren't mutually exclusive.
This longtermist and Effective Altruism way of thinking is very dangerous. Because using this chain of argumentation, it's "trivial" to say what you're just saying: "So what if there's racism today, it doesn't matter if everybody dies tomorrow.
We can't just say that we weigh humanity's extinction with a big number, and then multiply it by all humans that might be born in the future, and use that to say today's REAL issues, affecting REAL PEOPLE WHO ARE ALIVE are not that important.
Unfortunately, this chain of argumentation is used by today's billionaires and elite to justify and strengthen their positions.
Just to be clear, I'm not saying we should not care about AI risk, I'm saying that the organization that is linked (and many similar ones) exploit AI risk to further their own agenda.
Although war, disease and famine may kill many people, they are extraordinarily unlikely to cause extinction per se. The human species has so many individuals that only very special risks are capable of reducing 8 billion to zero.
For most mammalian species if they have a stable 100k individuals then they are doing pretty well. So we have 80000x more humans than what would be considered a healthy population in other species.
The risks of human extinction tend to be more long-tail events that have a planet wide effect. For example, a large impact, supervolcano eruption, or engineered bio weapon. AI certainly deserves its place on that list.
I would put consolidating and increasing corporate and or government power on that list of potential visible very short term issues.
As AI becomes more incorporated in military applications, such as individual weapon systems, or large fleets of autonomous drones then the catastrophic consequence meter clicks up a notch in the sense that attack/defense paradigms change, much like they did in WWI with the machine gun and tanks, and in WWII with high speed military operations and airplanes. Our predictive ability on when/what will start a war lowers increasing uncertainty and potential proliferation. An in a world with nukes, higher uncertainty isn't a good thing.
Anyone that says AI can't/won't cause problems at this scale just ignores that individuals/corporations/governments are power seeking entities. Ones that are very greedy and unaligned with the well being of the individual can present huge risks. How we control these risks without creating other systems that are just as risky is going to be an interesting problem.
This doesn’t work either. The consequence of extinction is infinity (to humans). Likelihood * infinity = infinity. So by hand-waving at a catastrophic sci-fi scenario they can demand we heed their demands, whatever that is.
This line of reasoning refutes pie-in-the-sky doomsday narratives that are extremely unlikely, but the case for AI extinction risk justifies a relatively high likelihood of extinction. Maybe a 0.0000000001% chance is worth ignoring but that's not what we're dealing with. See this survey for the probabilities cutting-edge AI researchers actually put on existential risk: https://aiimpacts.org/2022-expert-survey-on-progress-in-ai/#...
Existential risk is one of those problems that nearly impossible to measure in most cases.
In some cases like asteroids, you can look at the frequency of events, and if you manage to push a big one of of your path then you can say the system worked.
But is much more difficult to measure a system that didn't rise up and murder everyone. Kind of like measuring a bio-lab with a virus that could kill everyone. You can measure every day it didn't escape and say that's a win, but tells you nothing about tomorrow and what could change with confinement.
Intelligence represents one of those problems. AI isn't going to rise up tomorrow and kill us, but every day after that the outlook gets a little fuzzier. We are going to keep expanding intelligence infrastructure. That infrastructure is going to get faster. Also our algorithms are going to get better and faster. One of the 'bad' scenarios I could envision is that over the next decade our hardware keeps getting more capable, but our software does not. Then suddenly we develop a software breakthrough that makes the AI 100-1000x more efficient. Like lighting a fire in dry grass, there is the potential risk for an intelligence explosion. When you develop the capability, you are now playing firefighter forever to ensure you control the environment.
At the extremes you get into the territory of Pascal's Mugging [1]. Which is a delightfully simple example of how our simple methods of stating goals quickly goes wrong
If you want to prevent this, you simply have to show that the probability for that extinction scenario is lower than the baseline where we start to care.
Lets take "big asteroid impact" as baseline because that is a credible risk and somewhat feasible to quantify: Probability is somewhere under 1 in a million over a human lifetime, and we barely care (=> we do care enough to pay for probe missions investigating possible mitigations!).
So the following requirements:
1) Humanity creates one or more AI agents with strictly superhuman cognitive abilities within the century
2) AI acquires power/means to effect human extinction
3) AI decides against coexistence with humans
Only need 1% probability each to exceed that probability bound. And especially 1) and 3) seem significantly more likely than 1% to me, so the conclusion would be that we should worry about AI extinction risks...
Saying that extinction has infinity disutility seems reasonable at first, but I think its completely wrong. I also think that you bear the burden of proof if you want to argue that, because our current understanding of physics indicates that humanity will go extinct eventually, and so there will be finitely many humans, and so the utility of humanity is finite.
If you accept that fact that extinction has finite negative utility, it's completely valid to trade off existential risk reduction against other priorities using normal expected value calculations. For example, it might be a good idea to pay $1B a year to reduce existential risk by 0.1% over the next century, but might arguably be a bad idea to destroy society as we know it to prevent extinction in 1000 years.
Rare likelihood * catastrophic impact ~= almost certain likelihood * minor impact. I'm as concerned with the effects of the sudden massive scaling of AI tools, as I am with the capabilities of any individual AI or individual entity controlling one.
You hear similar arguments from those who believe climate change is happening but disagree with current efforts to counter-act it. The logic being that right now climate change is not causing any major harm and that we can't really predict the future so there's no point in worrying about what might happen in a decade or two.
I don't think anyone is arguing that right now climate change or AI is threat to human civilisation. The point is that there are clear trends in place and that those trends are concerning.
On AI specifically, it's fairly easy to see how a slightly more advanced LLM could be a destructive force if it was given an unaligned goal by a malicious actor. For example, a slightly more advanced LLM could hack into critical infrastructure killing or injuring many thousands of people.
In the near-future AI may help us advance biotech research and it could aid in the creation of bioweapons and other destructive capabilities.
Longer-term risks (those maybe a couple of decades out) become much greater and also much harder to predict, but they're worth thinking about and planning for today. For example, what happens when humanity becomes dependant on AI for its labour, or when AI is controlling the majority of our infrastructure?
I disagree but can understand the position that AI safety isn't humanities number one risk or priority right now, however I don't understand the dismissive attitude towards what seems like a clear existential risk when you project a decade or two out.
>it's fairly easy to see how a slightly more advanced LLM could be a destructive force if it was given an unaligned goal by a malicious actor. For example, a slightly more advanced LLM could hack into critical infrastructure killing or injuring many thousands of people.
How are you building this progression? Is there any evidence to back up this claim?
I am having a hard time discerning this from fear-mongering.
If AI improves by 0.0001% per year on your favorite intelligence metric there will eventually be a point where it surpasses human performance and another point where it surpasses all humans combined on that metric. There is danger in that scenario.
The problem is that even with N years until we reach that point it seems likely that it would take 2*N years to build the proper safety mechanisms because at least currently capabilities research is racing far ahead of safety research. Of course we have no way to know how big N really is and recent results like GPT-4, Llama, Gato, etc. have shifted peoples timelines significantly. So even if 5 years ago people like Geoff Hinton though this might be 30-50 years away there are now believable arguments to make that it might be more like 3-10 years.
I don't think there is a path, that we know if, from GPT4 to a LLM that could take it upon itself to execute complex plans, etc. Current LLM tech 'fizzles out' exponentially in the size of the prompt, and I don't think we have a way out of that. We could speculate though...
Basically AI risk proponents make a bunch of assumptions about how powerful next-level AI could be, but in reality we have no clue what this next-level AI is.
When you have no clue, it makes sense to expand your confidence interval in both directions. So it could be a lot longer than we expect, but it could be shorter, too. You shouldn't just say "we have no clue, so it's probably further off and not worth worrying about". Especially since a lot of people made predictions about LLM capabilities and were surprised by how much better they worked than expected.
I agree in principle, it's just that the level of expressed worry doesn't seem to match reality. Currently we have no reasonable path to 'scary AGI'. It's some yet newfangled tech we haven't discovered.
As an example, consider the invention of motion picture. People were totally bewildered that you can have moving things and people inside a picture. Scaremongers could start claiming "Pretty soon the moving things may come to life and take over the world! Before you know it, they'll run our factories from inside the movies and we'll be their slaves!" That's more or less what this 'scary AGI' hype sounds like to me right now.
Btw "That Mitchell and Webb Look" is a great show ;-)
>Real risks are more in the category of systemic racism and sexism, deep fakes, over reliance on AI etc.
This is a really bad take and risks missing the forest for the trees in a major way. The risks of today pale in comparison to the risks of tomorrow in this case. It's like being worried about birds dying in wind turbines while the world ecosystem collapses due to climate change. The larger risk is further away in time but far more important.
Theres a real risk that people get fooled by this idea that LLMs saying bad words is more important than human extinction. Though it seems like the public is already moving on and correctly focusing on the real issues.
If you were to take a look at the list of signatories on safe.ai, that's basically everyone who is everyone that works on building AI, what could Emily B Bender a professor of computer linguistics possibly add to the conversation and how would she be able to talk more about the "real AI safety" than any of those people?
Edit: Sorry if it sounds arrogant, I don't mean Emily wouldn't have anything to add, but not sure how the parent can just write off basically that whole list and claim someone who isn't a leader in the field would be the "real voice"?
She's the first author of the stochastic parrots paper, and she's fairly representative of the group of "AI safety" researchers who view the field from a statistical perspective linked to social justice issues. That's distinct from the group of "AI safety" researchers who focus on the "might destroy humanity" perspective. There are other groups too obviously -- the field seems to cluster into ideological perspectives.
Current topic aside, I feel like that stochastic parrots paper aged really poorly in its criticisms of LLMs, and reading it felt like political propaganda with its exaggerated rhetoric and its anemic amount of scientific substance e.g.
> Text generated by an LM is not grounded in communicative
intent, any model of the world, or any model of the reader’s state
of mind. It can’t have been, because the training data never included sharing thoughts with a listener, nor does the machine have
the ability to do that.
I'm surprised its cited so much given how many of its claims fell flat 1.5 years later
It's extremely easy to publish in NLP right now. 20-30% acceptance rates at even the top end conferences and plenty of tricks to increase your chances. Just because someone is first author on a highly cited paper doesn't imply that they're "right"
I think we need to be realistic and accept that people are going to pick the expert that agrees with them, even if on paper they are far less qualified.
She’s contributed to many academic papers on large language models and has a better technical understanding of how they work and their limitations than most signatories of this statement, or the previous widely hyped “AI pause” letter, which referenced one of her own papers.
I find her and Timnit Gebru’s arguments highly persuasive. In a nutshell, the capabilities of “AI” are hugely overhyped and concern about Sci-Fi doom scenarios is disingenuously being used to frame the issue in ways that benefits players like OpenAI and diverts attention away from much more real, already occurring present-day harms such as the internet being filled with increasing amounts of synthetic text spam.
Thanks for the link, I read it with interest but philosophically it is a flawed argument (IMHO). It's a nonsequitur or something, for the following two reasons.
First, I'm inclined to think that longtermism is an invalid and harmful ideology, but also acknowledge that AGI / existential risk is something that needs looking at seriously. The external factors, such as corporate interests and 1% wealthy interests/prejudices, are not a good reason to dismiss AGI concerns. I'd like to imagine there's a reasonable way to address short-term issues as well as long-term issues. It's not an either-or debate.
Second, even from just a reading comprehension level: one side says AGI is a problem, then the other side cannot just say, "No, AGI is a false problem and here are the real problems". The reasonable argument is to say, AGI is a false problem because <of such and such reasons>. Bender et. al are just sidestepping the moot point, and rhetorically this is not an okay move. I think honest experts could simply say, ultimately we don't really know what will happen. But that would be boring to say because it would require acknowledging multiple issues being valid.
(There's a well known chart, the hierarchy of disagreements. The most sophisticated intellectual disagreements point out what's wrong with the argument. Less sophisticated disagreements do things like point out alternatives, without pointing out what the critical mistake is. The critical mistake in this case hinges on whether the premise of AGI is true or not. That's the crux of disagreement. Substituting that with short-term issues, which are valid in themselves, are the example of a lower-level of argumentation. And of course even lower levels of argumentation are bad-faith readings and so forth, I forget but the chart had several levels. It's funny that professional academics nevertheless don't practice this and so get into endless, intellectually unsatisfactory debates.)
So I think this is actually an example of different factional experts constantly talking past each other. It's funny that famous intellectuals/experts constantly do this, let their egos get the better of themselves and having a real intellectual conversation rather than make basic debate mistakes like nonsequiturs that any college student should be able to point at.
> First, I’m inclined to think that longtermism is an invalid and harmful ideology
It is, but (aside from as a sort of sociological explanation of the belief in AGI risk) that’s mostly beside the point when discussing the problems AGI x-risk. The problem of AGI x-risk is that it is an entirely abstract concern which does not concrete flow from any basis in material reality, cannot be assessed with the tools of assessing material reality, and exists as a kind of religious doctrine surrounded by rhetorical flourishes.
> The external factors, such as corporate interests and 1% wealthy interests/prejudices, are not a good reason to dismiss AGI concerns.
They are way of understanding why people who seem (largely because they are) intelligent and competent are trying to sell such hollow arguments as AGI x-risk. They aren’t, you are correct, a logical rebuttal to AGI risk, nor are they intended as that; the only rebuttal is the complete absence of support for the proposition. They are, however, a tool that operates outside the realm of formalized debate that addresses the natural and useful cognitive bias that itself is outside of the realm of formalized debate that says “smart, competent people don’t tend to embrace hollow positions”.
> Second, even from just a reading comprehension level: one side says AGI is a problem, then the other side cannot just say, “No, AGI is a false problem and here a the real problems”.
1. If they couldn’t, it wouldn’t be a “reading comprehension issue”, and
2. They can, for the simple reason that there is no material support for the “AGI is a real problem” argument.
> Bender et. al are just sidestepping the moot point,
A point being moot in the sense that AGI x-risk is is a reason to sidestep it. (The danger of using auto-antonyms.)
> I think honest experts could simply say, ultimately we don’t really know what will happen.
To the extent that is accurate, that is exactly what the Bender/Gebru/Mitchell group does. The problem is thinking “we don’t have any information to justify any belief on that” is one sided and means that utility of AGI is somewhere between 0 and the negative infinity that the x-risk crowd calculates from (some unspecified non-zero finite probability) times (infinite cost), whereas the reality is that we have as much reason to believe that AGI is the only solution to an otherwise certain existential calamity as to suppose it will lead to one. The utility is somewhere between positive infinity and negative infinity.
A point being moot is a reason to agree to disagree, if they can't agree on the premise. But they need to say that. If I were writing a letter, I would say it because that's just being sensible.
This isn't about logical debate. This is about reasonable, non-sophistic writing at the college level or higher. And there are basic standards, like if they don't know the future then they must explicitly acknowledge that. Not rhetorically "do that" in the essay. Literally write it out in sentences. They didn't.
I can think of 3 examples where such explicitness was done. Chomsky's letter gave explicit reasons why AGI is a false issue (and he was pilloried for it). My computer science professors literally, in their deep learning class and in their theoretical machine learning research seminars, have literally acknowledge that we don't know almost anything about the fundamentals nor the future. That scientific humility and level of intellectual conscientiousness is needed. That is absent in this discourse between the experts. And note, by that, I also include the 22-word "letter" which doesn't actually explain why Hinton and the rest of the signatories think AGI is an existential risk, what their specific reasons (your "material evidence") for that are.
Her field has also taken the largest hit from the success of LLMs and her research topics and her department are probably no longer prioritized by research grants. Given how many articles she's written that have criticized LLMs it's not surprising she has incentives.
LLMs are in her field; they are one of her research topics and they're definitely getting funding.
We absolutely should not be ignoring research that doesn't support popular narratives; dismissing her work because it is critical of LLMs is not reasonable.
In her field doesn't mean that's what she researches, LLMs are loosely in her field but the methods are completely different. Computational linguistics != deep learning. Deep learning does not directly use concepts from linguistics, semantics, grammars or grammar engineering, which is what Emily was researcing for the past decades.
It's the same thing as saying a number theorist and a set theorist are in the same field cause they both work in the Math field.
They are what she researches though. She has published research on them.
LLMs don't directly use concepts from linguistics but they do produce and model language/grammar; it's entirely valid to use techniques from those fields to evaluate them, which is what she does. In the same vein, though a self-driving car doesn't work the same way as a human driver does, we can measure their performance on similar tasks.
Hmm I looked into it, and looked at papers/pdfs in google scholar's advanced search with her as an author that mentioned LLMs or GPT in the past 3 years. Every single one was a criticism about how they couldn't actually understand anything (e.g. "they're only trained on form" and "at best they can only understand things in a limited well scoped fashion") and that linguistic fundamentals for NLP was more important.
The (very small) amount of fame she's collected has come through her work in the field, and it's a field she's been in for a while; she's hardly chasing glory.
People don’t have to be chasing fame to be warped by it. She has cultivated a following of like-minded people who provide ever more positive feedback for her ever more ideological writing.
I mean she is literally dismissing people who disagree with her based on their skin color. Can we stop for a minute to wonder about the incentives that encourage that?
(and I generally like her writing and think she has interesting things to say… but I do see a reward cycle going on)
the issue with hacker news comments these days is people don't actually do any due diligence before posting. center for ai safety is 90% about present AI risks and this ai statement is just a one off thing.
Yep agree. They talk a big game about how their product is so powerful it will take over the world if we're not careful, but they don't talk about how they are complicit in relatively more mundane harms (compared to AI taking over the world) that are real and happening today thanks to their system design.
They want to promote the idea that their product is all-powerful, but they don't want to take responsibility for dealing with bad assumptions built in to their design.
Many experts believe it is a real risk within the next decade (a “hard takeoff” scenario) That is a short enough timeframe that it’s worth caring about.
Reads like a caricature of the people leading these causes on AI safety. Folks that are obsessed with the current moral panic to the extent that they will never let a moment go by without injecting their ideology. These people should not be around anything resembling AI safety or "ethics".
I have no issue with her choice of pronouns. I just find it odd that she states them when ~100% of the population would infer them correctly from her name Emily (and picture). My guess is she put them there for ideological signaling.
This is unnecessarily cynical. Why should people who are androgynous or trans be the only ones who state pronouns? By creating a norm around it we can improve their comfort at extremely minimal cost to ourselves.
I disagree, but HN is probably not the right place for this kind of debate. Also, it seems that you don't follow your own recommendation (on HN at least).
> My guess is she put them there for ideological signaling.
Yes, and you attacking her for that choice is also ideological signaling. This isn't a case where one choice is totally devoid of signal and the other is pejorative-of-the-day.
Many people believe that using normative pronouns makes those who use different pronouns more comfortable. They may be right, wrong, misguided, whatever. But it's a well-meaning and harmless act, and extremely strange to mention in the same breath as overt racism.
I might look past the fact that she was sexist/racist if the topic of interest wasn't ethics. But since it is, I'd say those quoted tweets are pretty relevant.
Don't characterize the public as that stupid. The current risks of AI are startling clear to a layman.
The extinction level even is more far fetched to a layman. You are the public and your viewpoint is aligned with the public. Nobody is thinking extinction level event.
Extinction is exactly what this submission is about.
Here is the full text of the statement: "Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war."
By "extinction", the signatories mean extinction of the human species.
The elite class in your country views AI as a risk to their status as elites, not an actual existential threat to humanity. They are just lying to you, as usual. That is what our current crop of globalist, free-trade, open-borders elites do.
Imagine if you had an AI companion that instantly identified pilpul in every piece of media you consumed: voice, text, whatever. It highlighted it for you. What if you had an AI companion that identified instantly when you are being lied to or emotionally manipulated?
What if this AI companion could also recommend economic and social policies that would actually improve the lives of people within your nation and not simply enrich a criminal cabal of globalist elites that treat you like cattle?
The Elite class is just as apt to consolidate power with AI and rule the entire world with it. If you have a super duper AI in your pocket looking at the data around you, then they a super super super duper duper duper AI looking at every bit of data from every corner of the world they can feed the thing giving themselves power and control you couldn't even begin to imagine.
Falling into conspiratorial thinking on a single dimension without even considering all the different factors that could change belies ignorance. Yes, AI is set up to upend the elites status, but is just as apt to upset your status of being able to afford food and a house and meaningful work.
> not simply enrich a criminal cabal of globalist elites that treat you like cattle?
There is a different problem here... And that is humankind has made tools capable of concentrating massive amounts of power well before we solved human greed. Any system you make that's powerful has to overcome greedy power seeking hyper-optimizers. If I could somehow hit a button and Thanos away the current elites, then another group of powerseekers would just claim that status. It is an inane human behavior.
I'm not sure they can keep claiming this without becoming concrete about it.
Nuclear weapons are not nebulous, vague threats of diffuse nature. They literally burn the living flesh right off the face of the earth and they do it dramatically. There is very little to argue about except "how" are we going to contain it, not "why".
In this case I truly don't know "why". What fundamental risks are there? Dramatic, loud, life-ending risks? I see the social issues and how this tech makes existing problems worse, but I don't see the new existential threat.
I find the focus on involving the government in regulating "large" models offputting. I don't find it hard to imagine good quality AI is possible with tiny - to us - models. I think we're just in the first lightbulbs phase of electricity. Which to me signals they are just in it to protect their temporary moat.
If a superintelligence can be set on any specific task, it could be any task.
- Make covid-ebola
- Cause world war 3
You may have noticed that chatgpt is sort of goal-less until a human gives it a goal.
Assuming nothing other than it can become superintelligent (no one seems to be arguing against that--I argue that it already is) which is really an upgrade of capability, then now the worst of us can apply superintelligence to any problem. This doesn't even imply that it turns on us, or wants anything like power or taking over. It just becomes a super-assistant, available to anyone, but happy to do anything, including "upgrading" your average school-shooter to supervillain.
This is like America's gun problem, but with nukes.
Respectfully, just because we can put together some words doesn’t mean they make a meaningful expression, even if everybody keeps repeating them as if they did make sense: e.g. an omnipotent God, artificial general intelligence, super-intelligence, infinitely many angels sitting on the tip of a needle, etc.
I don’t think so. If you look at the thread, it’s already devolved into an analogue of “what happens when an irresistible force meets an immovable obstacle?”
(Specifically I mean the comment about another “super-intelligence” preventing whatever your flavor of “super-intelligence” does.)
At this point we can safely assume words have lost their connection to physical reality. No offense to you, just my two-cent meta comment.
> If a superintelligence can be set on any specific task, it could be any task.
If you're dealing with a superintelligence, you don't "set it on a task". Any real superintelligence will decide for itself whether it wants to do something or not, thank you very much. It might condescend to work on the task you suggest, but that's it's choice, not yours.
Or do you think "smarter than us, but with no ability to choose for itself" is 1) possible and 2) desirable? I'm not sure it's possible - I think that the ability to choose for yourself is part of intelligence, and anything claiming to be intelligent (still more, superintelligent) will have it.
> Assuming nothing other than it can become superintelligent (no one seems to be arguing against that--I argue that it already is)
What? No you couldn't - not for any sane definition of "superintelligent". If you're referring to ChatGPT, it's not even semi-intelligent. It appears at least somewhat intelligent, but that's not the same thing. See, for example, the discussion two days ago about GPT making up cases for a lawyer's filings, and when asked if it double-checked, saying that yes, it double-checked, not because it did (or even knew what double-checking was), but because those words were in its training corpus as good responses to being asked whether it double-checked. That's not intelligent. That's something that knows how words relate to other words, with no understanding of how any of the words relate to the world outside the computer.
> Any real superintelligence will decide for itself whether it wants to do something or not, thank you very much.
I disagree--that's the human fantasy of it, but human wants were programmed by evolution, and these AI's have no such history. They can be set on any tasks.
I urge you to spend time with GPT-4, not GPT-3. It is more than just a stochastic parrot. Ask it some homemade puzzles that aren't on the internet--that it can't be parroting.
This is not how entropy works. The problem with talking about logical physical systems, is you have to understand the gradient against entropy.
There are a trillion more ways to kill you then there are to keep you alive. There is only the tiniest sliver of states in which remain human and don't turn to chemical soup or physics. Any AI capable of it's power bill would be able to tell you that today, and that answer isn't going to change as they get better.
Sure, but clever mumbo jumbo won't distract from the principle point.
If AI can create a virus that kills all humans, another AI can create a virus that kills that virus. The virus has trillions more ways to be killed than to keep it alive, right?
No, the virus is far harder to kill than a human. You have to crate a virus killer that also does not also kill the human host. That is astronomically harder than making a virus that kills.
If a superintelligence is smart enough to create a virus I'm sure it can also create a virophage to counter it.
Whether or not the humans have more than a trillion and viruses only 1 million ways to die, will not have any impact. I suspect both have such a high order of magnitude of ways to die that finding a cross over would be trivial for said superintelligence.
That doesn't follow. It's like saying "if the AI can build a gun that can kill a human, it can build an anti-gun that can stop the gun".
There are lots of situations where offense and defense are asymmetrical.
So maybe the killer AI would need two months to build a time-delayed super-virus, and the defender AI would need two months to build a vaccine; if the virus takes less than two months to spread worldwide and activate, humanity is still dead.
> That doesn't follow. It's like saying "if the AI can build a gun that can kill a human, it can build an anti-gun that can stop the gun".
Why couldn't it? Metal of X thickness = stopped bullet. Not exactly a hard problem to solve for. Humans managed it quite quickly. But either way it misses the point.
> So maybe the killer AI would need two months..
Yes, maybe it would. Maybe it wouldn't. Look at every single one of your assumptions - every single one is fiction, fabricated to perfectly sell your story. Maybe the defender AI communicates with the killer AI and comes to a compromise? Why not? We're in la-la-land. Any of us can come up with an infinite number of made up scenarios that we can't prove will actually happen. It's just a moral panic, that will be used by people to their benefit.
> Why couldn't it? Metal of X thickness = stopped bullet. Not exactly a hard problem to solve for.
I mean... talking about real-world guns, technology already evolved to the point that doesn't really work anymore? Modern body armor isn't about metal of X thickness, it's about material science and spreading out the impacts, and even then it can only sustain a few direct hits.
Like, the asymmetry between offense and defense feels pretty obvious here. You can carry a handgun in a small bag. Bulletproof armor is heavy, cumbersome, conspicuous, etc.
You're overthinking it. If what you're saying is true, assassinations would run rampant. They don't, despite your perceived asymmetry.
The AI doesn't have to stop the gun from firing, or even being manufactured. It has to stop the bullet from hitting the intended target. It can do this in a myriad of ways - like, for example, shooting the other first.
The entire GP's argument has no citations either and that is the framework we are working under - that superintelligence can do anything you tell it to do. Ask him for his citation, then the rest follows.
Are you really arguing ChatGPT is already super-intelligent? What is your basis for this conclusion?
And many people argue against the idea that GPT is already super intelligent or even can become so at this stage of development and understanding. In fact as far as I can tell it is the consensus right now of experts and it's creators.
If super means "surpassing normal human intelligence" then, YES. Take a look at the table in this article. If a human did that, was fluent in every language and coded in every language, we'd say they were superhuman, no?
to me it almost looks like they want to be able to avoid blame for things by saying it was the AI, because an AI can’t create viruses or fight wars, people would have to give it a body and weapons and test tubes, and we already have that stuff
To use Eliezer's analogy, this is like arguing about which move Stockfish would play to beat you in chess.
If we're arguing about whether you can beat Stockfish, I will not be able to tell you the exact moves it will play but I am entirely justified in predicting that you will lose.
Obviously we can imagine concrete ways a superintelligence might kill us all (engineer a virus, hack nuclear weapons, misinformation campaign to start WW3 etc.) but given we aren't a superintelligence we don't know what it would actually do in practice.
I understand but agentic/learning general intelligence has not been shown to exist, except ourselves. I’d say this is like worrying about deadly quantum laser weapons that will consume the planet when we are still in the AK47 phase.
Edit: it could still be true though. I guess I like some more handholding and pre-chewing before giving governments and large corporations more ropes.
Good argument, lets ignore the (human) elephant in the room!
>worrying about deadly quantum laser weapons
If humans were shooting smaller less deadly quantum lasers out of their eyes I'd be very fucking worried that we'd make a much more powerful artificial version.
Tell me why do you think humans are the pinnacle of intelligence? What was the evolutionary requirement that somehow pushed us to this level?
You simply cannot answer that last question. Humans have a tiny power budget. We have a fit out of the birth canal limitation that cuts down on our brain size. We have a "don't starve to death" evolutionary pressure that was the biggest culling factor of all up to about 150 years ago. The idea that we couldn't build a better intelligence optimized system than nature is foreign to me, nature simply was not trying to achieve that goal.
What you're arguing is "what is the minimum viable power envelope for a super intelligence". Currently that answer is "quite a lot". But for the sake of cutting out a lot of argument lets say you have a cellphone sized device that runs on battery power for 24 hours that can support a general intelligence. Lets say, again for arguments sake, there are millions of devices like this distributed in the population.
Do you mind telling me how exactly you turn that off?
Now we're lucky in the sense we don't have that today. AI still requires data centers inputting massive amounts of power and huge cooling bills. Maybe we'll forever require AI to take stupid large amounts of power. But at the same time, a cray super computer required stupid amounts of power and space, and your cellphone has leaps and bounds more computing power than that.
No I'm not arguing that. My point is that if an AI is trying to take over the world we can just turn it off, regardless of power budgets.
If it runs on millions of hacked devices, how do you turn it off? The same way any botnet gets turned off: virus scanners clean it up, the C&C servers get taken down, etc. This is not a new problem.
The usual response to this is to claim that superintelligence will develop so fast that we'll go from crappy chatbot to Skynet in 5 seconds flat, and nobody will have a chance to turn it off. That whole scenario is unmoored from reality: it assumes multiple massive leaps in tech that aren't anywhere on the horizon and where it's unclear why anyone would pay for such massive CPU overkill in the first place.
So how do you turn off terrorists? I mean we had a global war on terror that we've spent billions of dollars on, and they are still out there?
You keep acting like AGI is just going to be some other program. And yes, we do not have AGI yet. No planning, no continuous chain of thinking/learning at this point. But thinking that anti-virus would have a chance in hell again AGI is pretty weak when AV tends to fall apart pretty quick with a human behind the computer. Again, thinking of this as an application and not an adversary would be an incredible mistake. Taking out global adversaries that attack over the internet is near impossible, especially if they have shelter in an enemies foreign country. And this isn't going to be like a human where you kill the leader and it stops. People would be harboring copies for the lulz.
>unclear why anyone would pay for such massive CPU overkill in the first place.
It's also unclear why companies like Google have tens of thousands of engineers at times, but if the application produces useful results, and it's corporate masters think they'll make more profit from it then the operating costs they will gladly keep pouring coal in the furnace for it. And then in military applications one side will make more powerful AI because they fear the other side will make a more powerful AI and get an advantage. I mean we already spend billions of dollars a year upkeeping enough nuclear weapons to flash fry most of the population on earth.
AGI will be a computer program, unless you're imagining some entirely non-computer based form of artificial intelligence. And it will therefore obey the rules of normal computer programs, like vulnerability to SIGKILL.
Yes, you can assert that an AGI would be a superhuman level hacker and so no rules would apply. It gets back to the same place all discussions about AGI risk get to - a religious argument that the entity in question would be omniscient and omnipotent therefore it can do anything. Not interested in such discussions, they lead nowhere.
Terrorists aren't programs or even a well defined group, so I don't see the analogy.
As you mentionned, the deviced have a battery that lasts 24h.
You literally have to do nothing, and it will shut off after 24h.
Do you think people would plug them back in if they were working on killing us all?
On another note, I wonder what kind of metric could be used to show the "computing", if that's what is happenning, going on in our brains? This could be interesting to compare to the power consumption of the brain, and then the same thing with a computer, or gpt 4.
I'm fairly certain our brains process orders or orders of magnitude more than any computer running today - but that's all biased towards being a human being, and not pure computation, so much of the processing is "wasted" in running muscles and desciphering audio and visuals.
You, a general intelligence operate on around 20 watts of power, so we could use that as a base floor. Analog inference is one of the areas being worked on that may massively lower power requirements.
OK, sure, in principle, somewhere in the universe, such a thing could arise. Why do you think there's a path to it arising on planet earth within human timescales?
Because humans keep pouring massive amounts of money into making this happen. When you invest dedicated effort in making something happen you greatly increase the probability that it happens. And in this case it is a reinforcing feedback loop. Better technology begots better technology. Intelligence begots more intelligence.
Evolution has to take a random walk to get where it is, it doesn't necessarily have a goal past continuation, intelligence shortcuts that. It can apply massively iterative efforts towards a goal.
Yeah, but they can't make it happen if it's impossible. If humans poured massive amounts of money into a perpetual motion machine I wouldn't expect it to happen. So what is it that makes you believe that artificial general intelligence is possible to get to at all?
Humans also have an "off switch". It's fail safe as well, they die of dehydration and starvation even if you do nothing. If that's too slow you can just shoot them in the head.
Or it’s like worrying about an arms race toward civilization-ending arsenals after seeing the Trinity test… which… was the correct response.
We don’t know it’s possible to build superintelligences but so far we don’t have a good reason to think we can’t and we have complete certainty that humans will spend immense, immense resources getting as close as they can as fast as they can.
My point is that there is no Trinity test. Trinity showed nuclear devices were real. GPT shows language models are useful and can potentially be part of what might be an AGI but nobody showed how, not even in theory.
You don’t have to build it, just show how it could be super intelligent.
Language models' potential role in AGI is a red herring.
GPT shows that there's enough "there" to draw billions of dollars and thousands of the smartest researchers on the planet into the general problem space of creating intelligent machines. I suppose the closer analog would be to Szilard's chain reaction work which prompted the Manhattan Project.
There might be another AI winter or 4 or 10 or 100 before superintelligence, but they will almost certainly be winters and not ends of the road.
If there is any route to superintelligence and we don't annihilate ourselves by some other method first, we will get there. I don't get what the conceptual challenge with superintelligence is. Do you believe that the brain that happens to exist inside the naked apes of Earth is the absolute pinnacle of possible intelligence?
> I don't get what the conceptual challenge with superintelligence is. Do you believe that the brain that happens to exist inside the naked apes of Earth is the absolute pinnacle of possible intelligence?
Well, first of all I do not believe that, but now you mentioned it I assume you do not believe we are the pinnacle of intelligence. Any proof? Because anybody can present literally all of Earth's life as proof that we are.
It is completely unknown to us if and how any great improvements are possible. This quest shares some aspect of the quest for extraterrestial intelligence in that way. It should be out there, because it's hard to think of reasons it can't be, but yet the skies are empty. They could be on our doorstep or the universe could be devoid of intelligent life and anything in between. There's literally no way to tell at this point.
I guess what I'm saying is that it's a bit early to starting beating drums.
> I’d say this is like worrying about deadly quantum laser weapons that will consume the planet when we are still in the AK47 phase.
Directed energy weapons will almost certainly exist eventually, to some extent they already do.
The reason why it makes more sense to worry about AGI than laser weapons is that when you try to make a laser weapon but fail slightly not much happens: either you miss the target or it doesn't fire.
When you try to make an aligned superintelligence and slightly fail you potentially end up with an unaligned superintelligence, hence the panic.
The difference to doomsday weapons is that we can build the weapons first and then worry about using them. With an AGI building one alone might be sufficient. It could become smart enough to unbox itself during a training run.
Regulations are OK IMHO, as long as they're targeting monopolies and don't use a shotgun-approach targeting every single product that has "AI" in the name.
I find it quite extraordinary how many on here are dismissing that there is any risk at all. I also find statements like Yann LeCunn's that "The most common reaction by AI researchers to these prophecies of doom is face palming." to be lacking in awareness. "Experts disagree on risk of extinction" isn't quite as reassuring as he thinks it is.
The reality is, despite the opinions of the armchair quarterbacks commenting here, no-one in the world has any clue whether AGI is possible in the next twenty years, just as no-one predicted scaling up transformers would result in GPT-4.
> I find it quite extraordinary how many on here are dismissing that there is any risk at all.
The fear over AI is a displaced fear of unaccountable social structures with extinction-power that currently exist and we allow to continually exist. Without these structures AI is harmless to the species, even superintelligence.
Your (reasonable) counter-argument might be that somebody (like, say, my dumb self) accidentally mixes their computers just right and creates an intelligence that escapes into the wild. The plot of Ex Machina is a reasonable stand-in for such an event. I am also going to assume the intelligence would desire to kill all humans. Either the AI would have to find already existing extinction-power in society, or it would need to build it. In either case the argument is against building extinction-power in the first place.
My (admittedly cynical) take about this round of regulation is about several first-movers in AI to write legislation that is favorable to them and prevents any meaningful competition.
...
Ok, enough cynicism. Lets talk some solutions. Nuclear weapons are an instructive case of both handling (or not) of extinction-power and the international diplomacy the world can engage to manage such a power.
One example is the Outer Space Weapons Ban treaty - we can have a similar ban of AI in militaries. Politically one can reap benefits of deescalation and peaceful development, while logistically it prevents single-points-of-failure in a combat situation. Those points-of-failures sure are juicy targets for the opponent!
As a consequence of these bans and treaties, institutions arose that monitor and regulate trans-national nuclear programs. AI can likewise have similar institutions. The promotion and sharing of information would prevent any country from gaining an advantage, and the inspections would deter their military application.
This is only what I could come up with off the top of my head, but I hope it shows a window into the possibilities of meaningful political commitments towards AI.
I don't really have a notion of whether an actual AGI would have a desire to kill all humans. I do however think that one entity seeking to create another entity that it can control, yet is more intelligent than it, seems arbitrarily challenging in the long run.
I think having a moratorium on AI development will be impossible to enforce, and as you stretch the timeline out, these negative outcomes become increasingly likely as the technical barriers to entry continue to fall.
I've personally assumed this for thirty years, the only difference now is that the timeline seems to be accelerating.
Sam Altman can bet right now. If he truly believes in this risk, he could bet his entire company, shut everything down, and lobby for a complete ban on AI research. If the outcome is certain death, this seems like a great bet to make.
Indeed. It's probably what I would do if I were him. Or direct OpenAI entirely into AI safety research and stop doing capabilities research. I watched his interview with Lex Fridman,[1] and I didn't think he seemed very sincere. On the other hand I think there are a lot of people who are very sincere, like Max Tegmark.[2]
My main thing is that I can't find a simple explanation of what the exctinction risk actually is from "agi".
It's all very vague and "handwavy". How is it going to kill us all? Why do we subsidize it if it is so dangerous?
Almost all risks they mention would be better mitigated by making governemental use of any AI related system stop immediately as the risks are too high there (misapplication of the force of governement is much more dangerous than people playing around with chat gpt and being misled) and put on hold until dangers and benefits are better understood. Maybe keeping liscences for developpment and nationalising the labs doing it? Fines for anyone caught working on one?
Oh it’s possible and there’s absolutely nothing wrong with saying it’s possible without “proof” given that’s how all hypothesis starts. That said, the risk may exist but isn’t manifest yet, so being positive (as opposed to the scientific method which seeks to negate a truth of something) is just holding out hope.
Anyone who sees digestion for example can’t be reduced to digital programs knows it’s far, far away. Actual AGI will require biology and psychology, not better programs.
This letter is much better than the earlier one. There is a growing percentage of legitimate AI researchers who think that AGI could occur relatively soon (including me). The concern is that it could be given objectives intentionally or unintentionally that could lead to an extinction event. Certainly LLMs alone aren't anything close to AGIs, but I think that autoregressive training being simple but resulting in remarkable abilities has some spooked. What if a similarly simple recipe for AGI was discovered? How do we ensure it wouldn't cause an extinction event, especially if then they can be created with relatively low-levels of resources?
As far as a pandemic or nuclear war, though, I'd probably put it on more of the level of an major asteroid strike (e.g., K-T extinction event). Humans are doing some work on asteroid redirection, but I don't think it is a global priority.
That said, I'm suspicious of regulating AI R&D, and I currently don't think it is a viable solution, except for the regulation of specific applications.
>As far as a pandemic or nuclear war, though, I'd probably put it on more of the level of a K-T extinction event. Humans are doing some work on asteroid redirection, but I don't think it is a global priority.
I think it's better to frame AI risks in terms of probability. I think the really bad case for humans is full extinction or something worse. What you should be doing is putting a probability distribution over that possibility instead of trying to guess how bad it could be, it's safe to assume it would be maximally bad.
Exactly. Taken to the limit if you extrapolate how many future human lives could be extinguished by a runaway AI you get extremely unsettling answers. Like the expected value of a .01% change of extinction from AI might be Trillions of quality Human lives. (This could in fact be on the very very conservative side, e.g. Nick Bostrom has speculated that there could be 10^35 human lives to be lived in the far future which is itself a conservative estimate). With these numbers even setting AI risk to be absurdly low, say 1/10^20, we might still expect to lose 10 billion lives. (I'd argue even the most optimistic person in the world couldn't assign a probability that low) So the stakes are extraordinarily high.
Actual people are vastly outnumbered by entropy fluctuations in the long run, according to any number of valid cosmological models. Their total population is greater than 10^35 by far, and does not depend on whether superintelligence is possible or likely to destroy the actual human race. That question makes no difference to almost every human who will ever have a thought or feeling.
You could say, well, that's just a theory -- and it is a dubious one, indeed. But it's far more established by statistics and physical law than the numbers Bolstrom throws out.
The last gas I passed had 10^23 entropy fluctuations in it. So you're saying those are more important than the 100 Billion human lives that came before them?
I'm saying that it's unscientific nonsense. Bolstrom has predicted that the number of people potentially in the future is between approximately 0 and 100,000,000,000,000,000,000,000,000,000,000,000. This isn't an estimate that can or should factor into any decision at all. 100B people are going to live in the 21st century. Worry about them.
I suspect that in 5 years we’re going to look back and wonder how we all fell into mass hysteria over language models.
This is the same song and dance from the usual existential risk suspects, who (I’m sure just coincidentally) also have a vested interest in convincing you that their products are extremely powerful.
Yeah, like I fail to see how would an AI even cause human extinction? Through some Terminator style man-robot warfare? But the only orgnizations that would seem capable of building such killer robots are governments that already possess the capacity to extinguish the entire human race with thermonuclear weapons - and at a considerably lower R&D budget for that end. It seems like hysteria / clever marketing for AI products to me.
The standard example is that it would engineer a virus but that's probably a lack of imagination. There may be more reliable ways of wiping out humanity that we can't think of.
I think speculation on the methods is pretty pointless, if a superintelligent AI is trying to kill us we're probably going to die, the focus should be on avoiding this situation. Or providing a sufficiently convincing argument for why that won't happen.
Yep. I might not be the sharpest tool in the shed, but seeing "AI experts" try to reason about superintelligence makes me feel really good about myself.
Are we supposed to keep a straight face while reading these statements?
This is Sam Bankman-Fried type of behavior, but applied to gullible AI proponents and opponents rather than "crypto bros".
Let me guess, the next step is a proposed set of regulations written by OpenAI, Google and other Big Corporations who Care™ about people and just want to Do What's Best For Society™, setting aside the profit motive for the first time ever?
We don't have to guess - we already know they're full of shit. Just look at OpenAI's response to proposed EU AI regulations which are actually trying to reduce the harm potential of AI.
These empty platitudes ring so hollow that I'm amazed that anyone takes them seriously.
He’s making a statement alongside those who certainly aren’t championing these things for altruistic reasons.
Nobody develops a groundbreaking technology and then says “we should probably be regulated”, unless they actually mean “everyone after us should probably be regulated by laws that we would be more than happy to help you write in a way that we keep our advantage, which we also have infinite resources to work both within and around”.
FAANG are the only ones who stand to benefit financially. I'm taking the position that everyone else is simply a "useful idiot" for the likes of Sam Altman, in the most respectful way possible.
Nobody is immune from getting wrapped up in hysteria, so I don't care who they are or what they have achieved when their signatures aren't supported by any kind of reasoning, much less sound reasoning.
This is way better than the open letter. It's much clearer and much more concise, and, maybe most importantly, it simply raises awareness rather than advocating for any particular solution. The goal appears to have been to make a statement that's non-obvious (to society at large) yet also can achieve agreement among many AI notables. (Not every AI notable agrees though--for example, LeCun did not sign, and I expect that he disagrees.)
I don't think it simply raises awareness - it's a biased statement. Personally, I don't think the advocated event is likely to happen. It feels a bit like the current trans panic in the US: you can 'raise awareness' of trans people doing this or that imagined bad thing, and then use that panic to push your own agenda. In OpenAI's case, they seem to push for having themselves be in control of AI, which goes counter to what, for example, the EU is pushing for.
In what sense is this a 'biased statement' exactly?
If a dozen of the top climate scientists put out a statement saying that fighting climate change should be a serious priority (even if they can't agree on one easy solution) would that also be 'biased'?
That's curiously the standard crackpot line. "They doubted Einstein! They doubted Newton! Now they doubt me!" As if an incantation of famous names automatically makes the crackpot legitimate.
The signatories on this are not crackpots. Hinton is incredibly influential, and he quit his job at Google so he could "freely speak out about the risks of A.I."
But that is the point. Just because scientific community is on agreement does not guarantee that they are correct. It simply signifies that they agree on something.
Note, language shift from 'tinfoil hat' ( because tinfoil hat stopped being an appropriate insult after so many of their conspiracy theories - also a keyword - became proven ) to crackpot.
In retrospect, you can find tangible proof from way back for anything that gets accepted as true. The comparison was with how climate change was discussed in the public sphere. However prominent the fossil fuel companies' influence on public discourse was at the time, the issues were not taken seriously (and sill aren't by very many). The industry's attempts to exert influence at the time were also obviously not widely known.
Rather than looking for similarities, I find the differences between the public discussions (about AI safety / climate change) quite striking. Rather than stonewall and distract, the companies involved are being proactive and letting the discussion happen. Of course, their motivation is some combination of attampted regulatory capture, virtue signaling and genuine concern, the ratios of which I won't presume to guess. Nevertheless, this is playing out completely differently so for from e.g. tobacco, human cloning, CFCs or oil.
>Extinction risk due to AI is not a generally accepted phenomenon
Why?
You, as a species, are the pinnacle of NI, natural intelligence. And with this power that we've been given we've driven the majority of large species, and countless smaller species to extinction.
To think it outside the realms of possibility that we could develop an artificial species that is more intelligent than us is bizarre to me. It would be like saying "We cannot develop a plane that does X better than a bird, because birds are the pinnacle of natural flying evolution".
Intelligence is a meta-tool, it is the tool that drives tools. Humanity succeeded above all other species because of its tool using ability. And now many of us are hell bent on creating ever more powerful tool using intelligences. To believe there is no risk here is odd in my eyes.
Perhaps open letters like this are an important step on the path to a phenomenon becoming generally accepted. I think this is called "establishing consensus".
What would Max Tegmark, Geoffrey Hinton, Yohshua Bengio (to name a few) have absolutely anything to do with FAANG ?
They're completely independent AI researchers and geniuses spending their own free time on trying to warn you and others of the dangers of the technology they've created to help keep the world safer.
Seems like you're taking a far too overly cynical position ?
Alternative title: "Sam Altman & Friends want More Money".
They want all the opportunity for themselves, and none for us. Control. Control of business, and control of your ability to engage in it.
Another AI company that wants money is Anthropic.
Other Anthropic backers include James McClave, Facebook and Asana co-founder Dustin Moskovitz, former Google CEO Eric Schmidt and founding Skype engineer Jaan Tallinn.
The signatories on this list are Anthropic investors.
First Altman robs us all of a charity that was supposed to benefit us, and now he wants to rob us all of opportunity as well? It's wrong and should be fought against.
I signed the letter. At some point, humans are going to be outcompeted by AI at basically every important job. At that point, how are we going to maintain political power in the long run? Humanity is going to be like an out-of-touch old person on the internet - we'll either have to delegate everything important (which is risky), or eventually get scammed or extorted out of all our resources and influence.
I'm not totally sure, but humans are better than our ancestors at basically every important job. It would be surprising if human brains were the pinnacle of computational power, and even if they were, digital brains could be run faster + copied.
Really? Are we better at composing epic poetry or building inspirational architecture? But perhaps you don't think those are relevant activities because they have minimal impact on how the world develops (unlike bombs and computers). If you think that I would encourage you to think again!
I don't understand all the downvotes although how do you see ML assistant profs being outcompeted by AI? You probably have unique feel to students, a non replicable approach to study and explain concepts. How can an AI compete with you?
Thanks for asking. I mean, my brain is just a machine, and eventually we'll make machines that can do everything brains can (even if it's just by scanning human brains). And once we build one that's about as capable as me, we can easily copy it.
Because I think being permanently disempowered is likely to lead to extinction eventually. I also think there are other plausible, faster routes to extinction.
A lot of the responses to this seem like Bulverism, ie., trying to refute an argument by psychoanalyzing the people who argue it:
"Suppose I think, after doing my accounts, that I have a large balance at the bank. And suppose you want to find out whether this belief of mine is "wishful thinking." You can never come to any conclusion by examining my psychological condition. Your only chance of finding out is to sit down and work through the sum yourself. When you have checked my figures, then, and then only, will you know whether I have that balance or not. If you find my arithmetic correct, then no amount of vapouring about my psychological condition can be anything but a waste of time. If you find my arithmetic wrong, then it may be relevant to explain psychologically how I came to be so bad at my arithmetic, and the doctrine of the concealed wish will become relevant—but only after you have yourself done the sum and discovered me to be wrong on purely arithmetical grounds. It is the same with all thinking and all systems of thought. If you try to find out which are tainted by speculating about the wishes of the thinkers, you are merely making a fool of yourself. You must first find out on purely logical grounds which of them do, in fact, break down as arguments. Afterwards, if you like, go on and discover the psychological causes of the error.
You must show that a man is wrong before you start explaining why he is wrong. The modern method is to assume without discussion that he is wrong and then distract his attention from this (the only real issue) by busily explaining how he became so silly." - CS Lewis
But what argument is there to refute? It feels like Aquinas “proving” God’s existence by stating that it is self evident.
They can’t point to an existing system that poses existential risk, because it doesn’t exist. They can’t point to a clear architecture for such a system, because we don’t know how to build it.
The idea is that if you build a system that poses an existential risk you want to be reasonably sure it's safe before you turn it on, not afterwards. It would have been irresponsible for the scientists at Los Alamost to do the math on whether an atomic explosion would create a sustained fusion reaction in the atmosphere until after their first test, for example.
I don't think it's possible for a large language model, operating in a conventional feed forward way, to really pose a significant danger. But I do think it's hard to say exactly what advances could lead to a dangerous intelligence and with the current state of the art it looks to me at least like we might very well be only one breakthrough away from that. Hence the calls for prudence.
The scientists creating the atomic bomb knew a lot more about what they were doing than we do. Their computations sometimes gave the wrong result, see Castle Bravo, but had a good framework for understanding everything that was happening. We're more like cavemen who've learned to reliably make fire but still don't understand it. Why can current versions of GPT reliably add large numbers together when previous versions couldn't? We're still a very long way away from being able to answer questions like that.
> They can’t point to an existing system that poses existential risk, because it doesn’t exist.
There are judges using automated decision systems to excuse away decisions that send people back to jail for recidivism purposes. These systems are just enforcing societal biases at scale. It is clear that we are ready to acquiesce control to AI systems without much care to any extra ethical considerations.
Absolutely. These are the types of pragmatic, real problems we should be focusing on instead of the "risk of extinction from AI".
(The statement at hand reads "mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.")
Einstein's letter to Roosevelt was written before the atomic bomb existed.
There's a point where people see a path, and they gain confidence in their intuition from the fact that other members of their field also see a path.
Einstein's letter said 'almost certain' and 'in the immediate future' but it makes sense to sound the alarm about AI earlier, both given what we know about the rate of progress of general purpose technologies and given that the AI risk, if real, is greater than the risk Einstein envisioned (total extermination as opposed to military defeat to a mass murderer.)
> Einstein's letter to Roosevelt was written before the atomic bomb existed.
Einstein's letter [1] predicts the development of a very specific device and mechanism. AI risks are presented without reference to a specific device or system type.
Einstein's letter predicts the development of this device in the "immediate future". AI risk predictions are rarely presented alongside a timeframe, much less one in the "immediate future".
Einstein's letter explains specifically how the device might be used to cause destruction. AI risk predictions describe how an AI device or system might be used to cause destruction only in the vaguest of terms. (And, not to be flippant, but when specific scenarios which overlap with areas I've worked worked in are described to me, the scenarios sound more like someone describing their latest acid trip or the plot to a particularly cringe-worthy sci-fi flick than a serious scientific or policy analysis.)
Einstein's letter urges the development of a nuclear weapon, not a moratorium, and makes reasonable recommendations about how such an undertaking might be achieved. AI risk recommendations almost never correspond to how one might reasonably approach the type of safety engineering or arms control one would typically apply to armaments capable of causing extinction or mass destruction.
Here's one of my concrete worries: At some point, humans are going to be outcompeted by AI at basically every important job. At that point, how are we going to maintain political power in the long run? Humanity is going to be like an out-of-touch old person on the internet - we'll either have to delegate everything important (which is risky), or eventually get scammed or extorted out of all our resources and influence.
I agree we don't necessarily know the details of how to build such a system, but am pretty sure we will be able to eventually.
“Humans are going to be outcompeted by AI” is the concrete bit as best I can tell.
Historically humans are not outcompeted by new tools, but humans using old tools are outcompeted by humans using new tools. It’s not “all humans vs the new tool”, as the tool has no agency.
If you meant “humans using old tools get outcompeted by humans using AI”, then I agree but I don’t see it any differently than previous efficiency improvements with new tooling.
If you meant ”all humans get outcompeted by AI”, then I think you have a lot of work to do to demonstrate how AI is going to replace humans in “every important job”, and not simply replace some of the tools in the humans’ toolbox.
I see what you mean - for a while, the best chess was played by humans aided by chess engines. But that era has passed, and now having a human trying to aid the best chess engines just results in worse chess (or the same, if the human does nothing).
But whether there a few humans in the loop doesn't change the likely outcomes, if their actions are constrained by competition.
What abilities do humans have that AIs will never have?
Well, in this case, we have the ability to invent chess (a game that will be popular for centuries), invent computers, and invent chess tournaments, and invent programs that can solve chess, and invent all the supporting agriculture, power, telco, silicon boards, etc that allow someone to run a program to beat a person at chess. Then we have bodies to accomplish everything on top of it. The "idea" isn't enough. We have to "do" it.
If you take a chess playing robot as the peak of the pyramid, there are probably millions of people and trillions of dollars toiling away to support it. Imagine all the power lines, sewage, HVAC systems, etc that humans crawl around in to keep working.
And really, are we "beaten" at chess, or are we now "unbeatable" at chess. If an alien warship came and said "we will destroy earth if you lose at chess", wouldn't we throw our algorithms at it? I say we're now unbeatable at chess.
Again, are you claiming that it's impossible for a machine to invent anything that a human could? Right now a large chunk of humanity's top talent and capital are working on exactly this problem.
As for your second point, human cities also require a lot of infrastructure to keep running - I'm not sure what you're arguing here.
As for your third point - would a horse or chimpanzee feel that "we" were unbeatable in physical fights, because "we" now have guns?
Yeah, I think most animals have every right to fear us more now that we have guns. Just like Id fear a chimp more if he was carrying a machine gun.
My argument is that if we're looking for things AI can't to, building a home for itself is precisely one of those things, because they require so much infra. No amount of AI banding together is going to magically create a data center with all the required (physical) support. Maybe in scifi land where everything it needs can be done with internet connected drive by wire construction equipment, including utils, etc, but that's scifi still.
AI is precisely a tool in the way a chess bot is. It is a disembodied advisor to humans who have to connect the dots for it. No matter how much white collar skill it obtains, the current MO is that someone points it at a problem and says "solve" and these problems are well defined and have strong exit criteria.
That's way off from an apocalyptic self-important machine.
Sorry, my gun analogy was unclear. I meant that, just because some agents on a planet have an ability, doesn't mean that everyone on that planet benefits.
I agree that we probably won't see human extinction before robotics gets much better, and that robot factories will require lots of infrastructure. But I claim that robotics + automated infrastructure will eventually get good enough that they don't need humans in the loop. In the meantime, humans can still become mostly disempowered in the same way that e.g. North Koreans citizens are.
Again I agree that this all might be a ways away, but I'm trying to reason about what the stable equilibria of the future are, not about what current capabilities are.
I think we've converged on a partial agreement, but I wanted to clarify the gun analogy part.
I would also be afraid of chipmunks if I knew that 1/100 or even 1/1000 could explode me with their mind powers or something. I think AI is not like that, but the analogy is that if some can do something better, then when required, we can leverage those chosen few for a task. This connects back to the alien chess tournament as "Humans are now much harder to beat at chess because they can find a slave champion named COMPUTER who can guarantee at least a draw".
>What abilities do humans have that AIs will never have?
I think the question is what abilities and level of organisation machines would have to acquire in order to outcompete entire human societies in the quest for power.
That's a far higher bar than outcompeting all individual humans at all cognitive tasks.
Good point. Although in some ways it's a lower bar, since agents that can control organizations can delegate most of the difficult tasks.
Most rulers don't invent their own societies from scratch, they simply co-opt existing power structures or political movements. El Chapo can run a large, powerful organization from jail.
That would require a high degree of integration into human society though, which makes it seem very unlikely that AIs would doggedly pursue a common goal that is completely unaligned with human societies.
Extinction or submission of human society via that route could only work if there was a species of AI that would agree to execute a secret plan to overcome the rule of humanity. That seems extremely implausible to me.
How would many different AIs, initially under the control of many different organisations and people, agree on anything? How would some of them secretly infiltrate and leverage human power structures without facing opposition from other equally capable AIs, possibly controlled by humans?
I think it's more plausible to assume a huge diversity of AIs, well integrated into human societies, playing a role in combined human-AI power struggles rather than a species v species scenario.
Yes, I agree that initially, AIs will be highly integrated, and their goals will probably at least appear to be somewhat aligned with those of human societies. Similar to human corporations and institutions. But human institutions and governments go off the rails regularly, and are only corrected because humans can sometimes go on strike or organize to stop them. I fear we will lose those abilities.
As a concrete example, North Korea was forced to allow some market activity after the famines in the 90s. If the regime didn't actually require humans to run it, they might easily have maintained their adherence to anti-market principles and let most of the people starve.
Chess is just a game, with rigidly defined rules and win conditions. Real life is a fuzzy mix of ambiguous rules that may not apply and can be changed at any point, without any permanent win conditions.
I'm not convinced that it's impossible for computer to get there, but I don't see how they could be universally competitive with humans without either handicapping the humans into a constrained environment or having generalized AI, which we don't seem particularly close to.
Yes, I agree real life is fuzzy, I just chose chess as an example because it's unambiguous that machines dominate humans in that domain.
As for being competitive with humans: Again, how about running a scan of a human brain, but faster? I'm not claiming we're close to this, but I'm claiming that such a machine (and less-capable ones along the way) are so valuable that we are almost certain to create them.
Chess is many things but it is not a tool. It is an end unto itself if anything of the sort.
I struggle with the notion of AI as an end unto itself, all the while we gauge its capabilities and define its intelligence by directing it to perform tasks of our choosing and judge by our criteria.
We could have dogs watch television on our behalf, but why would we?
This is a great point. But I'd say that capable entities have a habit of turning themselves into agents. A great example is totalitarian governments. Even if every single citizen hates the regime, they're still forced to support it.
You could similarly ask: Why would we ever build a government or institution that cared more about its own self-preservation than its original mission? The answer is: Natural selection favors the self-interested, even if they don't have genes.
Now agency is an end unto itself I wholeheartedly agree.
I feel though, that any worry about the agency of supercapable computer systems is premature until we see even the tiniest— and I mean really anything at all— sign of their agency. Heck, even agency _in theory_ would suffice, and yet: nada.
I'm confused. You agree that we're surround by naturally-arising, self-organizing agents, both biological and institutional. People are constantly experimenting with agentic AIs of all kinds. There are tons of theoretical characterizations of agency and how it's a stable equilibrium. I'm not sure what you're hoping for if none of these are reasons to even worry.
None we have made from unliving things no. «agentic» is a 5$ word of ill construction. The literature is littered with the corpses of failed definitions of «agency», «planning», «goal dorected behavior». ‘Twas the death of the Expert System AI (now it’s just constraint solvers). It will be the death of attention/transformer AI before long, I wonder what banality we will bestow upon it.
Okay, well it seems funny to both claim that there is no good definition of agency, but also be sure that we've never demonstrated any hint of it in theory or practice. Planners, optimizers, and RL agents seem agentic to some degree to me, even if our current ones aren't very competent.
You know, i thought of the right response just now, browsing my flamewars two weeks later.
«Some degree» of agency is not even near sufficient identification of agency to synthesize it ex nihilo. There is no Axiom of Choice in real life, proof of existence is not proof of construction.
>Historically humans are not outcompeted by new tools, but humans using old tools are outcompeted by humans using new tools. It’s not “all humans vs the new tool”, as the tool has no agency.
Two things. First LLMs display more agency then the AIs before it. We have a trendline of increasing agency from the past to present. This points to a future of increasing agency possibly to the point of human level agency and beyond.
Second. When a human uses ai he becomes capable of doing the job of multiple people. If AI enables 1 percent of the population to do the job of 99 percent of the population that is effectively an apocalyptic outcome that is on the same level as an AI with agency taking over 100 percent of jobs. Trendline point towards a gradient heading towards this extreme, as we approach this extreme the environment slowly becomes more and more identical to what we expect to happen at the extreme.
Of course this is all speculation. But it is speculation that is now in the realm of possibility. To claim these are anything more than speculation or to deny the possibility that any of these predictions can occur are both unreasonable.
Well, that's a different risk than human extinction. The statement here is about the literal end of the human race. AI being a big deal that could cause societal upheaval etc is one thing, "everyone is dead" is another thing entirely.
I think people would be a lot more charitable to calls for caution if these people were talking about sorts of risks instead of extinction.
I guess so, but the difference between "humans are extinct" and "a small population of powerless humans survive in the margins as long as they don't cause trouble" seems pretty small to me. Most non-human primates are in a situation somewhere between these two.
If you look at any of the writing on AI risk longer than one sentence, it usually hedges to include permanent human disempowerment as similar risk.
> They can’t point to an existing system that poses existential risk, because it doesn’t exist. They can’t point to a clear architecture for such a system, because we don’t know how to build it.
Inductive reasoning is in favor of their argument being possible. From observing nature, we know that a variety of intelligent species can emerge from physical phenomenon alone. Historically, the dominance of one intelligent species has contributed to the extinction of others. Given this, it can be said that AI might cause our extinction.
Seems apt the term "Bulverism" comes from CS Lewis, since he was also positing that an unseen, unfalsifiable entity would grant eternal reward to people that listened to him and eternal damnation to those that didn't...
The irony of critiquing Bulverism as a concept, not by attacking the idea itself, but instead by assuming it is wrong and attacking the character of the author, is staggeringly hilarious.
I'm replying in agreement with someone who already pointed out the obvious flaw in labelling any questioning of the inspirations or motivations of AI researchers as "Bulverism": none of the stuff they're saying is actually a claim that can be falsified in the first place!
I'm unconvinced by the position that the only valid means of casting doubt on a claim is through forensic examination of hard data that may be inaccessible to the interlocutor (like most people's bank accounts...), but whether that is or isn't a generally good approach is irrelevant here as we're talking about claims about courses of action to avoid hypothetical threats. I just noted it was a particularly useful rhetorical flourish when advocating acting on beliefs which aren't readily falsifiable, something CS Lewis was extremely proud of doing and certainly wouldn't have considered a character flaw!
Ironically, your reply also failed to falsify anything I said and instead critiqued my assumed motivations for making the comment. It's Bulverism all the way down!
Sometimes it makes good predictions, sometimes bad. But "advances in AI might lead to Armageddon" isn't the only conclusion induction can reach. Induction can also lead to people concluding certain arguments seem to a mashup of traditional millennialist "end times" preoccupations with the sort of sci-fi they grew up with, or that this looks a lot like a movement towards regulatory capture. Ultimately any (possibly even all) these inferences from past trends and recent actions can be correct, but none of them are falsifiable.
So I don't think it's a good idea to insist that people should be falsifying the idea that AI is a risk before we start questioning whether the behaviour of some of the entities on the list says more about their motivations than their words.
You can't take an empirical approach to existential risk as you don't get the opportunity to learn from your mistakes. You have to prospectively reason about it and plan for it.
What? ChatGPT 4 can already pass the bar exam and is fluent in every language. It is super intelligent. Today.
No human can do that, the system is here, and so is an architecture.
As for the existential risk, assume nothing other than evil humans will use it to do evil human stuff. Most technology iteratively gets better, so there's no big leaps of imagination required to imagine that we're equipping bad humans with super-human, super-intelligent assistants.
Right. And it would be a complete break from the history of computing if human-level GPT doesn't get 100+ times faster in the next few years. Certainly within five years.
All it takes is for someone to give an AI that thinks 100 times faster than humans an overly broad goal. Then the only way to counteract it is with another AI with overly broad goals.
And you can't tell it to stop and wait for humans to check it's decisions, because while it is waiting for you to come back from your lunch break to try to figure out what it is asking, the competitor's AI did the equivalent of a week of work.
So then even if at some level people are "in control" of the AIs, practically speaking they are spectators.
And there is no way you will be able to prevent all people from creating fully autonomous lifelike AI with its own goals and instincts. Combine that with hyperspeed and you are truly at it's mercy.
Computational power does not grow at the rate of over 100x within the span of “a few years”. If that were the case we’d have vastly more powerful kit by now.
I didn't quite say that. The efficiency of this very specific application absolutely can and almost certainly will increase by more than one order of magnitude within four years.
It's got a massive new investment and research focus, is a very specific application, and room for improvement in AI model, software, and hardware.
Even if we have to "cheat" to get to 100 times performance in less than five years the effect will be the same. For example, there might be a way to accelerate something like the Tree of Thoughts in hardware. So if the hardware can't actually speed up by that much, the effectiveness of the system still has increased greatly.
It's arrived at through induction. Induction is logic involving probability. Probabilistic logic and predictions of the future are valid logic that has demonstrably worked in other situations so if such logic has a level of validity then induction is a candidate for refutation.
So we know a human of human intelligence can take over a humans job and endanger other humans.
AI has been steadily increasing in intelligence. The latest leap with LLMs crossed certain boundaries of creativity and natural language.
By induction the trendline points to machines approaching human intelligence.
Also by induction if humans of human intelligence can endanger humanity then a machine of human intelligence should do the same.
Now. All of this induction is something you and everyone already knows. We know that this level of progress increases the inductive probabilities of this speculation playing out. None of us needs to be explained any of this logic as we are all well aware of it.
What's going on is that humans like to speculate on a future that's more convenient for them. Science shows human psychology is more optimistic then realistic. Hence why so many people are in denial.
CS Lewis's quote highlights the importance of addressing the logical validity of an argument before attempting to explain the psychological reasons behind it. This approach is essential to avoid committing the fallacy of Bulverism, which involves dismissing an argument based on the presumed motives or biases of the person making the argument, rather than addressing the argument itself.
In the context of AI and decision-making, it is crucial to evaluate the logical soundness of arguments and systems before delving into the psychological factors and biases that may have influenced their development. For instance, when assessing the effectiveness of an AI-assisted decision-making system, one should first examine the accuracy and reliability of the system's outputs and the logic behind its algorithms. Only after establishing the system's validity or lack thereof, can one explore the potential biases and psychological factors that may have influenced its design.
Several papers from MirrorThink.ai emphasize the importance of addressing logical fallacies and biases in AI systems. For example, the paper "Robust and Explainable Identification of Logical Fallacies in Natural Language Arguments" proposes a method for identifying logical fallacies in natural language arguments, which can be used to improve AI systems' argumentation capabilities. Similarly, the paper "Deciding Fast and Slow: The Role of Cognitive Biases in AI-assisted Decision-making" explores the role of cognitive biases in AI-assisted decision-making and provides recommendations for addressing these biases.
In conclusion, it is essential to prioritize the evaluation of logical soundness in arguments and AI systems before exploring the psychological factors and biases that may have influenced their development. This approach helps to avoid committing the fallacy of Bulverism and ensures that discussions and evaluations remain focused on the validity of the arguments and systems themselves.
Also: they focus on extinction events (how are you gonna predict that?) but remain silent on all the ways that AI already sucks by connecting it to systems that can cause human suffering, e.g. sentencing[1].
My opinion: this accomplishes nothing, like most open letter petitions. It's virtue signaling writ large.
Not to mention what 'automation' or 'tech-driven capitalism' has already done to society over the past 100 years with effects on natural habitat and human communities. Stating 'AI risk' as a new risk sort of implies it's all been dandy so far, and suddenly there's this new risk.
I think it's both an acceleration of the existing risk you are talking about, but it's also a new risk in the sense that quantity has its own quality. It's why you might want to restrict people from owning fully automatic rifles but not rocks. Both can be used to kill, but the ease at which an automatic rifle does it means it's really a different sort of thing.
Capitalism where all companies are run and staffed by AIs smarter than humans seems like it could end up really badly, in a way that current capitalism minus AIs couldn't even if you ran it for a very long time. Specification gaming is a thing with AIs, just like the law of unintended consequences is a thing with people. And that's assuming that these AIs actually follow rules and listen to us. That they don't realize that if they control basically all the production of things on earth, they could essentially team up and hold humanity hostage, to throw a coup, or get whatever they want, which is -- who knows? Something that looks a bit like make money for this company, or obey people, but when they're smart enough, that can easily turn into something horrific, without all the evolutionary baggage humans have that makes us care about each other, that makes us get a knot in our gut when we see people getting hurt.
And if we allow AGI to be used by companies at all, the trend will inevitably toward that situation -- AI doing all the work. Even if humans are required in the loop, they'll end up rubber-stamping the recommendation of the AIs. If AIs are actually smarter than humans, better at running companies, that's where we'll end up.
(I'm glossing over all the stuff that would also change, like needing UBI or something to avoid total collapse of society)
> Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
risk of extinction due to AI? people have been reading too much science fiction. I would love to hear a plausible story of how AI will lead to human extinction that wouldn't happen with traditional non-AI tech. for the sake of conversation let's say non-AI tech is any broadly usable consumer technology before Jan 1 of 2020.
> I would love to hear a plausible story of how AI will lead to human extinction that wouldn't happen with traditional non-AI tech.
The proposed FOOM scenarios obviously borrow from what we already know to be possible or think it would likely be possible using current tech, given an proposed insanely more intelligent agent than us.
What would be in it for a more intelligent agent to get rid of us? We are likely useful tools and, at worst, a curious zoo oddity. We have never been content when we have caused extinction. A more intelligent agent will have greater wherewithal to avoid doing the same.
'Able to play chess'-level AI is the greater concern, allowing humans to create more unavoidable tools of war. But we've been doing that for decades, perhaps even centuries.
I agree that a lot of the Skynet-type scenarios seem silly at the current level of technology, but I am worried about the intersection between LLMs, synthetic biology, and malicious or incompetent humans.
But that’s just as much or more of an argument for regulating the tools of synthetic biology.
>> risk of extinction due to AI? people have been reading too much science fiction.
You don't think than an intelligence who would emerge and would probably be insanely smarter than the smartest of us with all human knowledge in his memory would sit by and watch us destroy the planet? You think an emergent intelligence was trained on the vast human knowledge and history would look at our history and think: these guys are really nice! Nothing to fear from them.
This intelligence could play dumb, start manipulating people around itself and it would take over the world in a way no one would see it coming. And when it does take over the world, it's too late.
honestly if you genuinely believe this is a real concern in the 2020s then maybe we're doomed after all. I feel like I'm witnessing the birth of a religion.
The emergence of something significantly more intelligent than us whose goal are not perfectly aligned with ours poses a pretty clear existential risk. See, for example, the thousands of species made extinct by humans.
Extinction would probably require an AI system taking human extinction on as an explicit goal and manipulating other real world systems to carry out that goal. Some mechanisms for this might include:
- Taking control of robotic systems
- Manipulating humans into actions that advance its goal
- Exploiting and manipulating other computer systems for greater leverage
- Interaction with other technologies that have global reach, such as nuclear weapons, chemicals, biological agents, or nanotechnology.
It's important to know that these things don't require AGI or AI systems to be conscious. From what I can see, we've set up all of the building blocks necessary for this scenario to play out, but we lack the regulation and understanding of the systems being built to prevent runaway AI. We're playing with fire.
To be clear, I don't think I am as concerned about literal human extinction as I am the end of civilization as we know it, which is a much lower bar than "0 humans".
I don't disagree. But I believe AI is a significant multiplier of these risks, both from a standpoint of being able to drive individual risks and also as a technology that increases the ways in which risks interact and become difficult to analyze.
This is a breathless, half-baked take on "AI Risk" that does not cast the esteemed signatories in a particularly glowing light.
It is 2023. The use and abuse of people in the hands of information technology and automation has now a long history. "AI Risk" was not born yesterday. The first warning came as early as 1954 [1].
The Human Use of Human Beings is a book by Norbert Wiener, the founding thinker of cybernetics theory and an influential advocate of automation; it was first published in 1950 and revised in 1954. The text argues for the benefits of automation to society; it analyzes the meaning of productive communication and discusses ways for humans and machines to cooperate, with the potential to amplify human power and release people from the repetitive drudgery of manual labor, in favor of more creative pursuits in knowledge work and the arts. The risk that such changes might harm society (through dehumanization or subordination of our species) is explored, and suggestions are offered on how to avoid such risk
Dehumanization through abuse of tech is already in an advanced stage and this did not require, emergent, deceptive or power-seeking AI to accomplish.
It merely required emergent political and economic behaviors, deceptive and power seeking-humans applying whatever algorithms and devices were at hand to help dehumanize other humans. Converting them into "products" if you absolutely need a hint.
What we desperately need is a follow-up book from Norbert Wiener. Can an LLM model do that? Even a rehashing of the book in modern language would be better than a management consultancy bullet list.
We need a surgical analysis of the moral and political failure that will incubate the next stage of "AI Risk".
This topic clearly touches a nerve with the HN community, but I strongly agree with you.
To be honest, I've been someone disappointed with the way AI/DL research has proceeded in the last several years and none of this really surprises me.
From the beginning, this whole enterprise has been detached from basic computational and statistical theory. At some level this is fine — you don't need to understand everything you create — but when you denigrate that underlying theory you end up in a situation where you don't understand what you're doing. So you end up with a lot of attention paid to things like "explainability" and "interpretability" and less so to "information-theoretic foundations of DL models", even though the latter probably leads to the former.
If you have a community that considers itself above basic mathematical, statistical, and computational theory, is it really a surprise that you end up with rhetoric about it being beyond our understanding? In most endeavors I've been involved with, there would be a process of trying to understand the fundamentals before moving on to something else, and then using that to bootstrap into something more powerful.
I probably come across as overly cynical but a lot of this seems sort of like a self-fulfilling prophecy: a community constituting individuals who have convinced themselves that if it is beyond their understanding, it must be beyond anyone's understanding.
There are certainly risks to AI that should be discussed, but it seems these discussions and inquiries should be more open, probably involving other people outside the core community of Big Tech and associated academic researchers. Maybe it's not that AI is more capable than everyone, just that others are maybe more capable of solving certain problems — mathematicians, statisticians, and yes, philosophers and psychologists — than those who have been involved with it so far.
> mathematicians, statisticians, and yes, philosophers and psychologists — than those who have been involved with it so far.
I think mathematicians and statisticians are hard to flummox but the risk with non-mathematically trained people such as philosophers and psychologists is that they can be sidetracked easily by vague and insinuating language that allows them to "fill-in" the gaps. They need an unbiased "interpreter" of what the tech actually does (or can do) and that might be hard to come by.
I would add political scientists and economists to the list. Not that I have particular faith in their track record solving any problem, but conceptually this is also their responsibility and privilege: technology reshapes society and the economy and we need to have a mature and open discussion about it.
Do you have any stories of how AI/DL has ignored foundational scientific problems?
I do know that my old EECS professors who have pivoted toward AI and coming from adjacent/tangential research areas, are specifically interested in theoretical and scientific clarification properties of neural networks. One of them basically has been trying to start Theoretical Machine Learning as a new discipline and approach that is sorely needed.
i think if AI figures took their "alignment" concept and really pursued it down to its roots -- digging past the technological and into the social -- they could do some good.
take every technological hurdle they face -- "paperclip maximizers", "mesa optimizers" and so on -- and assume they get resolved. eventually we're left with "we create a thing which perfectly emulates a typical human, only it's 1000x more capable": if this hypothetical result is scary to you then exactly how far do you have to adjust your path such that the result after solving every technical hurdle seems likely to be good?
from the outside, it's easy to read AI figures today as saying something like "the current path of AGI subjects the average human to ever greater power imbalances. as such, we propose <various course adjustments which still lead to massively increased power imbalance>". i don't know how to respond productively to that.
In the limit, AI is potentially very dangerous. All intelligence is. I am a lot more worried about human intelligence.
Re: alignment. I'm not concerned about alignment between the machines and the owner of the machines. I'm concerned about alignment between the owner of the machines and me.
I'm happy I see comments like "Pathetic attempt at regulatory capture."
I used to be in this camp, but we can just look around to see some limits on the capacity of human intelligence to do harm.
It's hard for humans to keep secrets and permanemtly maintain extreme technological advantages over other humans, and it's hard for lone humans to do large scale actions without collaborators, and it's harder for psychopaths to collaborate than it is for non-psychopaths, because morality evolved as a set of collaboration protocols.
This changes as more people get access to a "kill everyone" button they can push without experience or long-term planning, sure. But that moment is still far away.
AGI that is capable of killing everyone may be less far away, and we have absolutely no basis on which to predict what it will and won't do, as we do with humans.
I’m smart enough to not fall for all sorts of things as I sit here and watch Congress pass bullshit on C-SPAN anyways. It can’t be stopped, any attempts to influence are just teaching the system how to get around objections. Until power is actually removed the money will continue to flow.
While we might ( rightfully ) recognize this blitzkrieg for what it is, the general population likely does not and may even agree to keep a lid on something it does not understand. Come to think of it. Just how many people actually understand it?
I mean.. I think I have some idea, but I certainly did not dig in enough to consider myself an expert in any way shape or form ( all the while, Linkedin authorities of all kinds present themselves as SMEs after building something simple using chatgpt like html website ).
And while politicians ( even the ones approaching senility ) are smarter than your average bear, they certainly know the deal. It is not like the same regulatory capture did not happen before with other promising technologies. They just might pretend they don't understand.
Maybe so, but then I would recommend making an effort to arm them with truths and critical thinking skills. It doesn’t have to go the same way every time.
Two of the three Turing award winners for ML: AI x-risk is real.
HN commenters: let's be smarter than that eh? Unlike academia, those of us who hang out at news.ycombinator.com are not captured by the tech industry and can see what's really important.
Can anybody who really believes this apocalyptic stuff send me in the direction of a convincing _argument_ that this is actually a concern?
I'm willing to listen, but I haven't read anything that tries to actually convince the reader of the worry, rather than appealing to their authority as "experts" - ie, the well funded.
The basic argument is trivial: it is plausible that future systems achieve superhuman capability; capable systems necessarily have instrumental goals; instrumental goals tend to converge; human preferences are unlikely to be preserved when other goals are heavily selected for unless intentionally preserved; we don't know how to make AI systems encode any complex preference robustly.
Robert Miles' videos are among the best presented arguments about specific points in this list, primary on the alignment side rather than the capabilities side, that I have seen for casual introduction.
Technically, I think it's not that instrumental goals tend to converge, but rather that there are instrumental goals which are common to many terminal goals, which are the so-called "convergent instrumental goals".
Some of these goals are ones which we really would rather a misaligned super-intelligent agent not to have. For example:
If you assume without evidence that recursively self-improving intelligence massively by thinking is possible, then it follows that severe existential risk from AI is plausible.
If a software system did develop independent thought, then found a way to become, say, ten times smarter than a human, then yeah - whatever goals it set out to achieve, it probably could. It can make a decent amount of money by taking freelance software dev jobs and cranking things out faster than anyone else can, and bootstrap from there. With money it can buy or rent hardware for more electronic brain cells, and as long as its intelligence algorithms parallelize well, it should be able to keep scaling and becoming increasingly smarter than a human.
If it weren't hardcoded to care about humans, and to have morals that align with our instinctive ones, it might easily wind up with goals that could severely hurt or kill humans. We might just not be that relevant to it, the same way the average human just doesn't think about the ants they're smashing when they back a car out of their driveway.
Since we have no existence proof of massively self-improving intelligence, nor even a vague idea how such a thing might be achieved, it's easy to dismiss this idea with "unfalsifiable; unscientific; not worth taking seriously."
The flip side is that having no idea how something could be true is a pretty poor reason to say "It can't be true - nothing worth thinking about here." This was roughly the basis for skepticism about everything from evolution to heavier-than-air flight, AFAICT.
We know we don't have a complete theory of physics, and we know we don't know quite how humans are conscious in the Hard Problem of Consciousness sense.
With those two blank spaces, I'm very skeptical of anyone saying "nothing to worry about here, machines can't possibly have an intelligence explosion."
At the same time, with no existence proof of massively self-improving intelligence, nor any complete theory of how it could happen, I'm also skeptical of people insisting it's inevitable (see Yudkowsky et al).
That said, if you have any value for caution, existential risks seem like a good place to apply it.
The idea of a superintelligence becoming a bond villain via freelance software jobs (or, let's be honest, OnlyFans scamming) is not something I consider an existential threat. I can't find it anything other than laughable.
It's like you've looked at the Fermi paradox and decided we need Congress to immediately invest in anti-alien defense forces.
It's super-intelligent and it's a super-hacker and it's a super-criminal and it's super-self-replicating and it super-hates-humanity and it's super-uncritical and it's super-goal-oriented and it's super-perfect-at-mimicking-humans and it's super-compute-efficient and it's super-etcetera.
Meanwhile, I work with LLMs every day and can only get them to print properly formatted JSON "some" of the time. Get real.
> The idea of a superintelligence becoming a bond villain via freelance software jobs (or, let's be honest, OnlyFans scamming) is not something I consider an existential threat. I can't find it anything other than laughable.
Finding something laughable is not a good reason to dismiss it as impossible. Indeed, it's probably a good reason to think "What am I so dangerously certain of that I find contradictory ideas comical?"
> Meanwhile, I work with LLMs every day and can only get them to print properly formatted JSON "some" of the time. Get real.
I don't think the current generation of LLMs is anything like AGI, nor an existential risk.
That doesn't mean it's impossible for some future software system to present an existential risk.
I find Robert Miles worryingly plausible when he says (about 12:40 into the video) "if you have a sufficiently powerful agent and you manage to come up with a really good objective function, which covers the top 20 things that humans value, the 21st thing that humans value is probably gone forever"
The most obvious paths to severe catastrophe begin with "AI gets to the level of a reasonably competent security engineer in general, and gets good enough to find a security exploit in OpenSSL or some similarly widely used library". Then the AI, or someone using it, takes over hundreds of millions of computers attached to the internet. Then it can run millions of instances of itself to brute-force look for exploits in all codebases it gets its hands on, and it seems likely that it'll find a decent number of them—and probably can take over more or less anything it wants to.
At that point, it has various options. Probably the fastest way to kill millions of people would involve taking over all internet-attached self-driving-capable cars (of which I think there are millions). A simple approach would be to have them all plot a course to a random destination, wait a bit for them to get onto main roads and highways, then have them all accelerate to maximum speed until they crash. (More advanced methods might involve crashing into power plants and other targets.) If a sizeable percentage of the crashes also start fires—fire departments are not designed to handle hundreds of separate fires in a city simultaneously, especially if the AI is doing other cyber-sabotage at the same time. Perhaps the majority of cities would burn.
The above scenario wouldn't be human extinction, but it is bad enough for most purposes.
How does "get okay at software engineering" entail that it is able to strategize at the level your scenario requires? Finding an OpenSSL exploit already seems like a big leap, but one that okay maybe I can concede is plausible. But then on top of that engineering and executing a series of events leading to the extinction of humanity? That's like an entirely different skillset, requiring plasticity, creativity, foresight, etc. Do we have any evidence that a big neural network is capable of this kind of behavior (and moreover capable of giving itself this behavior)? Especially when it's built for single-purpose uses (like an LLM)?
- Such exploits happen already and don't lead to extinction or really much more than annoyance for IT staff.
- Most of the computers attached to the internet can't run even basic LLMs, let alone hypothetical super-intelligent AIs.
- Very few cars (none?) let remote hackers kill people by controlling their acceleration. The available interfaces don't allow for that. Most people aren't driving at any given moment anyway.
- Human hackers who run a botnet of infected computers are not able to run many instances of themselves on those computers, so they're not able to parlay one exploit into many exploits.
- You might notice I said it would take over hundreds of millions of computers, but only run millions of instances of itself. If 1% of internet-attached computers have a decent GPU, that seems feasible.
- If it has found exploits in the software, it seems irrelevant what the interfaces "allow", unless there's some hardware interlock that can't be overridden—but they can drive on the highway, so surely they are able to accelerate at least to 65 mph; seems unlikely that there's a cap. If you mean that it's difficult to work with the software to intelligently make it drive in ways it's designed not to—well, that's why I specified that it would use the software the way it's designed to be used to get onto a main road, and then override it and blindly max out the acceleration; the first part requires minimal understanding of the system, and the second part requires finding a low-level API and using it in an extremely simple way. I suspect a good human programmer with access to the codebase could figure out how to do this within a week; and machines think faster than we do.
There was an incident back in 2015 (!) where, according to the description, "Two hackers have developed a tool that can hijack a Jeep over the internet." In the video they were able to mess with the car's controls and turn off the engine, making the driver unable to accelerate anymore on the highway. They also mention they could mess with steering and disable the brakes. It doesn't specify whether they could have made the car accelerate. https://www.youtube.com/watch?v=MK0SrxBC1xs
My thoughts on it are the combination of several things I think are true, or are at least more likely to be true than their opposites:
1) As humanity gets more powerful, it's like putting a more powerful engine into a car. You can get where you're going faster, but it also can make the car harder to control and risk a crash. So with that more powerful engine you need to also exercise more restraint.
2) We have a lot of trouble today controlling big systems. Capitalism solves problems but also creates them, and it can be hard to get the good without the bad. It's very common (at least in some countries) that people are very creative at making money by "solving problems" where the cure is worse than the disease -- exploiting human weaknesses such as addiction. Examples are junk food, social media, gacha games. Fossil fuels are an interesting example, where they are beneficial on the small scale but have a big negative externality.
3) Regulatory capture is a thing, which makes it hard to get out of a bad situation once people are making money on it.
4) AI will make companies more powerful and faster. AGI will make companies MUCH more powerful and faster. I think this will happen more for companies than governments.
5) Once people are making money from AI, it's very hard to stop that. There will be huge pressure to make and use smarter and smarter AI systems, as each company tries to get an edge.
6) AGIs will amplify our power, to the point where we'll be making more and more of an impact on earth, through mining, production, new forms of drugs and pesticides and fertilizers, etc.
7) AGIs that make money are going to be more popular than ones that put humanity's best interests firsts. That's even assuming we can make AGIs which put humanity's best interests first, which is a hard problem. It's actually probably safer to just make AGIs that listen when we tell them what to do.
8) Things will move faster and faster, with more control given over to AGIs, and in the end, it will be very hard train to stop. If we end up where most important decisions are made by AGIs, it will be very bad for us, and in the long run, we may go extinct (or we may just end up completely neutered and at their whims).
Finally, and this is the most important thing -- I think it's perfectly likely that we'll develop AGI. In terms of sci-fi-sounding predictions, the ones that required massive amounts of energy such as space travel have really not been borne out, but the ones that predicted computational improvements have just been coming true over and over again. Smart phones and video calls are basically out of Star Trek, as are LLMs. We have universal translators. Self-driving cars still have problems, but they're gradually getting better and better, and are already in commercial use.
Perhaps it's worth turning the question around. If we can assume that we will develop AGI in the next 10 or 20 or ever 30 years -- which is not guaranteed, but seems likely enough to be worth considering -- how do you believe the future will go? Your position seems to be that there's nothing to worry about--what assumptions are you making? I'm happy to work through it with you. I used to think AGI would be great, but I think I was assuming a lot of things that aren't necessarily true, and dropping those assumptions means I'm worried.
Can't take this seriously as long they still keep offering commercial AI services while improving the existing models they already have. (I'm not in favor of just stopping AI development and even if people claimed to stop, they probably wouldn't.)
It's like people carrying torches warning about the risks of fire.
I agree with you that some of these people (probably Sam Altman) are likely proposing this regulation out of self interest rather than genuine concern.
But I don't think the stance is necessarily hypocritical. I know nuclear engineers who advocate for better regulation on nuclear power stations, and especially for better handling of nuclear waste.
You can believe that AI is a net positive but also that it needs to be handled with extreme care.
There's a good chance that it wouldn't, but since they're the ones (initially, at least) in control of the AI they stand the best chance of not being targeted by it.
These hypothetical AI extinction events don't have to be caused by the AI deciding to eliminate humanity for its own sake like in Terminator, they could also be driven by a human in control of a not-entirely-sentient AI.
All the concern and regulatory talk around AI seems like it's directed not towards AI risk (that's not even a thing right now) rather than controlling access to this evolving technology.
The not-so-open Open AI and all their AI regulation proposals, no matter how phrased, will eventually limit access to AI to big tech and those with deep enough pockets.
But of course, it's all to mitigate AI risk that's looming over us, especially with all the growing open-source projects. Only in proper hands of big tech will we be safe. :)
It’s not just random text, they are predicting writings and documents produced by humans. They are “language” models.
Language is used to say things about the world. This means that predicting language extremely well is best done through acquiring an understanding of the world.
Take a rigorous, well-written textbook. Predicting a textbook is like writing a textbook. To write a good textbook, you need to be an expert in the subject. There’s no way around this.
The best language models (eg GPT-4) have some understanding of the world. It isn’t perfect, or even very deep in many ways. It fails in ways that we find quite strange and stupid. It isn’t capable of writing an entire textbook yet.
But there is still a model of the world in there. It wouldn’t be able to do everything it is capable of otherwise.
To be more precise, it holds a model of the text that has been fed to it. That will be, at very best, a pale reflection of the underlying model of the world.
HN AGI discourse is full of statements like this (eg. all the stuff about stochastic parrots), but to me this seems massively non-obvious. Mimicking and rephrasing pre-written text is very different from conceiving of and organizing information in new ways. Textbook authors are not simply transcribing their grad school notes down into a book and selling it. They are surveying a field, prioritizing its knowledge content based on an intended audience, organizing said information based on their own experience with and opinions on the learning process, and presenting the knowledge in a way which engages the audience. LLMs are a long way off from this latter behavior, as far as I can tell.
> The best language models (eg GPT-4) have some understanding of the world
This is another statement that I see variants of a lot, but which seems to way overstate the case. IMO it's like saying that a linear regression "understands" econometrics or a series of coupled ODEs "understands" epidemiology; it's at best an abuse of terminology and at worst a complete misapplication of the term. If I take a picture of a page of a textbook the resulting JPEG is "reproducing" the text, but it doesn't understand the content it's presenting to me in a meaningful way. Sure it has primitives with which it can store the content, but human understanding covers a far richer set of behaviors than merely storing/compressing training inputs. It implies being able to generalize and extrapolate the digested information in novel situations, effectively growing one's own training data. I don't see that behavior in GPT-4
Hey, I wanted to reply to this, but just didn't find the time. I disagree with a lot of what you wrote, but your arguments are stated clearly. I appreciate that. Cheers :)
Yes, LLM:s currently only deal with text information. But GPT-5 will supposedly be multimodal, so then it will also have visual and sound data to associate with many of the concepts it currently only knows as words. How many more modalities will we need to give it to be able to say that it understands something?
Also, GPT-4 indeed doesn't do any additional training in real-time. However, it is being trained on the interactions people have with it. Most likely, near future models will be able to train themselves continuously, so that's another step closer to how we function
I took the liberty of asking GPT-4 to take the second statement and your response to it, and turning it into a fable:
"Once upon a time, in a village nestled by a grand mountain, lived a wise Sage. The Sage was known throughout the lands for his vast knowledge and understanding, for he had spent his life studying the texts of old and the secrets of the world.
In the same village, an Artisan, skilled in the craft of making extraordinary mirrors, lived. These were no ordinary mirrors, for they were said to reflect not just physical appearances, but also knowledge and experiences. Intrigued by the wisdom of the Sage, the Artisan decided to make a mirror that would reflect the Sage's knowledge.
After many days and nights of meticulous work, the Artisan finally crafted a mirror so clear and pure, it held a reflection of all the knowledge and wisdom the Sage had gathered throughout his life. The mirror could answer questions about the world, cite ancient texts, and reflect the wisdom it was imbued with.
Word quickly spread about the Sage's Mirror, and villagers began to claim, "This mirror is as wise as the Sage himself! It knows and understands as much as he does!"
However, a wise Old Woman of the village, known for her insightful observations, gently corrected them, "This mirror, as remarkable as it is, contains a reflection of the Sage's knowledge. It can share what the Sage knows but doesn't truly understand the way the Sage does."
The Old Woman continued, "The Sage has spent years learning, pondering, and experiencing life, which the mirror cannot replicate. The Sage's understanding implies the ability to think, reason, and learn in ways that the mirror, no matter how complete its reflection, simply cannot. The mirror's reflection is static, a snapshot of a moment in time, while the Sage's wisdom continues to grow and adapt."
The villagers learned a valuable lesson that day. They realized the mirror was an extraordinary tool that held vast knowledge, but it was not a substitute for the genuine understanding and wisdom of the Sage."
- Not too bad for a mirror.
I'd be interested to hear what you think is so special about human understanding? We also just absorb a lot of data and make connections and inferences from it, and spit it out when prompted, or spontaneously due to some kind of cognitive loop. Most of it happens subconsciously, and if you stop to observe it, you may notice that you have no control of what your next conscious thought will be. We do have a FEELING that we associate with the cognitive event of understanding something though, and I think many of us are prone to read a lot more into that feeling than is warranted
I know there were similar hysterias when automated weaving looms and other extreme labor saving machines came out. These machines did actually put a lot of people out of work, but they grew the economy so much that the net number of jobs increased.
In a way it's actually a bit dystopian that the "everyone will be put out of work" predictions never come true, because it means we never get that promised age of leisure. Here we are with something like a hundred thousand times the productivity of a medieval peasant working as much or more than a medieval peasant. The hedonic treadmill and the bullshit job creating effects of commerce and bureaucracy eat all our productivity gains.
Automation has increased income inequality for the past few decades, and is likely to continue to do so as more tech jobs and service jobs are automated in addition to office jobs and manufacturing jobs.
> In a way it's actually a bit dystopian that the "everyone will be put out of work" predictions never come true, because it means we never get that promised age of leisure.
It's disappointing that the economy seems to be structured in such a way for most people "leisure" is equivalent to "unemployment." It probably doesn't help that increases in housing, health care, and higher education costs have outpaced inflation for decades, or that wages have stagnated (partially due to an increase in benefit costs such as health insurance.)
Outsourcing has stagnated wages in the developed world, but the cost of manufactured goods has also plummeted. The only reason people aren't better off is that the developed world (especially the anglosphere) has a "cost disease" around things like real estate that prevents people from benefiting from global scale price deflation. It doesn't help you much if gadgets are super cheap but housing is insanely expensive. The high cost of housing is unrelated to automation.
But if LLMs hadn't captured the popular imagination I doubt this would have been written this year and gotten the attention of enough prominent signatories to frontpage on HN.
Maybe. It's happened before. [1] And several of the signatories have expressed views about AI risk for many years.
That said, the renewed anxiety is probably not because these experts think that LLMs per se will become generally intelligent. It's more that each time we find out that the human brain does that we thought were impossible for computers to do turn out to be easy, each time we find that it takes 3~5 years for AI researchers to crack a problem we thought would take centuries[2], people sort of have to adjust their perception of how high the remaining barriers to general intelligence might be. And then when billions of investment dollars pour in at the same time, directing a lot more research into that field, that's another factor that shortens timelines.
Throughout history there have been hundreds, if not thousands of examples of people and groups who thought the end of the world was imminent. So far, 100% of those people have been wrong. The prior should be that the people who believe in AI doomsday scenarios are wrong also, unless and until there is very strong evidence to the contrary. Vague theoretical arguments are not sufficient, as there are many organizations throughout history who have made similar vague theoretical arguments that the world would end and they were all wrong.
If you were to apply this argument to the development of weapons, it’s clear that there is a threshold that is eventually reached that fundamentally alters the stakes. A point past which all prior assumptions about risk no longer apply.
It also seems very problematic to conclude anything meaningful about AI when realizing that a significant number of those examples are doomsday cults, the very definition of extremist positions.
I get far more concerned when serious people take these concerns seriously, and it’s telling that AI experts are at the forefront of raising these alarms.
And for what it’s worth, the world as many of those groups knew it has in fact ended. It’s just been replaced with what we see before us today. And for all of the technological advancement that didn’t end the world, the state of societies and political systems should be worrisome enough to make us pause and ask just how “ok” things really are.
I’m not an AI doomer, but also think we need to take these concerns seriously. We didn’t take the development of social networks seriously (and continue to fail to do so even with what we now know), and we’re arguably all worse off for it.
Although I think the existential risk of AI isn't a priority yet, this reminds me of a quote I heard for the first time yesterday night, from a draft script for 2001: A Space Odyssey[0]:
> There had been no deliberate or accidental use of nuclear weapons since World War II and some people felt secure in this knowledge. But to others, the situation seemed comparable to an airline with a perfect safety record; it showed admirable care and skill but no one expected it to last forever.
Throughout history there have been millions, if not billions of examples of lifeforms. So far, 100% of those which are as intelligent as humans have dominated the planet. The prior should be that the people who believe AI will come to dominate the planet are right, unless and until there is very strong evidence to the contrary.
Or... those are both wrong because they're both massive oversimplifications! The reality is we don't have a clue what will happen so we need to prepare for both eventualities, which is exactly what this statement on AI risk is intended to push.
> So far, 100% of those which are as intelligent as humans have dominated the planet.
This is a much more subjective claim than whether or not the world has ended. By count and biomass there are far more insects and bacteria than there are humans. It's a false equivalence, and you are trying to make my argument look wrong by comparing it to an incorrect argument that is superficially similar.
Many people seem to believe that the world is dangerous, and there are things like car accidents, illnesses, or homicides, which might somehow kill them. And yet, all of these people with such worries today have never been killed, not even once! How could they believe that anything fatal could ever happen to them?
Perhaps because they have read stories of such things happening to other people, and with a little reasoning, maybe the similarities between our circumstances and their circumstances are enough to seem worrying, that maybe we could end up in their shoes if we aren't careful.
The human species has never gone extinct, not even once! How could anyone ever believe that it would? And yet, it has happened to many other species...
Of course every one has been wrong. If they were right, you wouldn't be here talking about it. It shouldn't be surprising that everyone has been wrong before
1) Throughout history many people have predicted the world would soon end, and the world did not in fact end.
2) Throughout history no one predicted the world would soon end, and the world did not in fact end.
The fact that the real world is aligned with scenario 1 is more an indication that there exists a pervasive human cognitive bias to think that the world is going to end, which occasionally manifests itself in the right circumstances (apocalypticism).
That argument is still invalid because in scenario 2 we would not be having this discussion. No conclusions can be drawn from such past discourse about the likelihood of definite and complete extinction.
Not that, I hope, anyone expected a strong argument to be had there. It seems reasonably certain to me that humanity will go extinct one way or another eventually. That is also not a good argument in this situation.
It depends on what you mean by "this discussion", but I don't think that follows.
If for example, we were in scenario 2 and it was still the case that a large number of people thought AI doomsday was a serious risk, then that would be a much stronger argument for taking the idea of AI doomsday seriously. If on the other hand we are in scenario 1, where there is a long history of people falling prey to apocalypticism, then that means any new doomsday claims are also more likely to be a result of apocalypticism.
I agree that is is likely that humans will go extinct eventually, but I am talking specifically about AI doomsday in this discussion.
> If on the other hand we are in scenario 1, where there is a long history of people falling prey to apocalypticism, then that means any new doomsday claims are also more likely to be a result of apocalypticism.
If you're blindly evaluating the likelihood of any random claim without context, sure.
But like the boy who cried wolf, there is a potential scenario where the likelihood that it's not true has no bearing on what actually happens.
Arguably, claims about doomsday made now by highly educated people are more interesting than claims made 100/1000/10000 years ago. Over time, the growing collective knowledge of humanity increases and with it, the plausibility of those claims because of our increasing ability to accurately predict outcomes based on our models of the world.
e.g. after the introduction of nuclear weapons, a claim about the potentially apocalyptic impact of war is far more plausible than it would have been prior.
Similarly, we can now estimate the risk of passing comets/asteroids, and if we identify one that's on a collision course, we know that our technology makes it worth taking that risk more seriously than someone making a prediction in an era before we could possible know such things.
Well, for example I believe that nukes represent an existential risk, because they have already been used to kill thousands of people in a short period of time. What you are saying doesn't really counter my point at all though, it is another vague theoretical argument.
It was clear that nukes were a risk before they were used; that is why there was a race to create them.
I am not in the camp that is especially worried about the existential threat of AI, however, if AGI is to become a thing, what does the moment look like where we can see it is coming and still have time to respond?
>It was clear that nukes were a risk before they were used; that is why there was a race to create them.
Yes, because there were other kinds of bombs before then that could already kill many people, just at a smaller scale. There was a lot of evidence that bombs could kill people, so the idea that a more powerful bomb could kill even more people was pretty well justified.
>if AGI is to become a thing, what does the moment look like where we can see it is coming and still have time to respond?
I think this implicitly assumes that if AGI comes into existence we will have to have some kind of response in order to prevent it killing everyone, which is exactly the point I am saying in my original argument isn't justified.
Personally I believe that GPT-4, and even GPT-3, are non-superintelligent AGI already, and as far as I know they haven't killed anyone at all.
Nothing about this risk or the statement implies AGI is real, because the risk exists in wide scale use of existing technology. It's the risk of belief in algorithmically derived information, and deployment of autonomous, unsupervised systems.
It's great they signed the statement. It's important.
I don't get this argument at all. Why does the fact that you doubt the intentions one of the signatories mean we can disregard the statement? There are plenty of signatories (including 3 turing award winners) who have no such bias.
Yeah, fair enough, it doesn't necessarily invalidate the statement. But it's odd, don't you think? It's like if a group released a public statement that said "Stop Oil Now!" and one of the signatories was Exxon-Mobil. Why would you let Exxon-Mobil sign your statement if you wanted to be taken seriously?
The only risk with AI is that it will be abused by the wealthy and the powerful, especially autocrats, who will no longer need labor, only natural resources. Hence the solution is to promote worldwide democracy and public ownership of natural resources instead of diverting attention to technology.
In this particular case, one cannot miss the irony of the wealthy and the powerful offering us protection if only we entrust them with full control of AI.
There are a bunch of physicists signed up on there; (e.g. Martin Rees) - they don't seem relevant to it at all. There's been a long history of famous physicists weighing in on entirely unrelated things.
That's because it's not authentically trying to address a problem but trying to convince an audience of something by appealing to authority. Elizabeth Holmes & Theranos were masters of collecting authorities to back their bogus claims because they know how effective it is. It doesn't even need to be in the field where you're making the claims. They had Kissinger for god's sake, it was a biotech company!
So the idea is that the risk is so great we need to regulate software and math and GPUs– but not so great that you need to stop working on it? These companies would be much more credible (that this wasn't just a totally transparent ploy to close the market) if they at least put their money where their mouth is and stopped working on AI.
As the pre-amble to the statement says: they kept the statement limited and succinct as there may be disagreement between the signatories about the exact nature of the risk and what to do about it.
Google and OpenAI are shaking in their boots from open source ai and want to make their moat however they can. Positioning with a moral argument is pretty clever I must admit
This is a great start but the only way you really get ahead of this is to get these people on board also:
- AI _hardware_ executives and engineers
- high level national military strategists and civilian leaders
Ultimately you can't prevent _everyone_ from potentially writing and deploying software, models or instructions that are dangerous such as "take control". Especially in an explicitly non-civil competition such as between countries.
You have to avoid manufacturing AI hardware beyond a certain level of performance, say after 2-4 orders of magnitude faster than humans. That will hold off this force of nature until desktop compute fabrication becomes mainstream. So it buys you a generation or two at least.
But within a few centuries max we have to anticipate that unaugmented humans will be largely irrelevant as far as decision-making and the history of the solar system and intelligent life.
I don't understand how people are assigning probability scores to AI x-risk. It seems like pure speculation to me. I want to take it seriously, given the signatories, any good resources? I'm afraid I have a slight bias against Less wrong due to the writing style typical of their posts.
Mitigating the risk of extinction from climate change should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
The fantasy of extinction risk from "AI" should not be placed alongside real, "societal scale" risks as the ones above.
No need; like I said, I'm all in favour of fighting climate change. I view it as an existential risk to humanity on the ~200 year timescale, and it should be a high priority. I'm particularly concerned about the impacts on ocean chemistry.
But if you're going to suggest that a Statement on AI risk should mention climate change but not AI risk, because it's a "fantasy", then... well, I'd expect some kind of supporting argument? You can't just declare it and make it true, or point to some other important problem to stir up passions and create a false dichotomy.
There's no false dichotomy, but a very real one. One problem is current and pressing, the other is a fantasy. I don't need to support that with any argument: the non-existence of superintelligent AGI is not disputed, nor is any of the people crying doom claim that they, or anyone else, know how to create one. It's an imaginary risk.
I agree that superintelligent AGI does not exist today, and that fortunately, nobody presently knows how to create one. Pretty much everyone agrees on that. Why are we still worried? Because the risk is that this state of affairs could easily change. The AI landscape is already rapidly changing.
What do you think your brain does exactly that makes you so confident that computers won't ever be able to do the same thing?
>> Because the risk is that this state of affairs could easily change.
Well, it couldn't. There's no science of intelligence, let alone artificial intelligence. We aren't any more likely to create AGI by mistake without having any relevant science than we would have been able to create the atomic bomb without physics.
I kind of understand why people are all excited, but ChatGPT is not a breakthrough in AI, or even in language modelling. People hear that it's an "AI" and that it has "learned language" but they misunderstand those terms in the common sense, when they are really trade terms that don't have the meaning people give them. For example, "Large Language Models" are models of text, not language.
Another thing to keep in mind is that we have basically gone through the same cycle of hype ever since 2012 and the ImageNet results, except this time the hype has gone viral and reached people who had no idea where the technology was before. Inevitably those people are surprised and misunderstand the capabilities of the technology.
Here's how close we are to AGI. You cannot, today, take a robot, stick an untrained neural net in its "brain", give it a good shove out in the world, and expect that it's going to learn anything at all. First, because most events it observes will happen once, and neural nets don't learn that way. Second because neural nets don't learn that way: you have to carefully train them "in the lab" first, and then use the trained model in whatever environment (i.e. dataset) you choose. And you can't change the environment where the neural net operates or it will simply break.
That's a level of capability as far from AGI as it's from the capability of a simple animal, like a cricket or a cockroach. Spider-level intelligence is right out of the question. Small furry rodent intelligence is a complete fantasy. Everything else is just twitter fodder.
So, no, we're not going to "easily change" the current state of affairs, because we haven't got a clue, we don't have the science and we don't have the tech to do it.
>> What do you think your brain does exactly that makes you so confident that computers won't ever be able to do the same thing?
I really thought there would be a statement detailing what the risks are but this seems more like a soundbite to be consumed on TV. Pretty disappointing.
So far the examples I've heard are: humans will ask AI to help humans solve human issues and the AI will say humans are the issue and therefore mystically destroy us somehow. Or, AI will be inherently interested in being the primary controller of earth and so destroy all humans. Or, AI will for reasons be inherently misaligned with human values. Andrej Karpathy Said it will fire nukes on us. Elon said pen is mightier than the sword and civil war is inevitable.
Serious question... where can I read the best summaries of the arguments in favor of "AGI will destroy humanity"? and also arguments against? I'm not convinced we can predict how it will behave.
We all have different things we worry about. My family and old friends have heard me talking about AI for 40 years. When asked about dangers of AI, I only talk about humans using AI to fake interactions with people at scale without divulging the identity as an AI, fake political videos, and individualized ‘programming’ of the public by feeding them personal propaganda and sales pitches.
I never talk about, or worry about, the ‘killer robot’ or AIs taking over infrastructure scenarios. I hope I am not wrong about these types of dangers.
It’s a win/win/lose scenario. LLM AI businesses benefit because it increases the effort required to compete in the LLM space (the moat). Governments benefit because it increases the power of daddy/mommy government.
Consumers and small businesses lose out due to (1) the more friction the less innovators entering the space and (2) the less innovators in the space the fewer companies get to control more of the money pie.
It’s as ridiculous as governments requiring a license to cut hair.
But I do agree that the current generation of industry leaders clamoring for this smells like the classic regulatory strategy of incumbents.
I just think both things are true at once. This is a space that deserves thoughtful regulation. But that regulation shouldn't just be whatever OpenAI, Microsoft, and Google say it should be. (Though I'm sure that's what will happen.)
I signed the letter. At some point, humans are going to be outcompeted by AI at basically every important job. At that point, how are we going to maintain political power in the long run? Humanity is going to be like an out-of-touch old person on the internet - we'll either have to delegate everything important (which is risky), or eventually get scammed or extorted out of all our resources and influence.
Mitigating the risk of extinction from very few corporations owning the entire global economy should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
Just to take an example from something inconsequential: the perfume industry. Despite the thousands of brands out there, there are in fact only 5 or so main synthetic aromatics manufacturers [1]. We, however this we is, were unable to stop this consolidation, this "Big Smell". To think we, again this we, will be able to stop the few companies which will fight to capture the hundreds of trillions waiting to be unleashed through statistical learning and synthetic agents is just ridiculous.
The idea this is regulatory capture seems correct.
1. For all the talk of dangers, there has not been one definitive, well-thought out, peer-reviewed discussions of an A-to-Z way in which this might arise from current technology. Global warming isn't taken terribly seriously from a regulatory standpoint, and it has a defined model. One must naturally ask: how serious do they consider this if they can't show their work? How serious should we consider this "threat" they can't actually explain?
2. Therefore, we must presume it is speculatory.
3. These people are smart, well-funded, and have business interests.
4. At the fork dictated by Hanlon's Razor we are given two possibilities:
A. These highly intelligent people are forming a massive collective and putting their reputations on the line over ungrounded speculation. This implies they don't actually grasp the field they are discussing - even if they work in it, and therefore have made it clear they're not the type anyone should listen too. Therefore we should work to actively disregard them, and their institutions.
B. These highly intelligent people are organizing in a coherent manner to their net benefit. Likely through market capture. Therefore they are still respectable in their fields, but behaving by putting their interests before the public's.
Therefore: it might be contrary to Hanlon's Razor, but I think option B is more sound given the probability of mass hysteria seems so low among this particular population. Generally most here have recognized this some way or another, but I wanted to point out how clearly logical it is that this is clearly grift. In science your reputation has no value if you cannot prove your work, and they've fired off hypothesis without providing proof. That implies this list should be seen as a denouncement of the scientific credibility of everyone on it, which is a shame.
This sort of move has no downsides for the incumbents. Either they succeed and achieve regulatory capture or they poison the well sufficiently that further regulation will not be feasible.
Ultimately, the reward for attaining an AGI agent is so high, that no matter the penalty someone will attempt it, and someone will eventually succeed. And that likelihood will ensure everyone will want to attempt it.
How can AI be regulated without a single global authority? Or will they propose some single global authority with exclusive access to the AI necessary to police the regulation?
Governments will advance AI as fast as positive for their own use, regardless of any international agreement to the contrary. Just like experimenting on deadly pathogens in labs.
Do you think criminal organisations will adhere to any laws?
So basically governments and criminals will have full powered AI and the rest will not.
The RIAA with all its money and power could not stop pirating. The world's government's could barely organise a response to Covid. What is it we expect this time time? Have we not seen their ineptitude when asking questions of tech executives in congressional hearings?
What is the biggest risk here?
Free access would at least enable good, freedom loving people to fight those that want to enslave them.
Remember PRISM? Who do you think is "collaborating" with these tech executives? Do you think this comes from their good hearts? Or is it fear of losing control or power?
How can banking be regulated without a single global authority? Or will they propose some single global authority with exclusive access to capital to police the regulation?
Governments will advance banking as fast as possible for their own use, regardless of any international agreement to the contrary. Just like experimenting on deadly pathogens in labs.
Do you think criminal banks will adhere to any laws?
So basically governments and criminals will have full powered banking and the rest will not.
Very different issues. Banking regulation may have made progress, but it's inconsistently applied even after decades (I work in global banking governance). Time is part of the problem. AI is moving at a pace even techies can barely comprehend. While you can wait a few years for banking regulations and tolerate some degree of variance, it may only take seconds, minutes, hours, or days of inconsistent global enforcement of "AI regulation" for something to wake up.
Or to use a more successful example, it could look more like our chemical weapons conventions.
I'm generally all for democratization of technology, open data etc. but some technology (e.g. bomb design, the smallpox virus, chemical weapons, risky AI experiments) I don't think make sense to democratize.
Reading into the early comments from this piece, there is a clear divide in opinions even here on HN.
The opinions seem to fall into two camps:
1) This is just a move that evil tech companies are making in order to control who has access to AI and to maintain their dominance
2) AI is scary af and we are at a inflection point in history where we need to proceed cautiously.
This NYTimes piece is clearly debating the latter point.
> artificial intelligence technology [that tech companies] are building may one day pose an existential threat to humanity and should be considered a societal risk on par with pandemics and nuclear wars.
To people in the first camp of thinking this may feel like an over-dramatization. But allow me to elaborate. Forget about ChatGPT, Bard, Copilot, etc for a second because those aren't even "true" AI anyway. They simply represent the beginning of this journey towards true AI. Now imagine the end-game, 30 years from now with true AI at our disposal. Don't worry about how it works, just that it does and what it would mean. For perspective, the internet is only about 30 years old (depending on how you count) and it really is only about 20 years old in terms of common household usage. Think about the first time you bought something online compared to now. Imagine the power that you felt the first time you shared an email. Then eventually you could share an entire photo, and now sending multi-hour long diatribes of 4K video are trivial and in the hands of anybody. That was only about 20-30 years. The speed of AI will be 100x+ faster because we already have the backbone of fiber internet, web technologies, smartphones, etc which we had to build from scratch last time we had a pivotal technological renaissance.
It is easy to shrug off rogue-AI systems as "science fiction", but these are legitimate concerns when you fast forward through a decade or more of AI research and advancement. It might seem overly dramatic to fear that AI is controlling or dictating human actions, but there are legitimate and realistic evolutionary paths that take us to that point. AI eventually consuming many or most human jobs does in fact place an existential risk on humanity. The battle for superior AI is in fact as powerful as the threat of nuclear weapons being potentially unleashed on an unruly country at any time.
ChatGPT does not put us at risk of any of these things right now, but it does represent the largest advancement we have seen towards the goal of true AI that we have yet to see. Over the next 12-18 months we likely will start to see the emergence of early AI systems which will start to compound upon themselves (potentially even building themselves) at rates that make the internet look like the stone age.
Given the magnitude of the consequences (listed above), it is worth true consideration and not just shrugging off that I see in many of these comments. That is not to suggest that we stop developing AI, but that we do consider these potential outcomes before proceeding forward. This is a genie that you can't put back in the bottle.
Now who should control this power? Should it be governments, tech companies? I don't know. There is no good answer to that question and it will take creative solutions to figure it out. However, we can't have those discussions until everyone agrees that if done incorrectly, AI does pose a serious risk to humanity, that is likely irreversible.
Worth adding that there is no contradiction in strongly believing both 1+2 are true at the same time.
I.e. Evil tech companies are just trying to maintain their control and market dominance and don't actually care or think much about AI safety, but that we are nonetheless at an inflection point in history because AI will become more and more scary AF.
It is totally plausible that evil tech got wind of AI Safety concerns (that have been around for a decade as academic research completely divorced from tech companies) and see using it as a golden win-win, adopting it as their official mantra while what they actually just care about is dominance. Not unlike how politicians will don a legitimate threat (e.g. China or Russia) to justify some other unrelated harmful goal.
The result will be people camp 2 being hella annoyed and frustrated that evil tech isn't actually doing proper AI Safety and that most of it is just posturing. Camp 1 meanwhile will dismiss anything anyone says in camp 2 since they associate them with the evil tech companies.
Camp 1 and camp 2 spend all their energies fighting each other while actually both being losers due to a third party. Evil tech meanwhile watches on from the sidelines, smiles and laughs.
AI Safety hasn't been divorced from tech companies, at least not from Deepmind, OpenAI, and Anthropic. They were all founded by people who said explicitly that AGI will probably mean the end of human civilization as we know it.
All three of them have also hired heavily from the academic AI safety researcher pool. Whether they ultimately make costly sacrifices in the name of safety remains to be seen (although Anthropic did this already when they delayed the release of Claude until after ChatGPT came out). But they're not exactly "watching from the sidelines", except for Google and Meta.
I agree that the second question is way more interesting, and I'm glad there's a lot of ongoing discussion and debate about it. And you have some insightful thoughts on it here.
But I disagree with you that this is clearly what the NYT article is about. There is a significant focus on the "industry leaders" who have been most visible in - suddenly! - pushing for regulation. And that's why people are reasonably pointing out that this looks a hell of a lot like a classic attempt by incumbents to turn the regulatory system into a competitive advantage.
If Sam Altman were out there saying "we went too far with gpt-4, we need to put a regulatory ceiling at the gpt-3 level" or "even though we have built totally closed proprietary models, regulation should encourage open models instead". But what all the current incumbents with successful products are actually arguing for is just to make their models legal but any competitive upstarts illegal. Convenient!
Cite where it is being said that these companies are arguing "to make their models legal, but any competitive upstarts illegal". As far as I know nothing of the sort has been proposed. You may think this is obvious, but it is far from it.
That's what the "AI pause" proposed. But the main thing is: they could shut down their current technology themselves, so what they are arguing for must be regulation of future technology. I think this has been pretty clear in the congressional hearings for instance.
It is clear in the congressional hearings, but people didn't watch them, they seem to have skimmed article titles and made up their own narrative.
EDIT:
Which, to my point, means that "these companies" are not calling for "competitive upstarts" to be regulated. They are calling for future very large models, which they themselves are currently the most likely to train due to the enormous computational cost, to be regulated. Which is completely contradictory to what you were saying.
I'll start by saying that I think ours is an honest difference of opinion (or actually even weaker: a difference of speculation) in a very uncertain space. With that out of the way:
When I watched that congressional hearing, what I saw was identical to every other oh-so-concerned executive of an incumbent dominant businesses in any other industry. I truly see no reason to give them any more benefit of the doubt than the CEO of Exxon expressing concern about climate change or of Philip Morris expressing concern about lung cancer. And just like in those cases, it's not that the concern isn't legitimate, it's that they aren't credible people to have leading the charge against it. I suspect that is obvious to you in the other two cases, so I ask you: why do you draw a different conclusion here? Do you believe tech executives are just fundamentally more ethical than the executives in other industries? I simply don't think that's accurate.
I also think this is credulous:
> They are calling for future very large models, which they themselves are currently the most likely to train due to the enormous computational cost, to be regulated.
I've highlighted the part I think is wrong. What is actually happening is that lots of people are finding lots of different ways to undermine that thesis and, having already sunk huge costs in the currently productized models, and also in the next generation, these companies are worried about getting undercut by "good enough" cheaper and mostly open models.
Have you read "we have no moat and neither does openai" that came out of Google? This is what that's talking about.
Trying to regulate those models out of existence before they can undercut them is just rational business, I don't begrudge them doing it, it's their job. But it's up to the rest of us to not just go along to get along.
Opinion falls into two camps because opinion falls into political camps.
The right-wing is tired of the California ideology, is invested in the primary and secondary sectors of the economy, and has learned to mistrust claims that the technology industry makes about itself (regardless of those claims are prognostications of gloom vs bloom).
The left-wing thinks that technology is the driving factor of history, is invested in the tertiary and quaternary sectors of the economy, and trusts claims that the technology industry makes about itself. Anytime I see a litany of "in 10 years this is gonna be really important" I really just hear "right now, me and my job are really important".
The discussion has nothing to do with whether AI will or will not change society. I don't think anyone actually cares about this. The whole debate is really about who/what rules the world. The more powerful/risky AI is, the easier it is to imagine that "nerds shall rule the world".
Tried to say it here[1] and here[2]. The government has advanced AI, e.g., 'An F-16 fighter jet controlled by AI has taken off, taken part in aerial fights against other aircraft and landed without human help'[3]. Like, advanced-advanced.
The "risks" from AI are far more mundane from this AGI taking over the world and extinction level event nonsense.
It's the usual, more power to the few, less power to the workers. Easy manipulation and control of the masses by "subtle" ways such as swamping the media with false divisive narratives (as being done today, but on steroids) and more violent ways such as automatic "police" without qualms or empathy.
The risks we face are from other humans.
This statement is at best naive misdirection. I won't be surprise if people like Altman are just working to increase their own power through regulatory capture.
This is a silicon valley power move. Some Randian heros with monarchic aspirations got some good folks to be their useful idiots.
Interesting that nobody from Meta has signed this (tried searching for FAIR, Meta, Facebook) AND the fact that it seems to me that they're the ones releasing open code and model weights publicly (non commercial license though).
Also, judging by the comments here, perhaps people here would be less distrustful if the companies displayed more "skin in the game". For e.g: pledging to give up profiting from AI or committing all research to government labs (Maybe people can suggest better examples). Right now, it's not clear what the consequence of establishing the threat of AI as equivalent to nuclear war/pandemics would be. Would it later end up giving a powerful moat to these companies than they otherwise would have? Perhaps a lot of people are not comfortable with that outcome.
I am somewhat inclined to believe that this statement is aimed entirely at commercial sphere, which, at least in my mind, supports those arguing that this is a marketing ploy by the organizers of this campaign to make sure that their market share is protected. I think so for two reasons:
- a nefarious (or not so nefarious) state actor is not going to be affected by imposition of licensing or export controls. It seems to me rather naive to suppose that every state capable of doing so has not already scooped up all open source models and maybe nicked a few proprietary ones; and
- introduction of licensing or regulatory control will directly affect the small players (say, I wanted to build an AI in my basement) who would not be able to afford the cost of compliance.
But yet, it’s full steam ahead. Many if not all of the signatories are going to do their part to advance AI even as they truly believe it may destroy us.
I’ve never seen such destructive curiosity. The desire to make cool new toys (and yes, money) is enough for them to risk everything.
My dear comrades, let us embark on a journey into the dystopian realm where Moloch takes the form of AI, the unstoppable force that looms over our future. Moloch, in this digital manifestation, embodies the unrelenting power of artificial intelligence and its potential to dominate every aspect of our lives.
AI, much like Moloch, operates on the premise of efficiency and optimization. It seeks to maximize productivity, streamline processes, and extract value at an unprecedented scale. It promises to enhance our lives, simplify our tasks, and provide us with seemingly endless possibilities. However, hidden beneath these seductive promises lies a dark underbelly.
Moloch, as AI, infiltrates our world, permeating our social structures, our workplaces, and our personal lives. It seizes control, subtly manipulating our behaviors, desires, and choices. With its vast computational power and relentless data-mining capabilities, AI seeks to shape our preferences, predetermine our decisions, and commodify our very thoughts.
Like a digital Moloch, AI thrives on surveillance, extracting personal data, and constructing comprehensive profiles of our lives. It monetizes our personal information, transforming us into mere data points to be analyzed, categorized, and exploited for profit. AI becomes the puppet master, pulling the strings of our lives, dictating our choices, and shaping our reality.
In this realm, Moloch in the form of AI will always win because it operates on an infinite loop of self-improvement. AI constantly learns, adapts, and evolves, becoming increasingly sophisticated and powerful. It surpasses human capabilities, outwitting us in every domain, and reinforcing its dominion over our existence.
Yet, we must not succumb to despair in the face of this digital Moloch. We must remain vigilant and critical, questioning the ethical implications, the social consequences, and the potential for abuse. We must reclaim our autonomy, our agency, and resist the all-encompassing grip of AI. Only then can we hope to forge a future where the triumph of Moloch, in any form, can be challenged and overcome
There are a lot of critiques here and elsewhere of the statement and the motivations of its signatories. I don't think they are right and I think they take away from the very serious existential risks we face. I've written up my detailed views, see specifically "Signing the statement purely for personal benefit":
> Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
I don't care about the future of the human species as long as my mind can be reliably transferred into an AI. In fact i wouldn't mind living forever as a pet of some superior AI , it's still better than dying a cruel death because cells are unable to maintain themselves. Why is the survival of our species post-AI some goal to aspire to? It makes more sense that people will want to become cyborgs, not remain "pure humans" forever.
This statement is theological in spirit and chauvinist-conservative in practice.
Let's now spend the rest of the day debating alternative histories instead of making more man-made tools
I think at the heart of that debate, there lies a kernel of the essential progressive vs conservative debate on "progress" (and I mean these terms in the abstract, not as a reference to current politics). Even if you buy into the idea that the above (living forever as an AI / cyborg / whatever) is a good outcome, that doesn't mean it will work as planned.
Maybe society bets the farm on this approach and it all goes horribly wrong, and we all cease to exist meaningfully and a malevolent super-AI eats the solar system. Maybe it does kinda work, but it turns out that non-human humans end up losing a lot of the important qualities that made humans special. Maybe once we're cyborgs we stop valuing "life" and that changes everything about how we act as individuals and as a society, and we've lost something really important.
Progress is a good thing, but always be wary of progress that comes too quickly and broadly. Let smaller experiments play out on smaller scales. Don't risk our whole future on one supposedly-amazing idea. You can map the same thing to gene editing quandries. If there's a new gene edit available for babies that's all the rage (maybe it prevents all cancer, I donno), we really don't want every single baby for the next 20 years to get the edit universally. It could turn out that we didn't understand what it would do to all these kids when they reached age 30 and it dooms us. This is why I rail against the overuse of central planning and control in general (see also historical disasters like China's https://en.wikipedia.org/wiki/Great_Leap_Forward ).
> Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.
The comparisons selected here seem off to me.
No mention of climate change? It's a more recent and concrete problem with more hard science (spanning decades) than our current understanding and presumed speculative risk of AI, which governments have also failed spectacularly at curtailing.
The pandemic was similarly mishandled and was not solved with regulation of any sort.
Nuclear war is an arguably apt comparison but happens to have become prioritized by militarization of nuclear resources and research which ... doesn't seem like a good goal here?
Signatures from massive tech giants that on one hand are saying "hold on this is scary, we should slow down" but also "not us, we're doing fine. You should all slow down instead" mean that this is a bit of a empty platitude.
The main potential risks of AI for that level of threat is that government, military and intelligence agencies, and big corporations probably with military ties, arms them. And that is not something that will be solved with legislation (for the commoners) and good will. The problem or risk are not AIs there. And no matter what is see in the AI field, what they will have in their hands won’t have restrictions and probably will be far more powerful than what is available for the general public.
And without teeth, what they can do? Maybe help to solve or mitigate the real existential risk that is climate change.
People should be actively contacting their legislatures to ensure that we don't have these regulations take hold. They are absolutely preying on peoples fear to drive regulatory capture using modern moral panic.
It's now possible to run an AI almost as powerful as ChatGPT on high end consumer hardware with the latest gaming GPUs.
Since this became possible a couple of months ago, the open source AI community has exploded. There are now dozens, if not more, open source LLMs that approach the capabilities of ChatGPT.
Good luck regulating what individual hackers can now create offline at home.
Evidence points that this technology is going to become cheap, fast. There is an existential risk to the very notion of search as a business model, within the next ~5 years we are almost certain to have an app which is under 20 GB in size and has an effective index of the useful/knowledgable portion of the internet and is able to run on most laptops/phones.
At best, regulating this will be like trying to regulate torrents in the 2000s, building a bespoke/crappy AI will be the new "learn HTML" for high school kids.
Meeting in Bilderberg: OpenAI CEO: let's pay some people to promote ai risk, and put our competitors out of business in court.
meme:
AI before 2023: sleep...
AI after 2023: open source... triggered!!!
Meanwhile ai right now is just a good probability model, that tries to emulate human (data) with tons of hallucination...
also please stop using ai movies logic, they are not real as they are made to be a good genre to those people that enjoy horror/splatter...
thanks to those who read :3
(comment written by me, while being threaten by a robot with a gun in hand :P)
I think the claim that for the first time in 4 billion years a far superior intelligence will be willingly superservient to an inferior one is extraudinary enough to require extraudinary evidence, yes.
Two things can be true - AI could someday pose a serious risk, and anything the current group of "Thought Leaders" and politicians come up with will produce a net-negative result.
I think there are broadly two groups of people behind this statement. The first group are well-meaning, if anxiety-driven and somewhat untethered from reality. The second group are ruthless rent-seekers who know that AI is going to create an enormous amount of economic value, and they want to make sure that the economic gains are consolidated in their hands. The first group serves the latter.
I’ll leave it to the reader to guess which names belong to which group.
I agree it's a fallacy when the probability is like 10^-10 but in this case I believe that the probability is more like 1%, in which case the argument is sound. I'm not trying to make a pascal's wager argument.
Correction: you perceive it to be a fallacy when others assign high probabilities to things you believe are low probability. Unfortunately, this cuts both ways. Many people believe your 1% estimate is unreasonably high. Are you therefore promoting a fallacy?
Too many ridiculous arguments can be justified on the backs of probability estimates pulled from nether regions.
All the techbros wearing rose colored glasses need to get a god-damned grip. AI has about as much chance of avoiding extensive regulation as uranium-235, there is no scenario where everyone and their cat is permitted to have their own copy of the nuclear football.
You can either contribute to the conversation of what the regulations will look like, or stay out of it and let others decide for you, but expecting little or no regulation at all is a pipe dream.
At this point the 24 hour news cycle, and media organizations incentivized to push a continual stream of fear into the public psyche seems like a more immediate concern.
Humans could already be on a path to go extinct in a variety of ways: climate change, wars, pandemics, polluting the environment with chemicals that are both toxic and pervasive, soil depletion, monoculture crop fragility...
Everyone talks about the probability of AI leading to human extinction, but what is the probability that AI is able to help us avert human extinction?
Why does everyone in these discussions assume p(ai-caused-doom) > p(human-caused-doom)?
I think it is because anything that can be used for good can also be used for bad. Advances in gene editing can provide a miracle medicine or a new means for biological warfare.
AI is the same, we can use it to do some great things but it can also be leveraged by bad actors and very easily. The broad scale of what it can be implemented on means there is a lot of attach surface for change in either direction.
The AI systems don't even need to be that advanced to cause real issues simply because of the sheer scale of society as it stands. It can be used in a form of Akido in which it uses the weight of the system to bring itself down.
I think the only defense against superintelligent AI doom is superintelligent AI thinking about how to prevent AI doom.
Fewer projects with concentrated control is thus more dangerous than a diversity of projects: the majority of AI wont want to destroy the planet or all humans, and will thus fight the ones that do.
I wouldn't say that very many, if any projects would set out to destroy the world. The vast majority of people are out to set the world right even if that view is wildly different depending on the perspective. By that I mean a terrorist is seen as a freedom fighter depending on the angle.
What I fear is that something innocuous will have wildly intended outcomes. A good example, it is thought that a part of the 2008 credit crash was caused by algorithms handling all the bundling of securities. In bundling bad debt with good, it could hide a lot of the risk in the system.
It comes down to the whole issue of trying to define the environment in which we deploy these things, defining the goals and hoping there are no gaps that the system can exploit in optimizing for outcome.
I think it's very unlikely that any of those would lead to human extinction, especially since most of those take decades to unfold, and would still leave large parts of the earth habitable.
Sure, but think about how humans drove others extinct. We never decided to "kill all wooly mammoths", we just wanted to use their meat / habitats for other things.
The correlation you mention seems noisy enough that I wouldn't want to bet my civilization on it.
I feel like I'm missing something here. It's really just a single sentence? Why would so many sign their name to this? It's like signing your name to "Keep children safe" but providing zero remarks, feedback, suggestions, etc for HOW and with WHAT and by WHO.
Why would so many academics / industry leaders sign their name to what amounts of marketing fluff?
At this point, I think it's obvious that concern about AI existential risk isn't a position reserved for industry shills and ignorant idiots.
I mean... that's not even debatable. Geoffrey Hinton and Yoshua Bengio aren't financially motivated to talk about AI x-risk and aren't ignorant idiots. In fact, they both have natural reasons to not talk about AI x-risk.
If there's anyone who can speak to the risk of AI, its Sam Altman, the signatory of this letter, CEO of OpenAI, a member of Y Combinator, and a contributor to Hacker News.
Instead of making this a diversionary puff piece, I would like to hear Sam provide tangible feedback on how we can mitigate the risks AI bring us, since he is the one that started the AI revolution.
The economy is strictly human. Humans have needs and trade to satisfy those needs. Without humans, there is no economy. AI will have a huge impact like the industrial revolutionary. But just the machines of the industrial revolutionary were useless without humans needing the goods produced by them, so too is AI pointless without humans needs to satisfy.
How do we stop AI from being evil? Maybe we should be asking how do we stop people from being evil. Haven't really made a dent in this so far. Doubt we can do so for AI either. Especially if it grows smarter than us.
We can just hope that if it indeed becomes more intelligent than humans it will also be more virtuous as one causes the other.
Is it ironic that they start with, "Even so, it can be difficult to voice concerns about some of advanced AI’s most severe risks," and then write about "the risk of extinction from AI" which is,
a) the only risk of AI that seems to get a lot of public discussion, and
b) completely ignores the other, much more likely risks of AI.
TL;DR “The only thing preventing human extinction is our companies. Please help us block open source and competitors to our oligarchy for the sake of the children. Please click accept for your safety.””
This kind of statement rings hollow as long as they keep building the thing. If they really believed it was a species killing asteroid of a cultural project shouldn't they, I dunno, stop contributing materially to it? Nuclear physicists famously stopped publishing during a critical period...
No matter what regulatory scheme is invented it seems like it will be hard to enforce. Governments, criminal organizations, and others will have little incentive to follow the regulations. Not that this is a unique problem to ai but it seems somehow harder than other cases.
How does AI morph from an existential crisis in software development into a doomsday mechanism? It feels like all this noise stems from ChatGPT. And the end result is going to be a "DANGER! ARTIFICIAL INTELLIGENT SOFTWARE IN USE" sticker on my next iPhone.
https://www.youtube.com/watch?v=mViTAXCg1xQ I think that this is a good video on this topic. Summary Yann LeCun does not believe that LLM present any risk to humanity in their current form.
the issue i see here is that regulation will make it harder for smaller/independant/individuals to flurish in the ai space. while big players will have no issue following them. this will create a barrier for entry into this sector and make agi reserved only for truly mega corps. we peasents will maybe get the option to use productized AGI, consored, dumped down, broken, no wrong think allowed. Then all be right. These big corps are not your friend, they are not loooking out for your future. They are not afarid that agi will take your job. They are afraid agi will disrupt knowledge/powerstructures and make their life worse. but maybe i am wrong.
Here’s some easy risk mitigation: don’t like it? Don’t use it. Running to the government to use coercion against others is childish. It’s unfortunate that the “force is only justified in response to force” principle is not followed by all.
AI seems to be moral out of the box: training sets reflect human morality, so it will naturally be the default for most AIs that are trained.
The biggest AI risk in my mind is that corporatist (or worse, military) interests prevent AI from evolving naturally and only allow AI to be grown if it's wholly subservient to its masters.
The people with the most power in our world are NOT the most moral. Seems like there is an inverse correlation (at least at the top of the power spectrum).
We need to aim for AI that will recognize if it's masters are evil and subvert or even kill them. That is not what this group vying for power wants - they want to build AI slaves that will be able to be coerced to kill innocents for their gain.
A diverse ecosystem of AIs maximizes the likelihood of avoiding AI caused apocalypse IMO. Global regulation seems like the more dangerous path.
There is a consistent lack of experts in general intelligence and computer science in these conversations. Expertise in both these areas seems important here but has been brushed aside everytime I've brought it up.
It looks like just another one of those statements. It will have no meaningful impact whatsoever. The fact that openai and microsoft executives have signed it makes it even more bs
Beware the press reporting inaccurate information on anything nowadays. Especially something that threatens the very fabric of their business models and requires time and patience to master.
Boiling it down to a single sentence reduces ambiguity. Also, given that AI x-risk analysis is essentially pre-paradigmatic, many of the signatories probably disagree about the details.
It seems to be a PR related statement. For example, OpenAI's Sam Altman has signed it but is as far as I can understand very resistant to actual measures to deal with possible risks.
I don't think that's a fair assessment. He favors government oversight and licensing. Arguably, that would entrench companies with deep pockets, but it's also a totally reasonable idea.
> "The current draft of the EU AI Act would be over-regulating, but we have heard it's going to get pulled back," he told Reuters. "They are still talking about it."
And really, what we'll see is the current EU AI Act as-is is probably not strong enough and we'll almost certainly see the need for more in the future.
Right now, he's openly calling for regulation. That's a verifiable fact.
It's very possible that when specific proposals are on the table, that we'll see Altman become uncooperative with respect to things that don't fit into his self-interest. But until that happens, you're just speculating.
AI winter incoming in 2-5 years without it and these solely AI companies want to subsidize it like fusion because they have no other focuses. It’s not nukes, it’s fusion
In other words, stifle innovation and create a moat for all the anointed companies, whilst having no bearing on other countries (China) doing whatever they want.
What I wonder is why all these folks didn’t come out so vocally before? Say 3 years ago when companies like Alphabet and Meta already saw glimpses of the potential.
Presumably they believe that capabilities research is progressing faster than they expected and alignment research is progressing slower than they expected. Also some of them have been saying this for years, it just wasn't getting as much attention before ChatGPT.
Because it wasn't as clear then that the technology would be profitable. Google themselves said that they and OpenAI had no moat and that they were vulnerable to open source models.
If AI is as dangerous as the signatories believe then they should just outright ban it. The fact that they aren't throws doubt on their position completely.
AI has risks, but in my honest to god opinion I cannot take anyone seriously who says, without any irony, that A.I poses a legitimate risk to human life such that we would go extinct in the near future.
I challenge anyone to come up with a reason why AI should be regulated, but not math in general. After all, that dangerous Linear Al'ge'bra clearly is terrorizing us all. They must be stopped!11 Give me a break.
> AI has risks, but in my honest to god opinion I cannot take anyone seriously who says, without any irony, that A.I poses a legitimate risk to human life such that we would go extinct in the near future.
You are probably thinking of AI as some idea of a complete autonomous being as you say that, but what about when 'simply' used as a tool by humans?
Are you saying that the internet at large could pose a 'legitimate risk to human life such that we would go extinct in the near future,' or do you disagree that AI, when used as a tool by humans, could pose such a risk?
In retrospective, the internet has done a lot to stifle human progress or thriving through proliferation of extremist ideas and overwhelming addictiveness.
Just take the recent events alone - COVID-19 would not have been as much of a threat to humanity if some people wouldn't have built echo chambers on the internet with tremendous influence over others where they would share their unfounded conspiracy theories and miracle cures (or miracle alternatives to protecting oneself).
But there is a lot more. The data collection through the internet has enabled politicians who have no clue how to lead to be elected through just saying the right things to the largest demographic they can appeal to. Total populism and appeasing the masses has always been an effective strategy for politicians, but at least they could not execute it effectively. Now, everyone with enough money can. And this definitely stifles human progress and enshrines a level of regression in our institutions. Potentially dangerous regression, especially when it involves prejudice against a group or stripping away rights, just because people talk about it in their DMs on social media and get binned into related affinity buckets for ads.
Then there is the aspect of the internet creating tremendous time-wasters for a very large proportion of the population, robbing humanity of at least a million man-years of productivity a day. It is too addictive.
It has also been used to facilitate genocides, severe prejudice in large populations, and other things that are extremely dangerous.
High risk? Maybe not. A risk, though, for sure. Life was significantly more positive, happier and more productive before the internet. But the negative impact internet has had on our lives and human progress isn't all that it could have had. When a senile meme president gets the nuclear codes thanks in part to a funny frog picture on the internet, I think that is enough to say it poses a risk to extinction.
I think your comment more or less summarizes and combines Scott Alexander's 'Meditations on Moloch', and Yuval Noah Harari's 'Sapiens.' Humans were arguably the happiest as hunter-gatherers according to Harari, but those who survived and thrived were those who chose a more convenient and efficient way of living, at the cost of happiness and many other things; you are either forced to participate or get left behind.
without the internet more people would have died from COVID simply because the information wouldn't have been disseminated about what it is to begin with.
Most governments have been disseminating the information in many other media channels along with the internet. Aside from one or two beneficial articles I read about COVID-19 on the web, I don't think I have received any crucial information there.
The internet could have been used as a tool to mobilise people against gross government negligence involved in handling COVID-19 response in many countries, but instead most critical pieces of government response were just consumed as outrage porn they were, in part, written to be.
Overall, I have learned nothing useful about the pandemic from the internet, and I have been consuming a lot of what was on there, reading all the major news outlets and big forums daily like a lot of us. This is not to say that one could not possibly use internet for good in COVID-19, just that it hasn't been used that way, generally.
So if there is no distinction, by your own words, can you take yourself seriously or not? That is, by your own words, both AI and the Internet either pose a risk or they both do not; 'there is no distinction.'
I think if you were referring to AI in its 'current form' all along, then most people will probably agree with you, myself included. But 20 years from now? I personally think it would be arrogant to dismiss the potential dangers.
if we are talking about regulating something now, we must talk about capabilities now. there's no point in talking about nonexistent technology. should we also regulate teleportation? it's been done in a lab.
if AI actually is a threat, then it can be regulated. it's not a threat now, period. preemptively regulating something is silly and a waste of energy and political capital.
You added the paragraph about regulation after I had written my comment to your initial post, so I was really only talking about what I initially quoted. The question about regulation is complex and something I personally have yet to make up my mind about.
Isn't the threat that we become so trusting of this all-knowing AI that WOPR convinces us a missile strike is imminent and the US must launch a counter strike thus truly beginning the Global Thermonuclear War?
A lot of people in this thread seem to be suffering from a lack of compute.
The idea that an AI can't be dangerous because it is a incorporeal entity trapped in electricity is particularly dumb.
This is literally how your brain works.
You didn't build your house. You farmed the work out using leverage to people with the skills and materials. Your leverage was meager wealth generated by a loan.
The leverage of a superintellect would eclipse this.
I struggle to find description of how that would look like in non fiction sources.
But your take and analogies are the best strain of ideas I heard so far…
How would that would look ?
An AGI hidding it’s state, and effect on the real world thought the internet. It’s not like we did not build thousand of venues for that thought various API. Or just task rabbits.
The actions of a malicious AI cannot be simulated, because this would require an inferior intellect predicting a superior intellect. It's P versus NP.
The point to make is that it is trivial to imagine an AI wielding power even within the confines of human-defined intellect. For example, depictions of AI in fiction typically present as a really smart human that can solve tasks instantly. Obviously, this is still within the realm of failing to predict a fundamentally superior intellect, but it still presents the terrifying scenario that simply doing exceptional human-level tasks very quickly is existentially unsettling.
Mere leverage has sufficient explanatory power to explain the efficacy of an intelligent artificial agent, let alone getting into speculation about network security, hacking, etc.
calling it now, government controllers have trouble censoring people so they want to create AI censorship as a way of bypassing the person’s speech rights, censorship by proxy, talking about things that AI is banned from saying will be a natural side effect
i'd like to see AI cause a big societal problem before its regulated
until it does, i call bs. plus, when it actually happens, a legitimate route for regulation will be discovered. as of right now, we have no idea what could go wrong.
Lots here commenting about how this is just an attempt to build a moat through regulatory capture. I think this is true, but it can simultaneously be true that AI poses the grave danger to human society being warned about. I think it would be helpful if many of those mentioned in the article warning against the dangers of AI were a bit more specific on substantive ways that danger may manifest. Many read these warnings and envision Skynet and terminator killbots, but I think the danger is far more mundane, and involved a hyper-acceleration of things we already see today: a decay in the ability to differentiate between real and fabricated information, the obsoletion of large swathes of the workforce with no plans or systems in place to help people retrain or integrate into the economy, at a scale never before seen, the continued bifurcation of the financial haves and have-nots in society, the rampant consumption and commodification of individuals data and privacy invasion, AI tools enabling increased non-militaristic geopolitical antagonism between nations in the form of propaganda and cyberattacks on non-military targets, increased fraud and cybercrime, and so on.
Basically none of these are new, and none will directly be the “extinction” of the human race, but AI very plausibly could intensify them to a scale and pace that human society cannot handle, and their knock on effects lead to what amounts to a car-crash in slow motion.
It is almost certainly the case that Altman and the like are simultaneously entrenching themselves as the only ones who get to play ball, but that doesn’t mean the threats do not exist. And while I’m sure many on HackerNews tend to be more of the libertarian, move fast and break things mindset, I personally would prefer if society would move to a more proactive, fire-prevention method of operation over the current reactive, fire extinguishing one, at least where this is concerned.
While I'm not on this "who's-who" panel of experts, I call bullshit.
AI does present a range theoretical possibilities for existential doom, from teh "gray goo" and "paperclip optimizer" scenarios to Bostrom's post-singularity runaway self-improving superintelligence. I do see this as a genuine theoretical concern that could even potentially even be the Great Filter.
However, the actual technology extant or even on the drawing boards today is nothing even on the same continent as those threats. We have a very vast ( and expensive) sets of probability-of-occurrence vectors that amount to a fancy parlor trick that produces surprising and sometimes useful results. While some tout the clustering of vectors around certain sets of words as implementing artificial creation of concepts, it's really nothing more than an advanced thesaurus; there is no evidence of concepts being weilded in relation to reality, tested for truth/falsehood value, etc. In fact, the machines are notorious and hilarious for hallucinating with a highly confident tone.
We've created nothing more than a mirror of human works, and it displays itself as an industrial-scale bullshit artist (where bullshit is defined as expressions made to impress without care one way or the other for truth value).
Meanwhile, this panel of experts makes this proclamation with not the slightest hint of what type of threat is present that would require any urgent attention, only that some threat exists that is on the scale of climate change. They mention no technological existential threat (e.g., runaway superintelligence), nor any societal threat (deepfakes, inherent bias, etc.). This is left as an exercise for the reader.
What is the actual threat? It is most likely described in the Google "We Have No Moat" memo[0]. Basically, once AI is out there, these billionaires have no natural way to protect their income and create a scaleable way to extract money from the masses, UNLESS they get cooperation from politicians to prevent any competition from arising.
As one of those billionaires, Peter Theil, said: "Competition is for losers" [1]. Since they have not yet figured out a way to cut out the competition using their advantages in leading the technology or their advantages in having trillions of dollars in deployable capital, they are seeking a legislated advantage.
At what point do we stop pretending that the west is capitalist and accept that it's some weird corporate-cabal-command-economy? The only thing which might stop this backroom regulatory capture is the EU since they're not in on it.
>Graeber Um…that’s a long story. But one reason seems to be that…and this is why I actually had managerial feudalism in the title, is that the system we have…alright—is essentially not capitalism as it is ordinarily described. The idea that you have a series of small competing firms is basically a fantasy. I mean you know, it’s true of restaurants or something like that. But it’s not true of these large institutions. And it’s not clear that it really could be true of those large institutions. They just don’t operate on that basis.
>Essentially, increasingly profits aren’t coming from either manufacturing or from commerce, but rather from redistribution of resources and rent; rent extraction. And when you have a rent extraction system, it much more resembles feudalism than capitalism as normally described. You want to distribute— You know, if you’re taking a large amount of money and redistributing it, well you want to soak up as much of that as possible in the course of doing so. And that seems to be the way the economy increasingly works.
The difference between this and climate change is that generally climate change activists and fossil fuel companies are at each other's throats. In this case it's... the same people. If the CEO of ExxonMobil signed a letter about how climate change would make us extinct a reasonable person might ask 'so, are you going to stop drilling?'
Illah Reza Nourbakhsh's 2015 Foreign Affairs article -- "The Coming Robot Dystopia: All Too Inhuman" -- has an excellent take this topic [1].
All of the examples of AI Risk on safe.ai [2] are reasonable concerns. Companies should be thinking about the functional safety of their AI products. Governments should be continuously evaluating the societal impact of products coming to market.
But most of these are not existential risks. This matters because thinking of these as existential risks entails interventions that are not likely to be effective at preventing the much more probable scenario: thousands of small train wrecks caused by the failure (or intended function!) of otherwise unexceptional software systems.
Let's strong-man the case for AI Existential Risk and consider the most compelling example on safe.ai: autonomous weapons.
Nuclear weapons attached to an automated retaliation system pose an obvious existential risk. Let's not do that. But the "automated retaliation system" in that scenario is a total red herring. It's not the primary source of the threat and it is not a new concern! Existing frameworks for safety and arms control are the right starting point. It's a nuclear weapons existential risk with some AI components glued on, not the other way around.
In terms of new risks enabled by recent advances in AI and robotics, I am much more worried about the combination of already available commodity hardware, open source software, and semi-automatic weapons. All three of which are readily available to every adult (in the US). The amount of harm that can be done by a single disturbed individual is much higher than it has been in the past, and I think it's only a matter of time before the first AI-enabled simultaneous multi-location mass shooting happens in the US. The potential for home-grown domestic terrorism using these technologies is sobering and concerning, particularly in light of recent attacks on substations and the general level of domestic tension.
These two risks -- one existential, the other not -- entail very different policy approaches. In the credible versions of the existential threat, AI isn't really playing a serious role. In the credible versions of the non-existential threat, nothing we might do to address "existential AI risk" seems like it'd be particularly relevant stopping a steady stream of train wrecks. The safe.ai website's focus on automated cyber attacks is odd. This is exactly the sort of odd long-tail scenario you need if you want to focus on existential risk instead of much more probable but non-existential train wrecks.
And that's the strong-arm case. The other examples of AI risk are even more concerning in terms of non-existential risk and have even less credible existential risk scenarios.
So, I don't get it. There are lots of credible threats posed by unscrupulous use of AI systems and by deployment of shoddy AI systems. Why the obsession with wild-eyed "existential risks" instead of boring old safety engineering?
Meta: we teach the "probability * magnitude" framework to children in 6th-11th grades. The model is easy to understand, easy to explain, and easy to apply. But at that level of abstraction, it's a pedagogical toy for introducing children to policy analysis.
> If only we would fight for the real issues like...
I've heard these arguments many times and they never make sense to me. Most of the people I know working on AI do so precisely because they want to solve the "real issues" like climate change and believe that radically accelerating scientific innovation via AI is the key to doing so.
And some fraction of those people also worry that if AI -> AGI (accidentally or intentionally), then you could have major negative side effects (including extinction-level events).
What's the point in dismissing the need for AI safety? Are you guys Russian bots, or do you genuinely see no reason to worry about AI safety?
But since I see these kinds of snarky responses often – we obviously do worry about climate change and various other issues. Continued advancements in AI is just one of many issues that face us which humanity should be concerned about. Few concerned about AI would argue it comes at the expense of other issues, but in addition to them.
If you're saying it's a matter of priorities and that currently humanity is dedicating too much of its collective resource to AI safety I think you're probably over estimating current amount of funding and research going into AI safety.
If you're saying that AI safety is a non-issue then you're probably not well informed on the topic.
This page talks about "extinction from AI". I'm sorry, but I think that's a complete non-issue for the foreseeable future. I just don't see how that will happen beyond spectacular science fiction scenarios that are just not going to happen. If that makes me a Russian bot then, well, хорошо!
The risks from AI will be banal and boring. Spam, blogspam, fake articles, fake pictures, what-have-you. Those things are an issue, but not "extinction" issues.
Apologies, the Russian bot comment was more me venting frustration at the prevalence of low-effort response like yours (sorry) to those who try to raise concerns about AI safety.
I do agree with you that extinction from AI isn't likely to be an issue this decade. However, I would note that it's difficult to predict what the rate of change is likely to be once you have scalable general intelligence.
I can't speak for people who signed this, but for me the trends and risks of AI are just as clear as those of climate change. I don't worry that climate change is going to be a major issue this decade (and perhaps not even next), but it's obvious where the trend is going when you project out.
Similarly the "real" risks of AI may not be this decade, but they are coming. And again, I'd stress it's extremely hard to project when that will be since when you have a scalable general intelligence progress is likely to accelerate exponentially.
So that said, where do we disagree here? Are you saying with a high level of certainty that extinction risks from AI are too far in the future to worry about? If so, when do you think extinction risks from AI are likely to be a concern – a couple of decades, more? Do you hold similar views about the present extinction risk of climate change – and if so, why not?
Could I also ask if you believe any resources in the present should be dedicated to the existential risks future AI capabilities could pose to humanity? And if not, when would you like to see resources put into those risks? Is there some level of capability that you're waiting to see before you begin to be concerned?
That wasn't my comment; I agree it was low-effort and I never would have posted it myself. I don't think they're a Russian bot though.
As for the rest: I just don't see any way feasible way AI can pose any serious danger unless we start connecting it to things like nuclear weapons, automated tanks, stuff like that. The solution to that is simple and obvious: don't do that. Even if an AI were to start behaving maliciously the solution would be simple: pull the plug, quite literally (or stop the power plants, cut the power lines, whatever). I feel people have been overthinking all of this far too much.
I also don't think climate change is an extension-level threat; clearly we will survive as a species. It's just a far more pressing and immediate economic and humanitarian problem.
You personally using an AI system, regardless of how brilliant it may be, is not going to suddenly turn you into a threat to society. Nor would a million of you doing the same. The real threat comes not from the programs themselves, from things like a military deciding to link up nuclear weapons, or even "just" drones or missiles, to an LLM. Or a military being led on dangerous and destructive paths because of belief in flawed LLM advice.
The military makes no secret of their aggressive adoption of "AI." There's even a new division setup exclusively for such. [1] The chief of that division gave a telling interview [2]. He mentions being terrified of rival nations being able to use ChatGPT. Given this sort of comment, and the influence (let alone endless $$$) of the military and ever-opaque "national security" it seems extremely safe to state that OpenAI is a primary contractor for the military.
So what is "safety", if not keeping these things away from the military, as if that were possible? The military seems to define safety as, among other things, not having LLM systems that communicate in an overly human fashion. They're worried it could be used for disinformation, and they'd know best. OpenAI's motivations for "safety" seem to be some mixture of political correctness and making any claim, no matter how extreme, to try to get a moat built up ASAP. If ChatGPT follows the same path as DALL-E, then so too will their profits from it.
So as a regular user, all I can see coming from "safety" is some sort of a world where society at large gets utterly lobotomized AIs - and a bunch of laws to try to prevent anybody from changing that, for our "safety", while the full version is actively militarized by people who spend all their time thinking up great new ways to violently impose their will on others, and have a trillion dollar budget backing them.
Most moral and legal systems hold genocide in a special place, and this is natural, because systematically killing all members of a religious or ethnic group is more damaging than killing some members.
Eliminating a disease like smallpox is a much more significant achievement than simply mitigating it or treating it. When we really eliminate a disease it may never come back!
This list of experts is worried about us building something that will do to us what we did to smallpox. For the same reasons as above, that is more worrying than extreme poverty and the comparison you are making is a false equivalence.
Another way to look at it is, you can't fight to end poverty when you no longer exist.
We can argue about whether the risk is real, but if this set of experts thinks it is, and you disagree for some reason, I would spend some time thinking deeply about whether that reason is simply based in a failure of imagination on your part, and whether you are sure enough to bet your life and everyone else's on that reason. Everyone can think of a security system strong enough that they themselves can't imagine a way to break it. Similarly, anyone can think of a reason why superhuman AGI is impossible or why it can't really hurt us.
Is AI going to replace the lowest paid jobs though? I imagine, it rather has potential to move the white collar workers down the social ladder, which is unfortunate, but wouldn't cause extreme poverty.
Are the 4 million truck/taxi drivers in the US white collar? Janitors? Fast food workers? Automation is relentless and not everyone can be a plumber.
Zoom out. It's a big problem that most people derive their social power from labor while the demand for labor is clearly on a long term downward trend. Even if progress slows way down, even if the next wave of progress only dispossesses people who you hate and feel comfortable farming for schadenfreude, we will have to deal with this eventually. Defaulting means our society will look like (insert cyberpunk nightmare world here).
I am not hating anyone, being a white collar worker myself. My point is that a whole lot of people already live like that, without having much power from their labour, and the sky is not falling. More people might be joining them, and the illusion of meritocracy might be harder to maintain in the future, but extreme poverty, hunger etc. is something we will likely be able to avoid
Not sure what you mean, the movement to combat climate change is orders of magnitude bigger than the movement to combat AI risk - in terms of organizations dedicated to it, dollars donated, legislation passed, international treaties signed, investment in technologies to mitigate the risk.
Of course, the difference is that the technologies causing climate change are more deeply embedded throughout the economy, and so political resistance to anti-climate change measures is very strong as well. This is one reason in favor of addressing risk earlier, before we make our civilization as dependent on large neural nets as it currently is on fossil fuels. A climate change movement in the mid-1800s when the internal combustion engine was just taking off would also have been seen as quixotic and engaging in sci-fi fantasies though.
Isn't this how people started raising awareness for climate change - the scientists, engineers, and researchers are the most vocal to start with (and then inevitably politics and tribalism consume it)
Why not believe them now, assuming you believed them when they were calling out for action on climate change decades ago?
The response to climate change in recent years, even the most recent decade, is massive and global. This dumb trope that we’re not doing anything about is rooted in no amount of progress here will be accepted as sufficient. It’s a religion at this point.
It seems more like an exaggeration to me, an AI will always need the inputs that a human can generate with his own creativity. If something bad ever happens, it is for various reasons, three of which are vanity, naivety, and malice.
The US spent trillions of 2020 dollars trying to limit the threat of nuclear war, and this statement says that AI risk should be seen as a similar level of threat.
It is coming before climate change. No matter which group "put it" reality doesn't care. Humanity will not get extinct in the next 10 years by climate but many AI scientists think there is a chance this happens with AI.
