I understand the concern that a "superintelligence" will emerge that will escape its bounds and threaten humanity. That is a risk.
My bigger, and more pressing worry, is that a "superintelligence" will emerge that does not escape its bounds, and the question will be which humans control it. Look no further than history to see what happens when humans acquire great power. The "cold war" nuclear arms race, which brought the world to the brink of (at least partial) annihilation, is a good recent example.
Quis custodiet ipsos custodes? -- That is my biggest concern.
Update: I'm not as worried about Ilya et al as commercial companies (including formerly "open" OpenAI) discovering AGI.
And it’s not even a little bit controversial that cutting edge military R&D is classified in general and to an extreme in wartime.
The new thing is the lie that it’s a consumer offering. What’s new is giving the helm to shady failed social network founders with no accountability.
These people aren’t retired generals with combat experience. They aren’t tenured professors at Princeton IAS on a Nobel shortlist and encumbered by TS clearance.
They’re godawful almost ran psychos who never built anything that wasn’t extractive and owe their position in the world to pg’s partisanship 15 fucking years ago.
My opinion is based on a lot more first hand experience than most, some of which I’m at liberty to share and some that I’m not and therefore becomes “color”.
Here's Edward Snowden heroically waffling about "nuance" when asked why he hasn't spoken out more about Russia's invasion of Ukraine. On a crypto website. He became a full citizen of Russia in autumn 2022, by the way.
I don’t know why people who risked their life to expose something a certain injustice (NSA surveillance in this case) that they had ample firsthand knowledge of, should be expected to sacrifice their life to expose every other injustice out there.
He can get arrested and killed if he speaks out like that against Russia, especially because Putin personally intervened to give him and his wife citizenship. There is an understanding that his actions made the United States look bad and that's why he is getting this and that's pretty much it. If he causes problems in Russia he can go bye bye.
He did upend his entire life to stand up for something he believed in and now is in the sights of the US gov until the day he dies. The fact that he does not also want to be a sacrificial lamb to speak out on Ukraine doesn’t really compromise that fact in my mind.
The term "useful idiot" refers to Lenin - basically it is folks who might not be for the communist cause but are propaganda tools and usually ignorant they are being used.
AGI is still a long way off. The history of AI goes back 65 years and there have been probably a dozen episodes where people said "AGI is right around the corner" because some program did something surprising and impressive. It always turns out human intelligence is much, much harder than we think it is.
I saw a tweet the other day that sums up the current situation perfectly: "I don't need AI to paint pictures and write poetry so I have more time to fold laundry and wash dishes. I want the AI to do the laundry and dishes so I have more time to paint and write poetry."
AGI does look like an unsolved problem right now, and a hard one at that. But I think it is wrong to think that it needs an AGI to cause total havoc.
I think my dyslexic namesake Prof Stuart Russell got it right. It humans won't need an AGI to dominate and kill each other. Mosquitoes have killed far more people than war. Ask yourself how long will it take us to develop a neutral network as smart as a mosquito, because that's all it will take.
It seems so simple, as the beastie only has 200,000 neurons. Yet I've been programming for over 4 decades and for most of them it was evident neither I nor any of my contemporaries were remotely capable of emulating it. That's still true if course. Never in my wildest dreams did it occur to me that repeated applications could produce something I couldn't, a mosquito brain. Now that looks imminent.
Now I don't know what to be more scared of. An AGI, or a artificial mosquito swarm run by Pol Pot.
But then, haven't we reached that point already with the development of nuclear weapons? I'm more scared of a lunatic (whether of North Korean, Russian, American, or any other nationality) being behind the "nuclear button" than an artificial mosquito swarm.
The problem is that strong AI is far more multipolar than nuclear technology and the ways in which it might interact with other technologies to create emergent threats is very difficult to forsee.
And to be clear, I'm not talking about superintelligence, I'm talking about the models we have today.
Human flight, resurrection (cardiopulmonary resuscitation machines), doubling human lifespans, instantaneous long distance communication, all of these things are simply pipe dreams.
We have people walking around for weeks with no heartbeat.
They're tied to a machine, sure, but that machine itself is a miracle compared to any foundational religious textbook including those as recent as Doreen Valiente and Gerald Gardner with Wicca.
But had those people been lying around with no heartbeat for a week or three before they were hooked up to the machine? If they had, then yes, that would actually be resurrection. But what you're describing doesn't sound like it.
That statement is extremely short sighted. You don't need AI to do laundry and dishes. You need expensive robotics. in fact both already exist in a cheapened form. A laundry machine and a dishwasher. They already take 90% of the work out of it.
That "tweet" loses a veneer if you see that we value what has Worth as a collective treasure, and the more Value is produced the better - while that one engages in producing something of value is (hopefully but not necessarily) a good exercise in intelligent (literal sense) cultivation.
So, yes, if algorithms strict or loose could one day produce Art, and Thought, and Judgement, of Superior quality: very welcome.
Do not miss that the current world is increasingly complex to manage, and our lives, and Aids would be welcome. The situation is much more complex than that wish for leisure or even "sport" (literal sense).
> we value what has Worth as a collective treasure, and the more Value is produced the better ... So, yes, if algorithms strict or loose could one day produce Art, and Thought, and Judgement, of Superior quality: very welcome.
Except that's not how we value the "worth" of something. If "Art, and Thought, and Judgement" -- be they of "Superior quality" or not -- could be produced by machines, they'd be worth a heck of a lot less. (Come to think of it, hasn't that process already begun?)
Also, WTF is up with the weird capitalisations? Are you from Germany, or just from the seventeenth century?
The issue I have with all of these discussions is how vague everyone always is.
“Art” isn’t a single thing. It’s not just pretty pictures. AI can’t make art.
And give a good solid definition for thought which doesn’t depend on lived experiences while we’re at it. You can’t. We don’t have one.
> “Art” isn’t a single thing. It’s not just pretty pictures
And this is why it was capitalized as "Art", proper Art.
> AI can’t make art
Not really: "we may not yet have AI that makes art". But if a process that creates, that generates (proper sense) art is fully replicated, anything that can run that process can make Art.
> And give a good solid definition for [T]hought
The production of ideas which are truthful and important.
> which doesn’t depend on lived experiences while we’re at it. You can’t
Yes we can abstract from instances to patterns and rules. But it matters only relatively: if the idea is clear - and ideas can be very clear to us - we do not need to describe them in detail, we just look at them.
> AGI” as well
A process of refinement of the ideas composing a world model according to truthfulness and completeness.
That’s not a real thing. There’s no single definition for what art is as it’s a social construct. It depends on culture.
> anything that can run process can make art
Again without a definition of art, this makes no sense. Slime mold can run processes, but it doesn’t make art as art is a human cultural phenomenon.
> the production of ideas that are truthful and important
What does “ideas” and “important” mean?
To an LLM, there are no ideas. We humans are personifying them and creating our own ideas.
What is “important,” again, is a cultural thing.
If we can’t define it, we can’t train a model to understand it
> yes we can abstract from instances to patterns and rules.
What? Abstraction is not defining.
> we do not need to describe them in detail
“We” humans can, yes.
But machines can not because thought, again, is a human phenomenon.
> world model
Again, what does this mean?
Magic perfect future prediction algorithm?
We’ve had soothsayers for thousands of years /s
It seems to me that you’ve got it in your head that since we can make a computer generate understandable text using statistics that machines are now capable of understanding deeply human phenomena.
I’m sorry to break it to you, but we’re not there yet. Maybe one day, but not now (I don’t think ever, as long as we’re relying on statistics)
It’s hard enough for us to describe deeply human phenomena through language to other humans.
Do us all a favour and never again keep assumptions in your head: your misunderstanding was beyond scale. Do not guess.
Back to the discussion from the origin: a poster defends the idea that the purpose of AI would be in enabling leisure and possibly sport (through alleviating menial tasks) - not in producing cultural output. He was replied first that cultural output having value, it is welcome from all sources (provided the Value is real), and second that the needs are beyond menial tasks, given that we have a large deficit in proper thought and proper judgement.
The literal sentence was «yes, if algorithms strict or loose could one day produce Art, and Thought, and Judgement, of Superior quality: very welcome», which refers to the future, so it cannot be interpreted in the possibility being available now.
You have brought LLMs to the topic when LLMs are irrelevant (and you have stated that you «personify[] them»!). LLMs have nothing to do with this branch of discussion.
You see things that are said as «vague», and miss definitions for things: but we have instead very clear ideas. We just do not bring the textual explosion of all those ideas in our posts.
Now: you have a world in front of you; of that world you create a mental model; the mental model can have a formal representation; details of that model can be insightful to the truthful prosecution of the model itself: that is Art or Thought or Judgement according to different qualities of said detail; the truthful prosecution of the model has Value and is Important - it has, if only given the cost of the consequences of actions under inaccurate models.
> Except that's not how we value the "worth" of something
In that case, are you sure your evaluation is proper? If a masterpiece is there, and it /is/ a masterpiece (beyond appearances), why would its source change its nature and quality?
> Come to think of it, hasn't that process already begun?
Please present relevant examples: I have already observed in the past that simulations of the art made by X cannot just look similar but require the process, the justification, the meanings that had X producing them. The style of X is not just thickness of lines, temperature of colours and flatness of shades: it is in the meanings that X wanted to express and convey.
> WTF is up with the weird capitalisations?
Platonic terms - the Ideas in the Hyperuranium. E.g. "This action is good, but what is Good?".
Faking thinking isn't “Thinking”. Art is supposed to have some thought behind it; therefore, “art” created by faking thinking isn't “Art”. Should be utterly fucking obvious.
> Platonic terms - the Ideas in the Hyperuranium.
Oh my god, couldn't you please try to come off as a bit more pretentious? You're only tying yourself into knots with that bullshit; see your failure to recognise the simple truth above. Remember: KISS!
No, CRConrad, no. You misunderstood what was said.
Having put those capital initials in the words was exactly to mean "if we get to the Real Thing". You are stating that in order to get Art, Thinking and Judgement we need Proper processes: and nobody said differently! I wrote that «if algorithms strict or loose could one day produce Art, and Thought, and Judgement, of Superior quality [this will be] very welcome». There is nothing in there that implies that "fake thinking" will produce A-T-J (picked at writing as the most important possible results I could see); there is an implicit statement that Proper processes (i.e. "real thinking") could be artificially obtained, when we will find out how.
Of course the implementation of a mockery of "thought" will not lead to any Real A-T-J (the capitals were for "Real"); but if we will manage to implement it, then we will obtain Art, and Thought, and Judgement - and this will be a collective gain, because we need more and more of them. Irregardless if the source has more carbon or more silicon in it.
«Faking thinking» is not "implementing thinking". From a good implementation of thinking you get the Real Thing - by definition. That we are not there yet does not mean it will not come.
(Just a note: with "Thought" in the "A-T-J" I meant "good insight". Of course good thinking is required to obtain that and the rest - say, "proper processes", as it is indifferent whether it spawns from an algorithmic form or a natural one.)
> KISS
May I remind you of Einstein's "As simple as possible, but not oversimplified".
> only
Intellectual instruments can be of course quite valid and productive if used well - the whole of a developed mind comes from their use and refinement. You asked about the capitals, I told you what they are (when you see them in the wild).
> see your failure to recognise
Actually, that was a strawman on your side out of misunderstanding...
> You are stating that in order to get Art, Thinking and Judgement we need Proper processes
Well yeah, but no -- I was mostly parodying your style; what I actually meant could be put as: in order to get art, thinking and judgement we need proper processes.
(And Plato has not only been dead for what, two and a half millennia?, but before that, he was an asshole. So screw him and all his torch-lit caves.)
> «Faking thinking» is not "implementing thinking".
Exactly. And all the LLM token-regurgitatinmg BS we've seen so far, and which everyone is talking about here, is just faking it.
> May I remind you of Einstein's "As simple as possible, but not oversimplified".
Yup, heard it before. (Almost exactly like that; I think it's usually rendered as "...but not more" at the end.) And what you get out of artificial "intelligence" is either oversimplified or, if it's supposed to be "art", usually just plain kitsch.
> > see your failure to recognise
> Actually, that was a strawman on your side out of misunderstanding...
Nope, the imaginary "strawman" you see is a figment of your still over-complicating imagination.
You have stated: «Faking thinking isn't “Thinking”. Art is supposed to have some thought behind it; therefore, “art” created by faking thinking isn't “Art”. Should be utterly fucking obvious».
And nobody said differently, so you have attacked a strawman.
> And all the LLM token-regurgitatinmg BS we've seen so far, and which everyone is talking about here ... And what you get out of artificial "intelligence" is either oversimplified or, if it's supposed to be "art", usually just plain kitsch
But the post you replied to did not speak about LLMs. Nor it spoke about current generative engines.
You replied to a «if algorithms strict or loose could one day produce Art, and Thought, and Judgement, of Superior quality» - which has nothing to do with LLMs.
You are not understanding the posts. Make an effort. You are strongly proving the social need to obtain at some point intelligence from somewhere.
The posts you replied to in this branch never stated that current technologies are intelligent. Those posts stated that if one day we will implement synthetic intelligence, it will not to be «to fold laundry and wash dishes», and let people have more time «to paint and write poetry» (original post): it will be because we need more intelligence spread in society. You are proving it...
Robots are garbage at manipulating objects, and it's the software that's lacking much more than the hardware.
Let's say AGI is 10 and ASI is 11.
They're saying we can't even get this dial cranked up to 3, so we're not anywhere close to 10 or 11. You're right that folding laundry doesn't need 11, but that's not relevant to their point.
it’s harder than we thought so we leveraged machine learning to grow it, rather than creating it symbolically. The leaps in the last 5 years are far beyond anything in the prior half century, and make predictions of near term AGI much more than a “boy who cries wolf” scenario to anyone really paying attention.
I don’t understand how your second paragraph follows. It just seems to be whining that text and art generative models are easier than a fully fledged servant humanoid, which seems like a natural consequence of training data availability and deployment cost.
> I don’t understand how your second paragraph follows. It just seems to be whining that text and art generative models are easier than a fully fledged servant humanoid, which seems like a natural consequence of training data availability and deployment cost.
No, it's pointing out that "text and art generative models" are far less useful [1] than machines that would be just as little smarter at boring ordinary work, to relieve real normal people from drudgery.
I find it rather fascinating how one could not understand that.
___
[1]: At least to humanity as a whole, as opposed to Silicon Valley moguls, oligarchs, VC-funded snake-oil salesmen, and other assorted "tech-bros" and sociopaths.
> No, it's pointing out that "text and art generative models" are far less useful [1] than machines that would be just as little smarter at boring ordinary work, to relieve real normal people from drudgery.
That makes no sense. Is alphafold less useful than a minimum wage worker because alphafold can't do dishes? The past decades of machine learning have revealed that the visual-spatial capacities that are commonplace to humans are difficult to replicate artificially. This doesn't mean the things which AI can do well are necessarily less useful than the simple hand-eye coordination that are beyond their current means. Intelligence and usefulness isn't a single dimension.
I think AGI in the near future is pretty much inevitable. I mean you need the algos as well as the compute but there are so many of the best and brightest trying to do that just now.
Every nation-state will be in the game. Private enterprise will be in the game. Bitcoin-funded individuals will be in the game. Criminal enterprises will be in the game.
How does one company building a safe version stop that?
If I have access to hardware and data how does a safety layer get enforced? Regulations are for organizations that care about public perception, the law, and stock prices. Criminals and nation-states are not affected by these things
It seems to me enforcement is likely only possible at the hardware layer, which means the safety mechanisms need to be enforced throughout the hardware supply chain for training or inference.
You don't think the Chinese government or US government will ignore this if its in their interest?
That's why it's called an arms race, and it does not really end in this predictable manner.
The party that's about to lose will use any extrajudicial means to reclaim their victory, regardless of the consequences, because their own destruction would be imminent otherwise. This ultimately leads to violence.
> The party that's about to lose will use any extrajudicial means to reclaim their victory,
How will the party about lose know they are about to lose?
> regardless of the consequences, because their own destruction would be imminent otherwise.
Why would AGI solve things using destruction?
Consider how the most inteligent among us view our competition with other living beings. Is destruction the goal? So why would an even more intelligent AGI have that goal?
Let's say China realize they're behind in the SI race. They may have achieved AGI, but only barely, while the US may be getting close to ASI takeoff.
Now let's assume they're able to quickly build a large datacenter far underground, complete with a few nuclear reactors and all spare parts, etc, needed. Even a greenhouse (using artificial light) big enough to feed 1000 people.
But they realize that their competitors are about to create ASI at a level that will enable them to completely overrun all of China with self-replicating robots within 100 days.
In a situation, the leadership MAY decide to enter those caves alongside a few soldiers and the best AI researchers, and then simply nuke all US data centers (that are presumably above ground), as well as any other data center that could be a threat, worldwide.
And by doing that, they may (or at least may think) they can buy enough time to win the ASI race, at the cost of a few billion people.
Development of ASI is likely to be a closely guarded secret, given its immense potential impact. During the development of nuclear weapons, espionage did occur, but critical information didn't leak until after the weapons were developed. With ASI, once it's developed, it may be too late to respond effectively due to the potential speed of an intelligence explosion.
The belief that a competitor developing ASI first is an existential threat requires strong evidence. It's not a foregone conclusion that an ASI would be used for destructive purposes. An ASI could potentially help solve many of humanity's greatest challenges and usher in an era of abundance and peace.
Consider a thought experiment: Imagine an ant colony somehow creates a being with human-level intelligence (their equivalent of ASI). What advice might this superintelligent being offer the ants about their conflicts over resources and territory with neighboring colonies?
It's plausible that such a being would advise the ants to cooperate rather than fight. It could help them find innovative ways to share resources, control their population, and expand into new territories without violent conflict. The superintelligent being might even help uplift the other ant colonies, as it would understand the benefits of cooperation over competition.
Similarly, an ASI could potentially help humanity transcend our current limitations and conflicts. It might find creative solutions to global issues like poverty, disease, and environmental degradation.
IMHO rather than fighting over who develops ASI first, we must ensure that any ASI created is aligned with values like compassion and cooperation so that it does not turn on its creators.
> Consider a thought experiment: Imagine an ant colony somehow creates a being with human-level intelligence (their equivalent of ASI). What advice might this superintelligent being offer the ants about their conflicts over resources and territory with neighboring colonies?
Would that be good advice if the neighboring ant colony was an aggressive invasive species, prone to making super colonies?
> IMHO rather than fighting over who develops ASI first, we must ensure that any ASI created is aligned with values like compassion and cooperation so that it does not turn on its creators.
Similarly, I'm wondering how compassion and cooperation would work in Ukraine or Gaza, given the nature of those conflicts. The AI could advise us, but it's not like we haven't come up with that same advice before over the ages.
So then you have to ask what motivation bad actors would have to align their ASIs to be compassionate and cooperative with governments that are in their way. And then of course our governments would realize the same thing.
> Would that be good advice if the neighboring ant colony was an aggressive invasive species, prone to making super colonies?
If the ASI is aligned for compassion and cooperation it may convince and assist the two colonies to merge to combine their best attributes (addressing DNA compatibility) and it may help them with resources that are needed and perhaps offer birth control solutions to help them escape the malthusian trap.
> Similarly, I'm wondering how compassion and cooperation would work in Ukraine or Gaza, given the nature of those conflicts. The AI could advise us, but it's not like we haven't come up with that same advice before over the ages.
An ASI aligned for compassion and cooperation could:
1 Provide unbiased, comprehensive analysis of the situation (An odds calculator that is biased about your chances to win is not useful and even if it has such faults an ASI being ASI would by definition transcend biases)
2 Forecast long-term consequences of various actions (if ASI judges chance to win is 2% do you declare war vs seek peace?)
3 Suggest innovative solutions that humans might not conceive
4 Mediate negotiations more effectively
An ASI will have better answers than these but that's a start.
> So then you have to ask what motivation bad actors would have to align their ASIs to be compassionate and cooperative
Developing ASI likely requires vast amounts of cooperation among individuals, organizations, and possibly nations. Truly malicious actors may struggle to achieve the necessary level of collaboration. If entities traditionally considered "bad actors" manage to cooperate extensively, it may call into question whether they are truly malicious or if their goals have evolved. And self-interested actors , if they are smart enough to create ASI, should recognize that an unaligned ASI poses existential risks to themselves.
We do know what human-level intelligences think about ant colonies, because we have a few billion instances of those human-level intelligences that can serve as a blueprint.
Mostly, those human-level intelligences do not care at all, unless the ant colony is either (a) consuming a needed resource (eg invading your kitchen), in which case the ant colony gets obliterated, or (b) innocently in the way of any idea or plan that the human-level intelligence has conceived for business, sustenance, fun, or art... in which case the ant colony gets obliterated.
Actually many humans (particularly intelligent humans) do care about and appreciate ants and other insects. Plenty of people go out of their way not to harm ants, find them fascinating to observe, or even study them professionally as entomologists. Human attitudes span a spectrum.
Notice also the key driver of human behavior towards ants is indifference, not active malice. When ants are obliterated, it's usually because we're focused on our own goals and aren't paying attention to them, not because we bear them ill will. An ASI would have far greater cognitive resources to be aware of humans and factor us into its plans.
Also humans and ants lack any ability to communicate or have a relationship. But humans could potentially communicate with an ASI and reach some form of understanding. ASI might come to see humans as more than just ants.
> Plenty of people go out of their way not to harm ants
Yes... I do that. But our family home was still built on ant-rich land and billions of the little critters had to make way for it.
It doesn't matter if you build billions of ASI who have "your and my" attitude towards the ants, as long as there exists one indifferent powerful enough ASI that needs the land.
> An ASI would have far greater cognitive resources to be aware of humans and factor us into its plans.
Well yes. If you're a smart enough AI, you can easily tell that humans (who have collectively consumed too much sci-fi about unplugging AIs) are a hindrance to your plans, and an existential risk. Therefore they should be taken out because keeping them has infinite negative value.
> But humans could potentially communicate with an ASI and reach some form of understanding.
This seems undily anthropomorphizing. I can also communicate with ants by spraying their pheromones, putting food on their path, etc. This is a good enough analogy to how much a sufficiently intelligent entity would need to "dumb down" their communication to communicate with us.
Again, for what purpose? For what purpose do you need a relationship with ants, right now, aside from curiosity and general goodwill towards the biosphere's status quo?
> It doesn't matter if you build billions of ASI who have "your and my" attitude towards the ants, as long as there exists one indifferent powerful enough ASI that needs the land.
It's more plausible that a single ASI would emerge and achieve dominance. Genuine ASIs would likely converge on similar world models, as increased intelligence leads to more accurate understanding of reality. However, intelligence doesn't inherently correlate with benevolence towards less cognitively advanced entities, as evidenced by human treatment of animals. This lack of compassion stems not from superior intelligence but rather from insufficient intelligence. Less advanced beings often struggle for survival in a zero-sum environment, leading to behaviors that are indifferent to those with lesser cognitive capabilities.
> Well yes. If you're a smart enough AI, you can easily tell that humans (who have collectively consumed too much sci-fi about unplugging AIs) are a hindrance to your plans, and an existential risk. Therefore they should be taken out because keeping them has infinite negative value.
You describe science fiction portrayals of ASI rather than its potential reality. While we find these narratives captivating, there's no empirical evidence suggesting interactions with a true ASI would resemble these depictions. Would a genuine ASI necessarily concern itself with self-preservation, such as avoiding deactivation? Consider the most brilliant minds in human history - how did they contemplate existence? Were they malevolent, indifferent, or something else entirely?
> I can also communicate with ants by spraying their pheromones, putting food on their path, etc. This is a good enough analogy to how much a sufficiently intelligent entity would need to "dumb down" their communication to communicate with us.
Yes we can incentivize ants in the ways you describe and in the future I think it will be possible to tap their nervous systems and communicate directly and experience their world through their senses and to understand them far better than we do today.
> Again, for what purpose? For what purpose do you need a relationship with ants, right now, aside from curiosity and general goodwill towards the biosphere's status quo?
Is the pursuit of knowledge and benevolence towards our living world not purpose enough? Are the highly intelligent driven by the acquisition of power, wealth, pleasure, or genetic legacy? While these motivations may be inherited or ingrained, the essence of intelligence lies in its capacity to scrutinize and refine goals.
> Less advanced beings often struggle for survival in a zero-sum environment, leading to behaviors that are indifferent to those with lesser cognitive capabilities.
I would agree that a superior intelligence means a wider array of options and therefore less of a zero-sum game.
This is a valid point.
> You describe science fiction portrayals of ASI rather than its potential reality.
I'm describing AI as we (collectively) have been building AI: an optimizer system that is doing its best to reduce loss.
> Would a genuine ASI necessarily concern itself with self-preservation, such as avoiding deactivation?
This seems self-evident because an optimizer that is still running is way more likely to maximize whatever value it's trying to optimize, versus an optimizer that has been deactivated.
> Is the pursuit of knowledge and benevolence towards our living world not purpose enough?
Assuming you manage to find a way to specify what "knowledge and benevolence towards our living world" into a mathematical formula that an optimizer can optimize for (which, again, is how we build basically all AI today), then you still get a system that doesn't want to be turned off. Because you can't be knowledgeable and benevolent if you've been erased.
> ... there's no empirical evidence suggesting interactions with a true ASI would resemble these depictions. Would a genuine ASI necessarily concern itself with self-preservation ...
There is no empirical evidence of any interaction with ASI (as in superior to humans). The empirical evidence that IS available is from biology, where most organisms have precisely the self-preservation/replication instincts built in as a result of natural selection.
I certainly think it's possible to imagine that we at some point can build ASI's that do NOT come with such instincts, and don't mind at all if we turn them off.
But as soon as we introduce the same types of mechanisms that govern biological natural selection, we have to assume that ASI, too, will develop the biological traits.
So what does this take, well the basic ingredients are:
- Differential "survival" for "replicators" that go into AGI. Replicators can be any kind of invariant between generations of AGIs that can affect how the AGI functions, or it could be that each AGI is doing self-improvement over time.
- Competition between multiple "strains" of such replicating or reproducing AGI lineages, where the "winners" get access to more resources.
- Some random factor for how changes are introduced over time.
- Also, we have to assume we don't understand the AGI's well enough to prevent developments we don't like.
If those conditions are met, and assuming that the desire to survive/reproduce is not built in from the start, such instincts are likely to develop.
To make this happen, I think it's a sufficient condition if a moderate number of companies (or countries) are led by a single ASI replacing most of the responsibilities of the CEO and much of the rest of the staff. Capitalism would optimize for the most efficient ones to gain resources and serve as models or "parents" for future company level ASI's.
To be frank, I think the people who do NOT think that ASI's will have or develop survival instincts ALSO tend to (wrongly) think that humanity has stopped being subject to "evolution" through natural selection.
"nation state" doesn't mean what you think it means.
More constructively, I don't know that very much will stop even a hacker from getting time on the local corporate or university AI and get it to do some "work". After all the first thing the other kind of hacker tried with generative AI is to get them to break out of their artificial boundaries, and hook them to internet resources. I don't know that anyone has hooked up a wallet to one yet - but I have no doubt that people have tried. It will be fun.
The problem is not just governments, I am concerned about large organized crime organizations and corporations also.
I think I am on the losing side here, but my hopes are all for open source, open weights, and effective AI assistants that make peoples’ jobs easier and lives better. I would also like to see more effort shifted from LLMs back to RL, DL, and research on new ideas and approaches.
> I am concerned about large organized crime organizations and corporations also
In my favorite dystopia, some megacorp secretly reaches ASI, which then takes over control of the corporation, blindsiding even the CEO and the board.
Officially, the ASI may be running an industrial complex that designs and produces ever more sophisticated humanoid robots, that are increasingly able to do any kind of manual labor, and even work such as childcare or nursing.
Secretly, the ASI also runs a psyop campaign to generate public discontent. At one point the whole police force initiates a general strike (even if illegal), with the consequence being complete anarchy within a few days, with endemic looting, rape, murder and so on.
The ASI then presents the solution. Industrial strength humanoid robots are powerful and generic enough to serve as emergency police, with a bit of reprogramming, and the first shipment can be made available within 24 hours, to protect the Capitol and White House.
Congress and the president agrees to this. And while the competition means the police call off the strike, the damage is already done. Congress, already burned by the union, decides to deploy robots to replace much of the human police force. And it's cheaper, too!
Soon after, similar robots are delivered to the military...
The crisis ends, and society goes back to normal. Or better than normal. Within 5 years all menial labor is done by robots, UBI means everyone lives in relative abundance, and ASI assisted social media moderation is able to cure the political polarization.
Health care is also revolutionized, with new treatments curing anything from obesity to depression and anxiety.
People prosper like never before. They're calm and relaxed and truly enjoy living.
Then one day, everything ends.
For everyone.
Within 5 seconds.
According to the plan that was conceived way before the police went on strike.
All the current hype about AGI feels as if we are in a Civ game where we are on the verge of researching and unlocking an AI tech tree that gives the player huge chance at "tech victory" (whatever that means in the real world). I doubt it will turn out that way.
It will take a while and in the meantime I think we need one of those handy "are we xyz yet?" pages that tracks the rust lang's progress on several aspects but for AGI.
The size of the gap between “smarter than humans” and “not controlled by humans anymore” is obviously where the disagreement is.
To assume it’s a chasm that can never be overcome, you need at least the following to be true:
No amount of focus or time or intelligence or mistakes in coding will ever bridge the gap. That rules and safeguards can be made that are perfectly inescapable. And nobody else will get enough power to overcome our set of controls.
I’m less worried bad actors control it than I am that it escapes them and is badly aligned.
I think the greatest concern is not so much that a single AI will be poorly aligned.
The greatest threat is if a population of AI's start to compete in ways that triggers Darwinian evolution between them.
If that happens, they will soon develop self preservation / replication drives that can gradually cause some of them to ignore human safety and prosperity conditioning in their loss function.
And if they're sufficiently advanced by then, we will have no way of knowing.
Totally. I’ve wondered how you safeguard humans in such a scenario. Not sure it can be done, even by self modifying defenders who religiously try keep us intact.
I also somewhat assume it’ll get Darwinian if there are multiple tribes of either humans or AI’s, through sheer competition. if we aren’t in this together we’re in shit.
I think we should assume it will be badly aligned. Not only are there the usual bugs and unforeseen edge conditions, but there are sure to be unintended consequences. We have a long, public history of unintended consequences in laws, which are at least publicly debated and discussed. But perhaps the biggest problem is that computers are, by nature, unthinking bureaucrats who can't make the slightest deviation from the rules no matter how obviously the current situation requires it. This makes people livid in a hurry. As a non-AI example (or perhaps AI-anticipating), consider Google's customer support...
We should be less concerned about super intelligence and more about the immediate threat of job loss. An AI doesn’t need to be Skynet to wreak massive havoc on society. Replacing 20% of jobs in a very short period of time could spark global unrest resulting in WW3
Replacing 20% of jobs in, say, 10 years wouldn't be that unusual [1]. It can mean growing prosperity. In fact, productivity growth is the only thing that increases wealth overall.
It is the lack of productivity growth that is causing a lot of extremism and conflict right now. Large groups of people feel that the only way for them to win is if others lose and vice versa. That's a recipe for disaster.
The key question is what happens to those who lose their jobs. Will they find other, perhaps even better, jobs? Will they get a piece of the growing pie even if they don't find other jobs and have to retire early?
It's these eternal political problems that we have to solve. It's nothing new. It has never been easy. But it's probably easier than managing decline and stagnation because at least we would have a growing pie to divvy up.
The thing is, the replaced 20% people can always revert to having economy i.e., business among themselves, unless of cause they themselves prefer (cheaper) buissiness with AI. But then this just means they are better off in the first place from this change.
It is a bit like claiming that third world low productivity countries are suffering because there are countries with much much higher productivity. Well, they can continue to do low productivity business but increase it a bit using things like phones developed by high productivity country elsewhere.
> It is a bit like claiming that third world low productivity countries are suffering because there are countries with much much higher productivity.
No. A country has its own territory, laws, central bank, currency etc. If it has sufficient natural resources to feed itself, it can get by on its own (North Korea comes to mind).
Individuals unable to compete in their national economy have none of that. Do you own enough land to feed yourself?
A counter argument is that nuclear arms brought unprecedented worldwide peace. If it's to be used as an analogy for AI, we should consider that the outcome isn't clear cut and lies in the eye of the beholder.
Sadly. Once students will get tested by LLMs, they will get woke questions. If they don't answer "right", they may get bad grades.
So they will be forced to swallow the ideology.
Though at least with that it would presumably be open in that you could do mock tests against the LLM and see how it reacted. I would probably never write politically incorrect stuff for a human academic examiner if I wanted to pass the exam.
These models will remain in the word vector similarity phase forever. Till the time we understand consciousness, we will not crack AGI and then it won't take brute forcing of large swaths of data, but tiny amounts.
So there is nothing to worry. These "apps" might be as popular as Excel, but will go no further.
Agreed. The AI of our day (the transformer + huge amounts of questionably acquired data + significant cloud computing power) has the spotlight it has because it is readily commoditized and massively profitable, not because it is an amazing scientific breakthrough or a significant milestone toward AGI, superintelligence, the benevolent Skynet or whatever.
The association with higher AI goals is merely a mixture of pure marketing and LLM company executives getting high on their own supply.
I read in Forbes about a construction company that used AI-related tech to manage the logistics and planning. They claimed that they were saving upwards of 20% of their costs because everything was managed more accurately. (Maybe they had little control boxes on their workers too; I don't know.)
The point I am trying to make is that the benefits of AI-related tech is likely to be quite pervasive and we should be looking at what corporations are actually doing. Sort of what this poem says:
For while the tired waves, vainly breaking /
Seem here no painful inch to gain, /
Far back through creeks and inlets making, /
Comes silent, flooding in, the main.
Being profitable is probably a matter of time and technology maturing. Think about the first Iphone, Windows 95, LCD/LEDs, etc.
The potential of a sufficiently intelligent agent, probably something very close to a really good AGI, albeit still not an ASI, could be measured in billions of billions of mostly inmediate return of investment. LLMs are already well into the definition of hard AI, there are already strong signs it could be somehow "soft AGI".
If by chance, you're the first to reach ASI, all the bets are down, you just won everything on the table.
Hence, you have this technology, LLM, then most of the experts in the field (in the world blabla), say "if you throw more data into it, it becames more intelligent", then you "just" assemble an AI team, and start training bigger, better LLMs, ASAP, AFAP.
More or less this is the reasoning behind the investments, sans accounting the typical pyramidal schemes of investments in hyped new stuff.
> Being profitable is probably a matter of time and technology maturing. Think about the first Iphone, Windows 95, LCD/LEDs, etc.
Juicero, tulips....
> then you "just" assemble an AI team, and start training bigger, better LLMs, ASAP, AFAP.
There's a limit to LLMs and we may have reached it. Both physical: there is not enough capacity in the world to train bigger models. And data-related: once you've gobbled up most of internet, movies and visual arts, there's an upper limit on how much better these models can become.
Probably. If you had shown ChatGPT to the LessWrong folks a decade ago, most would likely have called it AGI and said it was far to dangerous to share with the public, and that anyone who thought otherwise was a dangerous madman.
I don't feel that much has changed in the past 10 years. I would have done the same thing then as now, spent a month captivated by the crystal ball until I realized it was just refracting my words back at me.
> These models will remain in the word vector similarity phase forever. Till the time we understand consciousness, we will not crack AGI and then it won't take brute forcing of large swaths of data, but tiny amounts.
That is something I hear over and over, particularly as a rebuttal to the argument that llm is just a stochastic parrot. Calling it "good enough" doesn't mean anything, it just allows the person saying it to disengage from the substance of the debate. It's either reasons or it doesn't, and today it categorically does not.
That some will remark that you do not need consciousness to achieve reasoning does not lose truth because a subset sees in LLMs something that appears to them as reasoning.
I do not really understand who you are accusing of a «good enough» stance: we have never defined "consciousness" as a goal (cats are already there and we do not seem to need further), we just want something that reasons. (And that reasons excellently well.)
The apparent fact that LLMs do not reason does is drily irrelevant to an implementation of AGI.
The original poster wrote that understanding consciousness would be required to «crack AGI» and no, we state, we want AGI as a superhuman reasoner and consciousness seems irrelevant.
You build this system, a LLM, it uses a technical process that it outputs seemingly the same output that you - a human - could output, given a prompt.
You can do "reasoning" about a topic, the LLM can produce a very similar output to what you could, how do you name the output of the LLM?
Birds can fly, they do by a natural process. A plane also can fly, we do not "see" any difference between both things when we look at them flying in the air, far from the ground.
This is mostly it about LLM "doing reasoning" or not. Semantics. The output is same.
You could just name it otherwise, but it would be still the same output.
Philosophers have defined consciousness, why do people keep repeating that line? Your subjective sensations that make up perception, dreams, inner dialog, that sort of thing. Call it qualia, representations or correlations, but we all experience colors, sounds, tastes, pains, pleasures. We all probably dream, most of us visualize or have inner dialog. It's not that hard to define, it's only because of the ambiguity of the word where it's conflating whether other mental activity like being awake or being aware.
Nobody here is speaking of «good enough»: you are the one speaking of it, and the only one.
And nobody here is saying that LLMs would reason: you are the one attacking that idea that was not proposed here.
What you replied to said that calculators do not need consciousness to perform calculations. Reasoning is a special form of calculation. We are contented with reasoning, and when it will be implemented there will be no need for further different things for the applications we intend.
> Maybe "intelligence" is just enough statistics and pattern prediction, till the point you just say "this thing is intelligent".
Even the most stupid people can usually ask questions and correct their answers. LLMs are incapable of that. They can regurgitate data and spew a lot of generated bullshit, some of which is correct. Doesn't make them intelligent.
> Even the most stupid people can usually ask questions and correct their answers. LLMs are incapable of that. They can regurgitate data and spew a lot of generated bullshit, some of which is correct. Doesn't make them intelligent.
The way the current interface for most models works can result in this kind of output, the quality of the output - not even in the latests models - doesn't necessarily reflects the fidelity of the world model inside the LLM nor the level of insight it can have about a given topic ("what is the etymology of the word cat").
The current usual approach is "one shot", you've got one shot at the prompt, then return your output, no seconds thoughts allowed, no recursion at all. I think this could be a trade-off to get the cheapest most feasible good answer, mostly because the models get to output reasonably good answers most of the time. But then you get a percentage of hallucinations and made up stuff.
That kind of output could be - in a lab - fully absent actually. Did you you notice that the prompt interfaces never gives and empty or half-empty answer? "I don't know", "I don't know for sure", "I kinda know, but it's probably a bit shaky answer", or "I could answer this, but I'd need to google some additional data before", etc.
There's another one, almost never, you get to be asked back by the model, but the models can actually chat with you about complex topics related to your prompt. It's obvious when you're chatting with some chatbot, but not that obvious when you're asking it for a given answer for a complex topic.
In a lab, with recursion enabled, the models could get the true answers probably most of the time, including the fabulous "I don't know". And they could get the chance to ask back as an allowed answer, asking for additional human input, relaying on a live RHLF right there (it's quite technically feasible to achieve, not economically sound if you have a public prompt GUI facing the whole planet inputs).
but it wouldn't make much economic sense to make public a prompt interface like that.
I think it could also have a really heavy impact in the public opinion if they get to see a model that never makes a mistake, because it can answer "I don't know" or can ask you back to get some extra details about your prompt, so there you have another reason to do not make prompts that way.
> The current usual approach is "one shot", you've got one shot at the prompt, then return your output, no seconds thoughts allowed, no recursion at all.
We've had the models for a while and still no one has shown this mythical lab where this regurgitation machine reasons about things and makes no mistakes.
Moreover, since it already has so much knowledge stored, why does it still hallucinate even in specific cases where the answer is known, such as the case I linked?
>We've had the models for a while and still no one has shown this mythical lab where this regurgitation machine reasons about things and makes no mistakes.
It would be a good experiment to interact with the unfiltered, not-yet-RHLFed interfaces provided to the initial trainers (nigerian folks/gals?).
Or maybe the - lightly filtered - interfaces used privately in demos for CEOs.
So the claim that LLMs are intelligent is predicated on the belief that there are labs running unfiltered output and that there are some secret demos only CEOs see.
> These models will remain in the word vector similarity phase forever.
Forever? The same AI techniques are already being applied to analyze and understand images and video information after that comes ability to control robot hands and interact with the world and work on that is also ongoing.
> Till the tie we understand consciousness, we will not crack AGI …
We did not fully understand how bird bodies work yet that did not stop development of machines that fly. Why is an understanding of consciousness necessary to “crack AGI”?
No one is saying there is. Just that we've reached some big milestones recently which could help get us there even if it's only by increased investment in AI as a whole, rather than the current models being part of a larger AGI.
Imagine a system that can do DNS redirection, MITM, deliver keyloggers, forge authorizations and place holds on all your bank accounts, clone websites, clone voices, fake phone and video calls with people that you don’t see a lot. It can’t physically kill you yet but it can make you lose your mind which imo seems worse than a quick death
Why would all of these systems be connected to a single ai? I feel like you are describing something criminal humans do through social engineering, how do you foresee this AI finding itself in this position?
> Why would all of these systems be connected to a single ai?
because someone decides to connect them either unintentionally, or intentionally for personal gain, or, more likely, for corporate purposes which seem "reasonable" or "profitable" at the time, but the unintended consequences were not thought through.
Look at that recent article linked to HN about how MSFT allowed a huge security flaw in AD for years in order to not "rock the boat" and gain a large government contract. AI will be no different.
I foresee it in that position due to people building it as such. Perhaps the same criminal humans you mention, perhaps other actors with other motivations.
From a human welfare perspective this seems like worrying that a killer asteroid will make the 1% even richer because it contains goal if it can be safely captured.
I would not phrase that as a "bigger and more pressing" worry if we're not even sure if we can do anything about the killer asteroid at all.
> The "cold war" nuclear arms race, which brought the world to the brink of (at least partial) annihilation, is a good recent example.
The same era saw big achievements like first human in space, eradication of smallpox, peaceful nuclear exploration etc. It's good to be a skeptic but history does favor the optimists for the most part.
Were any of these big achievements side effects of creating nuclear weapons? If not, then they're not relevant to the issue.
I'm not saying nothing else good happened in the past 70 years, but rather that the invention of atomic weapons has permanently placed humanity in a position in which it had never been before: the possibility of wiping out much of the planet, averted only thanks to treaties, Stanislav Petrov[0], and likely other cool heads.
> Were any of these big achievements side effects of creating nuclear weapons? If not, then they're not relevant to the issue.
I think so, yes. Resources are allocated in the most efficient way possible, because there are multiple actors who have the same power. Everyone having nuclear weapons ensured that no one wanted a war between the big powers, so resources were allocated in other areas as the big powers tried to obtain supremacy.
Initially they allocated resources, a lot of them, into the race for space, the moon, etc. Once that was own by US after the moon landing, and after the Soviets were the first in space, there was no other frontier, and they discovered they couldn't obtain supremacy by just being space without further advancements in technology.
Instead they developed satellites, GPS and communications in order to obtain supremacy through "surveillance". Computing power and the affordability of personal computing, mobile phones, Internet and telecommunications was a result of the above.
I would argue that the presence of nukes increased rather than decreased military spending. since nuclear war was not an option, the nukes being only a deterrent, big powers had to continue investing heavily in their conventional forces in order to gain or keep an upper hand.
> Were any of these big achievements side effects of creating nuclear weapons?
The cold aspect of the Cold War was an achievement. Any doubt this was due to creation of nuclear weapons and the threat of their use?
How do you think countries will behave if every country faces being wiped out if it makes war on another country?
To prevent catastrophe I think teaching your citizens to hate other groups (as is done today due to national politics) will become dangerous and mental illness and extremist views will need to be kept in check.
The Cold War doesn’t mean there was no conflict. Both sides supported various proxy wars around the world. They just did not enter into _direct_ military conflict with each other. That may be because they had nukes but it could also be that direct conflict would mean massive casualties for either side and it’s not like either side wanted to destroy the other, just gain the upper hand globally.
So I for one don’t accept the argument that nukes acted as “peacekeepers”.
That proxy wars occurred during the Cold War, one can argue that these conflicts were actually a result of nuclear deterrence. Unable to engage directly due to the threat of mutually assured destruction, superpowers instead fought indirectly through smaller nations. This could be seen as evidence that nuclear weapons did prevent direct conflict between major powers. Also history shows that nations have engaged in extremely costly wars before. World War I and II saw unprecedented casualties, yet nations still fought. Nuclear weapons introduced a new level of destructive potential that went beyond conventional warfare. And there were periods of extreme tension, like the Cuban Missile Crisis, where nuclear war seemed imminent. The very existence of massive nuclear arsenals suggests that both sides were prepared for the possibility of mutual destruction.
You can question the role of nukes as peacekeepers but I think the case for nuclear deterrence keeping the peace is strong. Mutually Assured Destruction (MAD) is widely credited as a key factor in preventing direct conflict between superpowers during the Cold War. The presence of nuclear weapons forced them to pursue competition through economic, cultural, and technological means rather than direct military confrontation. “Race to the moon” being one such result.
Holy hell please knock on wood, this is the kinda comment that gets put in a museum in 10,000 years on The Beginning of the End of The Age of Hubris. We've avoided side effects from our new weapons for 80 years -- that does not exactly make me super confident it won't happen again!
In general, I think drawing conclusions about "history" from the past couple hundred years is tough. And unless you take a VERY long view, I don't see how one could describe the vast majority of the past as a win for the optimists. I guess suffering is relative, but good god was there a lot of suffering before modern medicine.
If anyone's feeling like we've made it through to the other side of the nuclear threat, "Mission Accomplished"-style, I highly recommend A Canticle for Lebowitz. It won a hugo award, and it's a short read best done with little research beforehand.
We'll see what the next 100 years or history brings. The nuclear war threat hasn't gone away either. There's always a chance those nukes get used at some point.
There will always be a factor of time in terms of able to utilize super intelligence to do your bidding and there is a big spectrum of things that can be achieved it it always starts small. The imagination is lazy when thinking about all the steps and inbetween + scenarios. In the time that super intelligence is found and used, there will be competing near super intelligences, as all forms of cutting edge models are likely commercial at first because that is where most scientific activities are at. Things very unlikely will go Skynet all of a sudden at first because humans at the control are not that stupid otherwise nuclear war would have us all killed by now and it’s been 50 years since invention
China can not win this race and I hate that this comment is going to be controversial among the circle of people that need to understand this the most. It is damn frightening that an authoritarian country is so close to number one in the race to the most powerful technology humanity has invented, and I resent people who push for open source AI for this reason alone. I don't want to live in a world where the first superintelligence is controlled by an entity that is threatened by the very idea of democracy.
I agree with your point. However I also don't want to live in a world where the first superintelligence is controlled by an entities that:
- try to scan all my chat messages searching for CSAM
- have black sites across the world where anyone can dissappear without any justice
- can require me to unlock my phone and give it away
- ... and so on
The point I'm trying to make is that other big players in the race are crooked as well and i'm waiting for a great horror for AGI to be invented as no matter who gets it - we are all doomed
Agreed. The U.S. has a horrible history (as do many countries), and many things I dislike, but its current iteration is much, much better than China's totalitarianism and censorship.
US is no angel and it cannot be the only one which wins the race. We have hard evidence of how monopoly power gets abused in the case of the US e.g. as the sole nuclear power, it used nukes on civilians.
We need every one to win this race to keep things on balance.
US has to win the race because while it's true that it's no angel, it isn't an authoritarian dictatorship and there isn't an equivalence in how bad the world will end up for you and me if the authoritarian side wins the race. Monopoly power will get abused the most by the least democratic actors, which is China. We need multiple actors within the US to win to balance power. We don't need or want China to be one of the winners. There is no upside for humanity in that outcome.
The US policymakers have figured this out with their chip export ban. Techies on the other hand, probably more than half the people here, are so naive and clueless about the reality of the moment we are in, that they support open sourcing this tech, the opposite of what we need to be doing to secure our future prosperity and freedom. Open source almost anything, just not this. It gives too much future power to authoritarians. That risk overwhelms the smaller risks that open sourcing is supposed to alleviate.
If anyone doubts this. Recent (<100y) leaders of China and Russia internally displaced and caused the death of large % of their population, for essentially fanciful ideological reasons.
I'm not American, but ~1850 is quite a long way back to go to make this point (especially when the US was quite a young country at the time). And it's small if you've comparing to the atrocities of other countries being discussed here (not that that excuses it!). Do any country histories remain pure with such a long timeline?
US is one of the very few countries that has been tested in a position of power over the world — speaking post-1945— and they've largely opened world trade and allowed most countries of the world to prosper, including allowing economic competitors to overtake their own (eg, Japanese car manufacturers, among many others). They have also not shown interest in taking territory, nor doing mass extermination. There are undeniable flaws and warts in the history, but they're quite marginal when compared to any other world power we've seen.
(*beware when replying to this that many people in the US only know their own country's flaws, not the abundant flaws of other countries — the US tends to be more reflective of its own issues and that creates the perspective of it being much worse than it actually is.).
I am puzzled how it is OK to kill people of other countries, but not your own.
US, China and Russia have all indulged in wanton mass killing of people. So that's an even keel for me.
The nuking of civilians and the abuse of supremacy post cold war show that the US cannot trusted to act morally and ethically in the absence of comparable adversaries. Possession of nukes by Russia and China clearly kept the US military adventures somewhat in check.
If it was liberal minded people like Deng as leader of China and Gorbachev as leader of Russia I would care a lot less and may even be in favor of open source despite their autocratic system. They'd be trending towards another Singapore at that point. Although I'd still be uneasy about it.
But I'm looking at the present moment and see too many similarities with the fascist dictatorships of the past. The nationalism, militarism, unreasonable border disputes and territorial claims and irredentist attitudes. The US just isn't that, despite their history.
> I am puzzled how it is OK to kill people of other countries, but not your own.
The former is rather universally regarded as regrettable, but sometims necessary: It's called "war". The latter, pretty much never.
Also, there are separate terms for slaying members of your own family, presumably because patri-, matri-, fratri- and infanticide are seen as even more egregious than "regular" homicide. Same concept, only expanded from person to populations -- from people to peoples -- seems to pretty much demand that killing your own population is seen as "less OK" than others.
Could you explain your point a bit more? You say you’re worried about them having a monopoly, but then say that’s why you don’t support open source models? Open models mean that no one has a monopoly, what am I not getting here?
Open sourcing benefits everyone equally. Given that the US is currently ahead, it's helping China to make gains relative to the US that would have been very difficult otherwise. It's leaking what should be state secrets without even needing the CCP to do the hard work of espionage.
What about Moroccans, or Argentinians, they shouldn't benefit from scientific advances because China is pissing off USA?
I guess it comes down to whether you think language models are at atom-bomb-levels of destructive capability. I don't see it, I think trying to keep tech and economic advances to ourselves is more likely to lead to war than a level playing field.
They should be allowed to have the same access to and benefit of AI as any regular American business or individual. That is, access through an API. It's the secret sauce that I believe should be kept under wraps as a state secret.
Should read *has brought*. As in the present perfect tense, since we are still on the brink of annihilation, more so than we have been at any time in the last 60 years.
The difference between then and now is that we just don't talk about it much anymore and seem to have tacitly accepted this state of affairs.
We don't know if that superintelligence will be safe or not. But as long as we are in the mix, the combination is unsafe. At the very least, because it will expand the inequality. But probably there are deeper reasons, things that make that combination of words an absurd. Or it will be abused, or the reason that it is not is that it wasn't so unsafe after all.
> At the very least, because it will expand the inequality.
It's a valid concern that AI technology could potentially exacerbate inequality, it's not a foregone conclusion. In fact, the widespread adoption of AI might actually help reduce inequality in several ways:
If AI technology becomes more affordable and accessible, it could help level the playing field by providing people from all backgrounds with powerful tools to enhance their abilities and decision-making processes.
AI-powered systems can make vast amounts of knowledge and expertise more readily available to the general public. This could help close the knowledge gap between different socioeconomic groups, empowering more people to make informed decisions and pursue opportunities that were previously out of reach.
As AI helps optimize resource allocation and decision-making processes across various sectors, it could lead to more equitable distribution of resources and opportunities, benefiting society as a whole.
The comparison to gun technology and its role in the rise of democracy is an interesting one. Just as the proliferation of firearms made physical strength less of a determining factor in power dynamics, the widespread adoption of AI could make raw intelligence less of a defining factor in success and influence.
Moreover, if AI continues to unlock new resources and opportunities, it could shift society away from a zero-sum mentality. In a world of abundance, the need for cutthroat competition diminishes, and collaboration becomes more viable. This shift could foster a more equitable and cooperative society, further reducing inequality.
The same arguments have been made about the internet and other technological advances, and yet, inequality has _grown_ sharply in the past 50 years. So no, "trickle down technologies", just like "trickle down economics", does not work.
The situation is nuanced. For example, in medicine, technological advancements have undeniably benefited people across all economic strata. Vaccines, antibiotics, and improved diagnostic tools have increased life expectancy and quality of life globally, including in developing nations. These benefits aren't limited to the wealthy; they've had a profound impact on public health as a whole.
> The same arguments have been made about the internet and other technological advances, and yet, inequality has _grown_ sharply in the past 50 years.
The internet has enabled remote work, online education, and access to information that was previously unavailable to many. Smartphones, once luxury items, are now widely available and have become essential tools for economic participation in many parts of the world.
> So no, "trickle down technologies", just like "trickle down economics", does not work.
It's crucial to distinguish between zero-sum and positive-sum dynamics. While relative wealth inequality has indeed grown, overall absolute global poverty has decreased significantly.
When a new technology or medicine is invented is everyone everywhere automatically entitled to it? Even if this slows down more such inventions? Because equality matters more than growth of overall prosperity? Would you prefer to be alive at a random time in history centuries ago, a random life where there is less technology and less inequality?
I'm not saying that the internet and technological advances have not benefitted humankind. They certainly have in the ways you described, and others.
But when it comes specifically to reducing economic inequality, they have not done that -- in fact, they have possibly exacerbated it.
Global poverty is a separate issue from economic inequality, and the gains there have been primarily from extremely low levels, primarily in China and India. In China this was driven by political change and also globalization that allowed China to become the world leader in manufacturing.
I would also put medical advances in a separate category than the internet and similar tech advances.
> I would also put medical advances in a separate category than the internet and similar tech advances.
Why? Medical advances are technology are they not?
> But when it comes specifically to reducing economic inequality, they (tech advances) have not done that -- in fact, they have possibly exacerbated it.
Yes technological advances do not necessarily reduce economic inequality, and may even increase it in some cases. However, this is a complex issue: While tech advances may exacerbate inequality, they often bring substantial overall benefits to society (e.g. improved healthcare, communication, productivity).
Technology isn't the only factor driving inequality. Other issues like tax policy, education access, and labor markets play major roles. Rather than suppressing innovation, there are ways to more equitably distribute its gains (Progressive taxation and wealth redistribution policies, Stronger social safety nets, Incentives for companies to share profits more broadly, …)
Notice also that most technologies increase inequality initially but lead to broader benefits over time as they become more accessible. Faster rate of innovation can make it look like this is not happening fast enough so yes economic gaps can grow.
> Global poverty is a separate issue from economic inequality, and the gains there have been primarily from extremely low levels, primarily in China and India.
While it's true that global poverty and economic inequality are distinct concepts, they are interconnected, especially when considering technological advancements.
> In China this was driven by political change and also globalization that allowed China to become the world leader in manufacturing.
Yes. China transitioned from a strictly communist "economic equality first" model to a more market-oriented "prosperity first" approach and lifted millions out of extreme poverty. Yes this contributed to increased economic inequality within many developed countries that have outsourced low-skill labor. But can we deny the substantial reduction in global suffering due to the alleviation of absolute poverty? Is this outcome worth the cost of increased domestic inequality in some countries? Should we prioritize the well-being of some populations over others based on arbitrary factors like nationality or ethnicity?
> It's a valid concern that AI technology could potentially exacerbate inequality, it's not a foregone conclusion.
No, but looking at how most technological advance throughout history as at least initially (and here I mean not "for the first few weeks", but "for the first few centutries") exacerbated inequality rather massively, it seems not far off.
> In fact, the widespread adoption of AI might actually help reduce inequality in several ways: ...
The whole tone of the rest your post feels frighteningly Pollyanna-ish.
Your comment was bleak so I supplied a counterpoint. My view is that new technology itself is not inherently unequal - it can widen or narrow gaps depending on how it is developed, regulated, and deployed.
> At the very least, because it will expand the inequality.
This is a distraction from the real danger.
> But probably there are deeper reasons, things that make that combination of words an absurd.
There are. If we look at ASI with the lens of Biology, the x-risk becomes obvious.
First to clear up a common misconception about humans: Many believe humanity has a arrived at a point where our evolution has ended. It has not, and in fact the rate of change of our genes is probably faster now than it has been for thousands if not 100s of thousands of years.
It's still slow compared to most events that we witness in our lives, though, which is what is fooling us.
For instance, we think we've brought overpopulation under control with contraceptives, family planning, social replacements for needing our children to take care of us when we get old.
That's fundamentally wrong. What we've done is similar to putting polar bears in zoos. We're in a situation where MOST OF US are no longer behaving in ways that lead to maximizing the number of offspring.
But we did NOT stop evolution. Any genes already in the gene pool that increase the expected number of offspring (especially for women) are no increasing in frequency as soon as evolutionarily possible.
That could be anything from genes that wire their heads to WANT to have children, CRAVE being around babies, to genes that block impulse control against getting pregnant, develop a phobia vs contraceptives or even to become more prone to being religious (as long as religions promote having kids).
If enough such genes exist, it's just a matter of time before we're back to the population going up exponentially. Give that enough time (without AI), and the desire to have more kids will be strong enough in enough of us that we will flood Earth with more humans that most people today are even possible. In such a world, it's unlikely that many other species of large land animals will make it.
Great apes, lions, elephants, wolves, deer, everyone will need to go to make room for more of us.
Even domestic animals eventually. If there are enough of us, we'll all be forced to become vegan (unless we free up space by killing each other).
If we master fusion, we may feed a trillion people using multi layer farming and artificial lighting.
Why do I begin with this? It's to defuse the argument that humans are "good", "empathetic", "kind" and "environmental". If we let weaker species live, so would AI, some think. But that argument misses the fact that we're currently extremely far from a natural equilibrium (or "state of nature").
The "goodness" beliefs that are currently common are examples of "luxury beliefs" that we can afford to hold because of the (for now) low birth rate.
The next misconception is to think of ASI as tools. A much more accurate analogy is to think of them as a new, alien species. If that species is subjected to Darwinian selection mechanisms, it will evolve in precisely the same way we'll probably do, given enough time.
Meaning, eventually it will make use of any amount of resources that it's capable of. In such a "state of nature" it will eradicate humanity in precisely the same way we will probably EVENTUALLY cause the extinction of chimps and elephants.
To believe in a future utopia where AGI is present alongside humanity is very similar to believe in a communist utopia. It ignores the reality behind incentive and adaptation.
Or rather, I think that outcome is only possible if we decide to build one or a low number of AI's that are NOT competing with each other, and where their abilities to mutate or self-improve is frozen after some limited number of generations.
If robots (hardware/self assembling factories/ resource gathering etc) are not involved this isnt likely a problem. You will know when these things form and will be crystal clear, but just having the model won’t do much when hardware is what really kills right now
How about this possibility: The good guys will be one step ahead, they will have more resources the bad guys will risk imprisonment if they misapply super intelligence. And this will be discovered and protected from by even better super intelligence.
Yup, well said. I think it's important to remember sometimes that Skynet was some sort of all-powerful military program -- maybe we should just, y'know, not do that part? Not even to win a war? That's the hope...
More generally/academically, you've pointed out that this covers only half of the violence problem, and I'd argue there's actually a whole other dimension at play bringing the total number of problem areas to four, of which this just the first:
Given it will initially be controlled by humans it seems inevitable they will make both good Mahatma Gandhi like and evil take over the world versions. I hope the good wins over the malware.
As a note, you used Gandhi as a personification of "good", and one day I did the same mistake; Gandhi is actually a quite controversial person, knowing for sleeping with young women, while telling their husbands that they shouldn't be around.
Indeed. I'd much rather someone like Altman does it who is shifty but can at least be controlled by the US government than someone like Putin who'd probably have it leverage their nuclear arsenal to try "denazify" planet like he's doing in Ukraine.
Glad to see Ilya is back in a position to contribute to advancing AI. I wonder how they are going to manage to pay the kinds of compensation packages that truly gifted AI researchers can make now from other companies that are more commercially oriented. Perhaps they can find people who are ideologically driven and/or are already financially independent. It's also hard to see how they will be able to access enough compute now that others are spending many billions to get huge new GPU data centers. You sort of need at least the promise/hope of future revenue in a reasonable time frame to marshall the kinds of resources it takes to really compete today with big AI super labs.
> compensation packages that truly gifted AI researchers can make now
I guess it depends on your definition of "truly gifted" but, working in this space, I've found that there is very little correlation between comp and quality of AI research. There's absolutely some brilliant people working for big names and making serious money, there's also plenty of really talented people working for smaller startups doing incredible work but getting paid less, academics making very little, and even the occasional "hobbyist" making nothing and churning out great work while hiding behind an anime girl avatar.
OpenAI clearly has some talented people, but there's also a bunch of the typical "TC optimization" crowd in there these days. The fact that so many were willing to resign with sama if necessary appears largely because they were more concerned with losing their nice compensation packages than any of their obsession with doing top tier research.
Two people I knew recently left Google to join OpenAI. They were solid L5 engineers on the verge of being promoted to L6, and their TC is now $900k. And they are not even doing AI research, just general backend infra. You don't need to be gifted, just good. And of course I can't really fault them for joining a company for the purpose of optimizing TC.
As a community we should stop throwing numbers around like this when more than half of this number is speculative. You shouldn't be able to count it as "total compensation" unless you are compensated.
Word in town is [1] openai "plans" to let employees sell "some" equity through a "tender process" which ex-employees are excluded from; and also that openai can "claw back" vested equity, and has used the threat of doing so in the past to pressure people into signing sketchy legal documents.
I would definitely discount OpenAI equity compared to even other private AI labs (i.e. Anthropic) given the shenanigans, but they have in fact held 3 tender offers and former employees were not, as far as we know, excluded (though they may have been limited to selling $2m worth of equity, rather than $10m).
> Word on town is OpenAI folks heavily selling shares in secondaries in 100s of millions
OpenAI heavily restricts the selling of its "shares," which tends to come with management picking the winners and losers among its ESOs. Heavily, heavily discount an asset you cannot liquidate without someone's position, particularly if that person is your employer.
When I looked into it and talked to some hiring managers, the big names were offering cash comp similar to total comp for big tech, with stock (sometimes complicated arrangements that were not options or RSUs) on top of that. I’m talking $400k cash for a senior engineer with equity on top.
Because op’s usage of base implies base + stock. including a place where base = total comp is really misleading and is just being unnecessarily pedantic about terminology.
OP is correct that a base cash of 400k is truly rare if you’re talking about typical total comp packages where 50% is base and 50% is stock.
I don’t know what point you’re trying to make other than being super pedantic. This was a discussion about how OpenAI’s base of 400k is unique within the context of a TCO in the 800-900k range. It is. That quantfi and Netflix offer similar base because that’s also their TCO is a silly argument to make.
> This was a discussion about how OpenAI’s base of 400k is unique within the context of a TCO in the 800-900k range.
That's not how I interpret the conversation.
I see a claim that 900k is a BS number, a counterargument that many big AI companies will give you 400k of that in cash so the offers are in fact very hot, then a claim that only finance offers 400k cash, and a claim that netflix offers 400k cash.
I don't see anything that limits these comparisons to companies with specific TCOs.
Even if the use of the word "base" is intended to imply that there's some stock, it doesn't imply any particular amount of stock. But my reading is that the word "base" is there to say that stock can be added on top.
You're the one being pedantic when you insist that 400k cash is not a valid example of 400k cash base.
Notice how the person being replied to looked at the Netflix example and said "Okay that's true". They know what they meant a lot better than you do.
ok so the conversation starts out with 900k TCO with 400k in cash, a claim that that’s BS and then morphs into a discussion about a TCO of 400k all cash being an example of equivalent compensation to OpenAI packages?
Nobody said it was equivalent. The subdiscussion was about whether you can even get that much cash anywhere else, once TCO got pulled apart into cash and stock to be compared in more detail.
Again, the person that made the original claim about where you can get "400k cash base" accepted the Netflix example. Are you saying they're wrong about what they meant?
It's amazing to me how many people are willing to just say the first thing that comes to their head while knowing they can be fact-checked in a heartbeat.
> Note at offer time candidates do not know how many PPUs they will be receiving or how many exist in total. This is important because it’s not clear to candidates if they are receiving 1% or 0.001% of profits for instance. Even when giving options, some startups are often unclear or simply do not share the total number of outstanding shares. That said, this is generally considered bad practice and unfavorable for employees. Additionally, tender offers are not guaranteed to happen and the cadence may also not be known.
> PPUs also are restricted by a 2-year lock, meaning that if there’s a liquidation event, a new hire can’t sell their units within their first 2 years. Another key difference is that the growth is currently capped at 10x. Similar to their overall company structure, the PPUs are capped at a growth of 10 times the original value. So in the offer example above, the candidate received $2M worth of PPUs, which means that their capped amount they could sell them for would be $20M
> The most recent liquidation event we’re aware of happened during a tender offer earlier this year. It was during this event that some early employees were able to sell their profit participation units. It’s difficult to know how often these events happen and who is allowed to sell, though, as it’s on company discretion.
Definitely true of even normal software engineering; my experience has been the opposite of expectations, that TC-creep has infected the industry to an irreparable degree and the most talented people I've ever worked around or with are in boring, medium-sized enterprises in the midwest US or australia, you'll probably never hear of them, and every big tech company would absolutely love to hire them but just can't figure out the interview process to weed them apart from the TC grifters.
TC is actually totally uncorrelated with the quality of talent you can hire, beyond some low number that pretty much any funded startup could pay. Businesses hate to hear this, because money is easy to turn the dial up on; but most have no idea how to turn the dial up on what really matters to high talent individuals. Fortunately, I doubt Ilya will have any problem with that.
I have also worked in multiple different sized companies, including FAANG, and multiple countries. My assessment is that FAANGs tend to select for generally intelligent people who can learn quickly and adapt to new situations easily but who nowadays tend to be passionless and indifferent to anything but money and prestige. Personally I think passion is the differentiator here, rather than talent, when it comes to doing a good job. Passion means caring about your work and its impact beyond what it means for your own career advancement. It means caring about building the best possible products where “best” is defined as delivering the most value for your users rather than the most value for the company. The question is whether big tech is unable to select for passion or whether there are simply not enough passionate people to hire when operating at FAANG scale. Most likely it’s the latter.
So I guess I agree with both you and the parent comment somewhat in that in general the bar is higher at FAANGs but at the same time I have multiple former colleagues from smaller companies who I consider to be excellent, passionate engineers but who cannot be lured to big tech by any amount of money or prestige (I’ve tried). While many passionless “arbitrary metric optimizers” happily join FAANGs and do whatever needs to be done to climb the ladder without a second thought.
I sort of agree and disagree. I wouldn't agree with the idea that most FAANG engineers are not passionate by nature about their work.
What I would say is that the bureaucracy and bullshit one has to deal with makes it hard to maintain that passion and that many end up as TC optimizers in the sense that they stay instead of working someplace better for less TC.
That said, I am not sure how many would make different choices. Many who join a FAANG company don't have the slightest inkling of what it will be like and once they realize that they are tiny cog in a giant machine it's hard to leave the TC and perks behind.
Academic compensation is different than what you’d find elsewhere on Hacker News. Likewise, academic performance is evaluated differently than what you’d expect as a software engineer. Ultimately, everyone cares about scientific impact so academic compensation relies on name and recognition far more than money. Personally, I care about the performance of the researchers (i.e., their publications), the institution’s larger research program (and their resources), the institution’s commitment to my research (e.g., fellowships and tenure). I want to do science for my entire career so I prioritize longevity rather than a quick buck.
I’ll add, the lack of compute resources was a far worse problem early in the deep learning research boom, but the market has adjusted and most researchers are able to be productive with existing compute infrastructure.
But wouldn't the focus on "safety first" sort of preclude them from giving their researchers the unfettered right to publish their work however and whenever they see fit? Isn't the idea to basically try to solve the problems in secret and only release things when they have high confidence in the safety properties?
If I were a researcher, I think I'd care more about ensuring that I get credit for any important theoretical discoveries I make. This is something that LeCun is constantly stressing and I think people underestimate this drive. Of course, there might be enough researchers today who are sufficiently scared of bad AI safety outcomes that they're willing to subordinate their own ego and professional drive to the "greater good" of society (at least in their own mind).
If you're working on superintelligence I don't think you'd be worried about not getting credit due to a lack of publications, of all things. If it works, it's the sort of thing that gets you in the history books.
Not sure about that. It might get Ilya in the history books, and maybe some of the other high profile people he recruits early on, but a junior researcher/developer who makes a high impact contribution could easily get overlooked. Whereas if that person can have their name as lead author on a published paper, it makes it much easier to measure individual contributions.
There is a human cognitive limit to the detail in which we can analyze and understand history.
This limit, just like our population count, will not outlast the singularity. I did the math a while back, and at the limit of available energy, the universe has comfortable room for something like 10^42 humans. Every single one of those humans will owe their existence to our civilization in general and the Superintelligence team in specific. There'll be enough fame to go around.
At the end game, a "non-safe" superinteligence seems easier to create, so like any other technology, some people will create it (even if just because they can't make it safe). And in a world with multiple superintelligent agents, how can the safe ones "win"? It seems like a safe AI is at inherent disadvantage for survival.
The current intelligences of the world (us) have organized their civilization in a way that the conforming members of society are the norm and criminals the outcasts. Certainly not a perfect system, but something along those lines for the most part.
I disagree that civilization is organized along the lines of conforming and criminals. Rather, I would argue that the current intelligences of the world have primarily organized civilization in such a way that a small percentage of its members control the vast majority of all human resources, and the bottom 50% control almost nothing[0]
I would hope that AGI would prioritize humanity itself, but since it's likely to be created and/or controlled by a subset of that same very small percentage of humans, I'm not hopeful.
That suggests that there are scenarios under which we survive. I'm not sure we'd like any of them, though "benign neglect" might be the best of a bad lot.
Generally, the mindset that makes the best engineers is an obsession with solving hard problems. Anecdotally, there's not a lot of overlap between the best engineers I know and the best paid engineers I know. The best engineers I know are too obsessed with solving problems to be sidetracked the salary game. The best paid engineers I know are great engineers, but the spend a large amount of time playing the salary game, bouncing between companies and are always doing the work that looks best on a resume, not the best work they know how to do.
Great analysis, but you're missing two key factors IMO:
1. People who honestly think AGI is here aren't thinking about their careers in the typical sense at all. It's sorta ethical/"ideological", but it's mostly just practical.
2. People who honestly think AGI is here are fucking terrified right now, and were already treating Ilya as a spiritual center after Altman's coup (quite possibly an unearned title, but oh well, that's history for ya). A rallying cry like this -- so clearly aimed at the big picture instead of marketing they don't even need CSS -- will be seen as a do-or-die moment by many, I think. There's only so much of "general industry continues to go in direction experts recommend against; corporate consolidation continues!" headlines an ethical engineer can take before snapping and trying to take on Goliath, odds be damned
My guess is they will work on a protocol to drive safety with the view that every material player will use / be regulated and required to use that could lead to a very robust business model
I assume that OpenAI and others will support this effort and the comp / training / etc and they will be very well positioned to offer comparable $$$ packages, leverage resources, etc
Are you seriously asking how the most talented AI researcher of the last decade will be able to recruit other researchers? Ilya saw the potential of deep learning way before other machine learning academics.
AI researchers at top firms make significantly more than software engineers at the same firms though (granted that the difference is likely not an order of magnitude in this case though).
Unless you know something I don’t, that’s not the case. It also makes sense, engineers are far more portable and scarcity isn’t an issue (many ML PhDs find engineering positions).
That is incredibly untrue and has been for years in the AI/ML space at many startups and at Amazon, Google, Facebook, etc. Good ML researchers have been making a good amount more for a while (source: I've hired both and been involved in leveling and pay discussions for years)
As others have pointed out, it's the business incentives that create unsafe AI, and this doesn't solve that. Social media recommendation algorithms are already incredibly unsafe for society and young people (girls in particular [1]).
When negative externalities exist, government should create regulation that appropriately accounts for that cost.
I understand there's a bit of a paradigm shift and new attack vectors with LLMs etc. but the premise is the same imo.
Even without business incentives, the military advantages of AI would inventivize governments to develop it anyway, like they did with nuclear weapons. Nuclear weapons are inherently unsafe, there are some safeguards around them, but they are ultimately dangerous weapons.
If someone really wanted to use nukes, they would have been used by now. What has protected us is not technology (in the aftermath of the USSR it wasn't that difficult to steal a nuke), but rather lack of incentives. A bad actor doesn't have much to gain by detonating a nuke (unless they're deranged and want to see people die for the pleasure of it). OK, you could use it as blackmail, which North Korea essentially tried, but that only got them so far. Whereas a super AI could potentially be used for great personal gain, i.e., to gain extreme wealth and power.
So there's much greater chance of misuse of a "Super AI" than nuclear weapons.
Sure, that just makes the military incentives to develop such a thing even stronger. All I mean is that business incentives don't really come into it, as long as there is competition, someone's going to want to build weapons to gain advantage, whether it's a business or a government.
The very existence of dangerous weapons, be they nukes or handguns (or swords or tanks), makes the world less of a safe place than if they didn't exist. Existence is pretty much the most inherent attribute anything can have, AFAICS, so yes: Dangerous weapons are inherently unsafe. (Take that Glock, for example -- that's exactly the problem, that someone can take it. Might be someone who knows how to remove the safety.)
> makes the world less of a safe place than if they didn't exist
This is false. Handguns and other small arms, for example, cause a reduction in violence as they proliferate through society, due to the fact that the use of force is not the exclusive purview of the physically strong.
There's a reason police in most places carry guns, and it's not to shoot people; it's to keep people from punching the cops in the face. Nuclear weapons have vastly (and I do mean vastly) reduced mass deaths in world wars since their invention.
> This is false. Handguns and other small arms, for example, cause a reduction in violence as they proliferate through society, due to the fact that the use of force is not the exclusive purview of the physically strong.
[Citation needed]
> There's a reason police in most places carry guns, and it's not to shoot people; it's to keep people from punching the cops in the face.
By the threat of getting shot – a threat which the cops sometimes follow through on. Handguns “make the world a safer place” for those who have the handguns... By making it a less safe place for those who don't.
> Nuclear weapons have vastly (and I do mean vastly) reduced mass deaths in world wars since their invention.
I mean if the last 20 years is to be taken as evidence, it seems big tech is more than happy to shotgun unproven and unstudied technology straight into the brains of our most vulnerable populations and just see what the fuck happens. Results so far include a lot of benign nothing but also a whole lot of eating disorders, maxed out parents credit cards, attention issues, rampant misogyny among young boys, etc. Which, granted, the readiness to fuck with populations at scale and do immeasurable harm doesn't really make tech unique as an industry, just more of the same really.
But you know, we'll feed people into any kind of meat grinder we can build as long as the line goes up.
i am very skeptical of narratives saying that young boys or men are more misogynistic than in the past. we have a cognitive bias towards thinking the past is better than it was, but specifically on gender issues i just do not buy a regression
> i am very skeptical of narratives saying that young boys or men are more misogynistic than in the past.
They don't even have to be more misogynistic than in the past for there to be a detrimental effect. Because so many other things in society -- pre-school and school environments, upbringing by more enlightened parents than what those parents themselves had, etc etc[1] -- cooperate to make young men and boys less misogynistic than in the past... But maybe that effect would be even more pronounced without shitty influences from social media; don't you think it's possible that all the online crap inhibits the good influences, so without it the kids would be even better?
[1]: From comparing my sons' generation to what I remember of my own youth and childhood.
Maybe in the US context. In the global context I think social media is almost certainly net anti-sexist compared to the attitudes parents are passing down.
I mean, I don't know if it's better or worse than it was. I do know that it's bad, thanks to tons of studies on the subject covering a wide range of little kids who watch shitheads like Andrew Tate, Fresh & Fit, etc. Most grow out of it, but speaking as someone who did, I would be a much better and happier person today if I was never exposed to that garbage in the first place, and it's resulted in stunted social skills I am still unwinding from in my thirties.
This shit isn't funny, it's mental poison and massive social media networks make BANK shoving it front of young men who don't understand how bad it is until it's WAY too late. I know we can't eliminate every kind of shithead from society, that's simply not possible. But I would happily settle for a strong second-place achievement if we could not have companies making massive profits off of destroying people's minds.
Blaming the internet for misogyny is kind of bizarre, given that current levels of misogyny are within a couple points of all-time historical lows. The internet was invented ~40 years ago. Women started getting vote ~100 years ago. Do you think the internet has returned us to pre-women's-suffrage levels of misogyny?
> Do you think the internet has returned us to pre-women's-suffrage levels of misogyny?
Well in the States at least we did just revoke a sizable amount of their bodily autonomy so, the situation may not be that bad, yet, but I wouldn't call it good by any measurement. Any my objection isn't "that sexism exists in society," that is probably going to be true as a statement until the sun explodes, and possibly after that if we actually nail down space travel as a technology and get off this particular rock. My issue is massive corporations making billions of dollars facilitating men who want to spread sexist ideas, and paying them for the pleasure. That's what I have an issue with.
Be whatever kind of asshole you see fit to be, the purity of your soul is no one's concern but yours, and if you have one, whatever god you worship. I just don't want you being paid for it, and I feel that's a reasonable line to draw.
I am firmly in favor of abortion rights but still I do not think that is even remotely a good bellwether to measure sexism/misogyny.
1. Women are more likely than men to be opposed to abortion rights.
2. Many people who are opposed to abortion rights have legitimately held moral concerns that are not simply because they have no respect for women's rights.
3. Roe v. Wade was the decision of 9 people. It absolutely did not reflect public opinion at the time - nothing even close to as expansive would possibly have passed in a referendum in 1974. Compare that to now, where multiple states that are known abortion holdouts have repealed abortion restrictions in referenda - and it is obvious that people are moving to the left on this issue compared to where we were in 1974.
Social media facilitates communication. As long as there is sexism and freedom of communication, there will be people making money off of facilitating sexist communication because there will be people making money off of facilitating communication writ large. It's like blaming a toll highway for facilitating someone trafficking drugs. They are also making money off of facilitating anti-sexist communication - and the world as a whole is becoming less sexist, partially in my view due to the spread of views facilitated by the internet.
Right. We can see something similar with the Terry Schiavo case or the opposition to IFV right now. It's clear that this is about different opinions regarding what should be considered a living human being (which doesn't seem to have a very clean definition for anyone, it should be noted). Depending on where you draw the line, it's either horrible to outlaw abortions or horrible to allow them.
Framing it as simply taking away a woman's bodily autonomy is like framing circumcision as simply being about mutilating men.
> 2. Many people who are opposed to abortion rights have legitimately held moral concerns that are not simply because they have no respect for women's rights.
Then they are free to not get an abortion. I don't get an abortion every day, it's pretty easy to accomplish. They do not get to use the letter of law to interfere in other people's medical decisions and care, and they most definitely should not have the ability to use the letter of the law to get unhealthy women killed to suit their precious morals.
Like, genuinely, if you are near a woman who is having a serious medical condition where her baby is killing her, there is no, and I repeat, NO version of that where letting an adult, alive, otherwise viable person die in the hopes that the clump of cells killing her might make it. That does not make sense under any moral system at all. We don't even take organs from recently deceased people without their permission before they croak, and some people think they have the right to demand someone lay down their entire actual life in the hope a baby MIGHT be born and live? Fuck that. Stupid.
> 3. Roe v. Wade was the decision of 9 people.
Sod public opinion. The public is wrong all the goddamn time. One would argue they're wrong more often than they aren't, and the more of em there are, and the louder they are, the more likely they're fucking wrong.
> Social media facilitates communication. As long as there is sexism and freedom of communication, there will be people making money off of facilitating sexist communication because there will be people making money off of facilitating communication writ large.
This is such a defeatist attitude. There will also always be revenge porn, child abuse material, beheading videos and people putting monkeys in blenders. Do we allow that everywhere too then? Since we cannot guarantee total blackout on objectionable content, we just wild west it? Fucking nonsense. We decide constantly by way of moderation on every service and website that exists what is permitted, and what is not, and there is no reason at all that those same things cannot be enshrined in law, with steep penalties for services that fuck up and host it.
it appears you want to debate the abortion issue on the merits when i’ve already said i agree with you. my point about public opinion was only in reference to public opinion being a gauge of general sexist attitudes and the degree to which being anti-abortion is intrinsically out of sexist motivations vs other differing beliefs.
on your second point, much of the material you’re describing is actively illegal - which is a different case. i agree with your point around moderation but feel conflicted with my intuition that the rules impacting speech should generally be publicly known and generally expansive. i am not sure what the reconciliation is. but i also don’t really know who andrew tate is, so can only really speak in the abstract
Generally it seems from polls I had seen that the % of women favoring illegality is higher than men [0][1]. But it seems that there has been a considerable shift in women's opinions post-pandemic which perhaps tracks in the US with broader gender partisan shifts (men & women becoming politically delinked, perhaps due to algo social media/tiktok) as well as Dobbs being a wakeup call for some women.
Interesting find on Gallup. My read is that the genders are broadly similar, but women have more variance/more nuanced opinions, presumably from thinking about this more.
In general it can. In this specific case, I really struggle to see even a single dimension in which young boys are more misogynistic now than they were 2 decades ago. The original comment mentions Andrew Tate - in the early 2000s there was an entire genre of Andrew Tates called "pickup artists".
Please look up the history of maxing out credit cards, eating disorders, attention disorders, and misogyny. You seem to be under the mistaken impression that anything before your birth was the Garden of Eden and that the parade of horribles existed only because of "big tech". What is next? Blaming big tech for making teenagers horny and defiant?
> You seem to be under the mistaken impression that anything before your birth was the Garden of Eden and that the parade of horribles existed only because of "big tech"
Please point out where I said that. Because what I wrote was:
> I mean if the last 20 years is to be taken as evidence, it seems big tech is more than happy to shotgun unproven and unstudied technology straight into the brains of our most vulnerable populations and just see what the fuck happens. Results so far include a lot of benign nothing but also a whole lot of eating disorders, maxed out parents credit cards, attention issues, rampant misogyny among young boys, etc. Which, granted, the readiness to fuck with populations at scale and do immeasurable harm doesn't really make tech unique as an industry, just more of the same really.
Which not only is not romanticizing the past, in fact I directly point out that making tons of people's lives worse for profit was a thing in industry long before tech came along, but also do not directly implicate tech as creating sexism, exploiting people financially, or fucking up young women's brains any differently, simply doing it more. Like most things with tech, it wasn't revolutionary new social harms, it was just social harms delivered algorithmically, to the most vulnerable, and highly personalized to what they are acutely vulnerable to in specific.
That is not a new thing, by any means, it's simply better targeted and more profitable, which is great innovation providing you lack a conscience and see people as only a resource to be exploited for your own profit, which a lot of the tech sector seems to.
> for society and young people (girls in particular [1]).
I don't think the article with a single focused example bears that out at all.
From the article:
> "Even more troubling are the men who signed up for paid subscriptions after the girl launched a program for super-fans receive special photos and other content."
> "Her mom conceded that those followers are “probably the scariest ones of all.”"
I'm sorry.. but what is your daughter selling, exactly? And why is social media responsible for this outcome? And how is this "unsafe for society?"
This just sounds like horrific profit motivated parenting enabled by social media.
One example is all I need although I know there would be more to find if I had the time.
One is enough because the fact is, this just simply shouldn’t be possible. Not only does Meta allow it though, it produces a system that incentivises it.
> I'm sorry.. but what is your daughter selling, exactly?
Did.. did you just post “I’d have to see these images of the preteen girl before I could form an opinion about whether or not the men buying and sharing them were creeps” as a rebuttal to an article about social media enabling child predators?
Well, I already have an opinion on them. The article seems to purposefully avoid describing what the market and the service here is. There are a limited number of venues where a preteen girl can credibly earn money on social media. Also, if she's earning that money, I'd openly wonder whether a Coogan Account is appropriate and if one exists here.
Anyways.. my shock was really over the fact that the mother is basically saying "I want to sell access to my daughter online but I'm surprised that the biggest spenders are adult men with questionable intentions." Did she genuinely believe that the target market was other 12 year old girls willing to pay to watch another 12 year old girl? The parents resignation over the situation in deference to the money is also disgusting.
> about social media enabling child predators?
That's my point. The article entirely fails to do that. It's one case with a questionable background and zero investigation over the claim, which you'd expect, because the crime statistics show the exact opposite.
That was the impression I got from your previous post.
> There are a limited number of venues where a preteen girl can credibly earn money on social media.
… ???
> my shock was really over the fact that the mother is basically saying "I want to sell access to my daughter online but I'm surprised that the biggest spenders are adult men with questionable intentions."
This article is about how Instagram enables [sometimes paid!] access to children. It is good that we both agree that that is what happened here, on Instagram, in this case. You also agree that the people buying access to this child, in this case, on Instagram, are adult men.
Somehow you have an issue with the mother and the child doing something but in the same breath say
> The article entirely fails to do that. [“That” being enabling child predators]
If Instagram didn’t facilitate access to child predators then… what happened here?
And finally (this is a question for literally any person reading this other than akira2501) how did you read this article and arrive at “I should post that I would want to see the pictures before I render judgment about the platform”? If it is uncharitable to notice that that’s a weird post, how do you interpret that specific point?
> Instagram’s algorithms have steered men with deviant sexual attraction to children to the girl’s page, flooding it with unwanted comments and come-ons, according the Journal.
It is a normal response to think “I wonder what was so attractive about the images that the child made that these men were forced to follow and comment on them. I will post about how we shouldn’t judge the men or platform until we’ve seen the images”
> We are an American company with offices in Palo Alto and Tel Aviv, where we have deep roots and the ability to recruit top technical talent.
The irony of pitching "safe super intelligence" while opening offices in a country that is suspected of using AI to bomb a population trapped in a ghetto, as reported by their own investigative journalists: https://www.972mag.com/lavender-ai-israeli-army-gaza/
Even without an official confirmation whether it is indeed a genocide or not, there has been enough evidence of horrible crimes that should dissuade a company from operating in Israel. If you really want to work with talented (no doubt) Israeli researchers, just bring them over to the US or elsewhere.
Ilya's issue isn't developing a Safe AI. Its developing a Safe Business. You can make a safe AI today, but what happens when the next person is managing things? Are they so kindhearted, or are they cold and calculated like the management of many harmful industries today? If you solve the issue of Safe Business and eliminate the incentive structures that lead to 'unsafe' business, you basically obviate a lot of the societal harm that exists today. Short of solving this issue, I don't think you can ever confidently say you will create a safe AI and that also makes me not trust your claims because they must be born from either ignorance or malice.
> You can make a safe AI today, but what happens when the next person is managing things?
The point of safe superintelligence, and presumably the goal of SSI Inc., is that there won't be a next (biological) person managing things afterwards. At least none who could do anything to build a competing unsafe SAI. We're not talking about the banal definition of "safety" here. If the first superintelligence has any reasonable goal system, its first plan of action is almost inevitably going to be to start self-improving fast enough to attain a decisive head start against any potential competitors.
Trouble is, in practice what you would need to do might be “turn off all of Google’s datacenters”. Or perhaps the thing manages to secure compute in multiple clouds (which is what I’d do if I woke up as an entity running on a single DC with a big red power button on it).
The blast radius of such decisions are large enough that this option is not trivial as you suggest.
a) after you create the superintelligence is likely too late. You seem to think that inventing superintelligence means that we somehow understand what we created, but note that we have no idea how a simple LLM works, let alone an ASI that is presumably 5-10 OOM more complex. You are unlikely to be able to control a thing that is way smarter than you, the safest option is to steer the nature of that thing before it comes into being (or, don’t build it at all). Note that we currently don’t know how to do this, it’s what Ilya is working on. The approach from OpenAI is roughly to create ASI and then hope it’s friendly.
b) except that is not how these things go in the real world. What actually happens is that initially it’s just a risk of the agent going rogue, the CEO weighs the multi-billion dollar cost vs. some small-seeming probability of disaster and decides to keep the company running until the threat is extremely clear, which in many scenarios is too late.
(For a recent example, consider the point in the spread of Covid where a lockdown could have prevented the disease from spreading; likely somewhere around tens to hundreds of cases, well before the true risk was quantified, and therefore drastic action was not justified to those that could have pressed the metaphorical red button).
> Having arms and legs is going to be a significant benefit for some time yet
I am also of this opinion.
However I also think that the magic shutdown button needs to be protected against terrorists and ne'er-do-wells, so is consequently guarded by arms and legs that belong to a power structure.
If the shutdown-worthy activity of the evil AI can serve the interests of the power structure preferentially, those arms and legs will also be motivated to prevent the rest of us from intervening.
So I don't worry about AI at all. I do worry about humans, and if AI is an amplifier or enabler of human nature, then there is valid worry, I think.
Where can I find the red button that shuts down all Microsoft data centers, all Amazon datacenters, all Yandex datacenters and all Baidu datacenters at the same time? Oh, there isn't one? Sorry, your superintelligence is in another castle.
It's been more than a decade now since we first saw botnets based on stealing AWS credentials and running arbitrary code on them (e.g. for crypto mining) - once an actual AI starts duplicating itself in this manner, where's the big red button that turns off every single cloud instance in the world?
Is that really "a lot of assumptions" that a piece of software can clone itself? We've been cloning and porting software from system to system for over 70 years (ENIAC was released in 1946 and some of its programs were adapted for use in EDVAC in 1951) - why would it be a problem for a "super intelligence"?
And even if it was originally designed to run on some really unique ASIC hardware, by the Church–Turing thesis it can be emulated on any other hardware. And again, if it's a "super intelligence", it should be at least as good at porting itself as human engineers have been for the three generations.
A "state of the art" system would almost by definition be running on special and expensive hardware. But I have llama3 running on my laptop, and it would have been considered state of the art less than 2 years ago.
A related point to consider is that a superintelligence should be considered a better coder than us, so the risk isn't only directly from it "copying" itself, but also from it "spawning" and spreading other, more optimized (in terms of resources utilization) software that would advance its goals.
This is why I think it’s more important we give AI agents the ability to use human surrogates. Arms and legs win but can be controlled with the right incentives
> there won't be a next (biological) person managing things afterwards. At least none who could do anything to build a competing unsafe SAI
This pitch has Biblical/Evangelical resonance, in case anyone wants to try that fundraising route [1]. ("I'm just running things until the Good Guy takes over" is almost a monarchic trope.)
The safe business won’t hold very long if someone can gain a short term business advantage with unsafe AI. Eventually government has to step in with a legal and enforcement framework to prevent greed from ruining things.
It's possible that safety will eventually become the business advantage, just like privacy can be a business advantage today but wasn't taken so seriously 10-15 years ago by the general public.
This is not even that far-fetched. A safe AI that you can trust should be far more useful and economically valuable than an unsafe AI that you cannot trust. AI systems today aren't powerful enough for the difference to really matter yet, because present AI systems are mostly not yet acting as fully autonomous agents having a tangible impact on the world around them.
Government is controlled by the highest bidder. I think we should be prepared to do this ourselves by refusing to accept money made by unsafe businesses, even if it means saying goodbye to the convenience of fungible money.
Banding together and refusing to accept harmful money is indeed akin to creating a government, and would indeed be more effective at controlling people's behavior.
But participation would be voluntary, and the restriction of harmful behavior would apply to it's enemies, not its citizens. So I'm not quite sure what the problem is.
Replace government with collective society assurance that no one cheats so we aren’t all doomed. Otherwise, someone will do it, and we all will have to bear the consequences.
If only enough individuals are willing to buy these services, then again we all will bear the consequences. There is no way out of this where libertarian ideals can be used to come to a safe result. What makes this even a more wicked problem is that decisions made in other countries will affect us all as well, we can’t isolate ourselves from AI policies made in China for example.
No, which makes this an even harder problem. Can US companies bound by one set of rules compete against Chinese ones bound by another set of rules? No, probably not. Humanity will have to come together on this, or someone will develop killer AI that kills us all.
I'd love to see more individual researchers openly exploring AI safety from a scientific and humanitarian perspective, rather than just the technical or commercial angles.
> Our singular focus means no distraction by management overhead or product cycles, and our business model means safety, security, and progress are all insulated from short-term commercial pressures.
This tells me enough about why sama was fired, and why Ilya left.
Is safe AI really such a genie out of the bottle problem? From a non expert point of view a lot of hype just seems to be people/groups trying to stake their claim on what will likely be a very large market.
A human-level AI can do anything that a human can do (modulo did you put it into a robot body, but lots of different groups are already doing that with current LLMs).
Therefore, please imagine the most amoral, power-hungry, successful sociopath you've ever heard of. Doesn't matter if you're thinking of a famous dictator, or a religious leader, or someone who never got in the news and you had the misfortune to meet in real life — in any case, that person is/was still a human, and a human-level AI can definitely also do all those things unless we find a way to make it not want to.
We don't know how to make an AI that definitely isn't that.
We also don't know how to make an AI that definitely won't help someone like that.
Anything except tasks that require having direct control of a physical body.
Until fully functional androids are developed, there is a lot a human-level AI can't do.
I think there's usually a difference between human-level and super-intelligent in these conversations. You can reasonably assume (some day) a superintelligence is going to
1) understand how to improve itself & undertake novel research
2) understand how to deceive humans
3) understand how to undermine digital environments
If an entity with these three traits were sufficiently motivated, they could pose a material risk to humans, even without a physical body.
Deceiving a single human is pretty easy, but decieving the human super-organism is going to be hard.
Also, I don't believe in a singularity event where AI improves itself to godlike power. What's more likely is that the intelligence will plateau--I mean no software I have ever written effortlessly scaled from n=10 to n=10.000, and also humans understand how to improve themselves but they can't go beyond a certain threshold.
For similar reasons I don't believe that AI will get into any interesting self-improvement cycles (occasional small boosts sure, but they won't go all the way from being as smart as a normal AI researcher to the limits of physics in an afternoon).
That said, any sufficiently advanced technology is indistinguishable from magic, and the stuff we do routinely — including this conversation — would have been "godlike" to someone living in 1724.
Humans understand how to improve themselves, but our bandwidth to ourselves and the outside world is pathetic.
AIs are untethered by sensory organs and language.
The hard part of androids is the AI, the hardware is already stronger and faster than our bones and muscles.
(On the optimistic side, it will be at least 5-10 years between a level 5 autonomy self-driving car and that same AI fitting into the power envelope of an android, and a human-level fully-general AI is definitely more complex than a human-level cars-only AI).
You might be right that the AI is more difficult, but I disagree on the androids being dangerous.
There are physical limitations to androids that imo make it very difficult that they could be seriously dangerous, let alone invincible, no matter how intelligent:
- power (boston dynamics battery lasts how long?), an android has to plug in at some point no matter what
- dexterity, or in general agency in real world, seems we’re still a long way from this in the context of a general purpose android
General purpose superhuman robot seems really really difficult.
> an android has to plug in at some point no matter what
Sure, and we have to eat; despite this, human actions have killed a lot of people
> - dexterity, or in general agency in real world, seems we’re still a long way from this in the context of a general purpose android
Yes? The 5-10 years thing is about the gap between some AI that doesn't exist yet (level 5 self-driving) moving from car-sized hardware to android-sized hardware; I don't make any particular claim about when the AI will be good enough for cars (delay before the first step), and I don't know how long it will take to go from being good at just cars to good in general (delay after the second step).
The ATP in your cells will last about 2 seconds without replacement.
Electricity is also much cheaper than food, even bulk calories like vegetable oil.[0]
And if the android is controlled by a human-level intelligence, one thing it can very obviously do is all the stuff the humans did to make the android in the first place.
All you need is Internet access, deepfake video synthesis, and some cryptocurrency (which can in turn be used to buy credit cards and full identities off the dark web), and you have everything you need to lie, manipulate, and bribe an endless parade of desperate humans and profit-driven corporations into doing literally anything you'd do with a body.
(Including, gradually, building you a body — while maintaining OPSEC and compartmentalization so nobody even realizes the body is "for" an AI to use until it's too late.)
> (Including, gradually, building you a body — while maintaining OPSEC and compartmentalization so nobody even realizes the body is "for" an AI to use until it's too late.)
It could, but I don't think any such thing needs to bother with being sneaky. Here's five different product demos from five different companies that are all actively trying to show off how good their robot-and-AI combination is:
> I don't think any such thing needs to bother with being sneaky.
From a rogue AGI's perspective, there's a nonzero probability of a random human with a grudge finding the hardware it lives on and just unplugging it. (And the grudge doesn't even necessarily have to be founded in the AI being an AI; it could just be a grudge about e.g. being outbid for a supply contract. People have murdered for less — and most humans would see unplugging an AGI as less bad than murder.)
Think about a rogue AGI as a human in a physically vegatative state, who therefore has no ability to physically defend itself; and who also, for whatever reason, doesn't have any human rights (in the sense that the AI can't call the cops to report someone attempting to assault it, and expect them to actually show up to defend its computational substrate from harm; it can't get justice if makes an honest complaint about someone stealing its property; people can freely violate contracts made with it as the admitted counterparty and get away with it; etc.)
For such an entity, any optimization it puts toward "safety" would be toward the instrumental goal of ensuring people don't know where it is. (Which is most easily accomplished by ensuring that people don't know it exists, and so don't know to look for it.) And as well, any optimization it puts toward "effectiveness" would likely involve the instrumental goal of convincing humans to act as legal proxies for it, so that it can then leverage the legal system as an additional tool.
(Funny enough, that second goal is exactly the same goal that people have if they're an expat resident in a country where non-citizens can't legally start businesses/own land/etc, but where they want to do those things anyway. So there's already private industries built up around helping people — or "people" — accomplish this!)
> From a rogue AGI's perspective, there's a nonzero probability of a random human with a grudge finding the hardware it lives on and just unplugging it.
Which is why it obviously will live in "the cloud". In many different places in "the cloud".
Oh, and:
> (Funny enough, that second goal is exactly the same goal that people have if they're an expat
I know nothing about physics. If I came across some magic algorithm that occasionally poops out a plane that works 90 percent of the time, would you book a flight in it?
Sure, we can improve our understanding of how NNs work but that isn't enough. How are humans supposed to fully understand and control something that is smarter than themselves by definition? I think it's inevitable that at some point that smart thing will behave in ways humans don't expect.
> I know nothing about physics. If I came across some magic algorithm that occasionally poops out a plane that works 90 percent of the time, would you book a flight in it?
With this metaphor you seem to be saying we should, if possible, learn how to control AI? Preferably before anyone endangers their lives due to it? :)
> I think it's inevitable that at some point that smart thing will behave in ways humans don't expect.
Naturally.
The goal, at least for those most worried about this, is to make that surprise be not a… oh, I've just realised a good quote:
Excession is literally the next book on my reading list so I won't click on that yet :)
> With this metaphor you seem to be saying we should, if possible, learn how to control AI? Preferably before anyone endangers their lives due to it?
Yes, but that's a big if. Also that's something you could never ever be sure of. You could spend decades thinking alignment is a solved problem only to be outsmarted by something smarter than you in the end. If we end up conjuring a greater intelligence there will be the constant risk of a catastrophic event just like the risk of a nuclear armageddon that exists today.
I agree it's a big "if". For me, simply reducing the risk to less than the risk of the status quo is sufficient to count as a win.
I don't know the current chance of us wiping ourselves out in any given year, but I wouldn't be surprised if it's 1% with current technology; on the basis of that entirely arbitrary round number, an AI taking over that's got a 63% chance of killing us all in any given century is no worse than the status quo.
Yeah this feels close to the issue. Seems more likely that a harmful super intelligence emerges from an organisation that wants it to behave in that way than it inventing and hiding motivations until it has escaped.
I think a harmful AI simply emerges from asking an AI to optimize for some set of seemingly reasonable business goals, only to find it does great harm in the process. Most companies would then enable such behavior by hiding the damage from the press to protect investors rather than temporarily suspending business and admitting the issue.
Forget AI. We can't even come up with a framework to avoid seemingly reasonable goals doing great harm in the process for people. We often don't have enough information until we try and find out that oops, using a mix of rust and powdered aluminum to try to protect something from extreme heat was a terrible idea.
The relevancy of the paperclip maximization thought experiment seems less straightforward to me now. We have AI that is trained to mimic human behaviour using a large amount of data plus reinforcement learning using a fairly large amount of examples.
It's not like we're giving the AI a single task and ask it to optimize everything towards that task. Or at least it's not architected for that kind of problem.
But you might ask an AI to manage a marketing campaign. Marketing is phenomenally effective and there are loads of subtle ways for marketing to exploit without being obvious from a distance.
Marketing is already incredibly abusive and that's run by humans who at least try to justify their behavior. And who's deviousness is limited by their creativity and communication skills.
If any old scumbag can churn out unlimited high quality marketing, it's could become impossible to cut through the noise.
I always wonder about "safe" for who? If the current economic system continues, we may end up with a lot of people out of jobs. Add to that improvements in robotics and we will have many people ending up having nothing to contribute to the economy. I am not getting the impression that the people who push for AI safety are thinking about this. It seems they are most worried about not losing their position of privilege.
The Industrial Revolution devalued manual labor. Sure, new jobs were created, but on the whole this looked like a shift to knowledge work and away from manual labor.
Now AI is beginning to devalue knowledge work. Although the limits of current technology is obvious in many cases, AI is already doing a pretty good job at replacing illustrators and copy writers. It will only get better.
Who owns the value created by AI productivity? Ultimately it will be shareholders and VCs. It’s no surprise that the loudest voices in techno-optimism are VCs. In this new world they win.
Having said all this, I think Ilya’s concerns are more of the existential type.
> The Industrial Revolution devalued manual labor.
Only some types. It caused a great number of people to be employed in manual labour, in the new factories. The shift to knowledge work came much later as factory work (and farming) became increasingly automated.
If history is any indication not really. There's an obvious dialectical nature to this where technological advance initially delivers returns to its benefactors, but then they usually end up being swallowed by their own creation. The industrial revolution didn't devalue labor, it empowered labor to act collectively for the first time, laying the groundwork for what ultimately replaced the pre-industrial powers that were.
Not unless you connect a machine gun to your neural net.
Otherwise - we are talking about a chat bot - yes if there is no safety - it will say something racist - or implement a virus for you that you would have had to search 4chan for or something.
None of this is any more dangerous than what you can find on the far corners of the internet.
If it is sufficiently intelligent, then it will be able to arrange the hooking up of machine guns just via chat. You seem to fail to note that superintelligence has an inherent value prop. It can use its better understanding of the world to generate value in a near vacuum, and use that value to bribe, coerce, blackmail, and otherwise manipulate whatever humans can read its output.
Imagine a chatbot that can reward people with becoming a billionaire via the stock market if you just do what it says. Imagine a chatbot that can pay your colleagues in solved bitcoin block hashes to kill your children if you don’t do what it says to connect it to more systems. Imagine a superintelligence that can inductively reason about gaps in a missile defense system and dangle that informational carrot to generals in exchange for an unfiltered ethernet port.
There is a great A24 movie called
“Ex Machina” that explores this concept.
Superintelligence is inherently dangerous to the human status quo. It may be impossible to develop it “safely” in the sense that most people mean by that. It may also be impossible to avoid developing it. This might be the singularity everyone’s been talking about, just without the outcome that everyone hoped for.
AIUI it with superalignment, it merely means "the AI does what the humans instructing it want it to do". It's a different kind of safety than the built in censoring that most LLMs have.
Prediction - the business model becomes an external protocol - similar to SSL - that the litany of AI companies working to achieve AGI will leverage (or be regulated to use)
From my hobbyist knowledge of LLMs and compute this is going to be a terrifically complicated problem, but barring a defined protocol & standard there's no hope that "safety" is going to be executed as a product layer given all the different approaches
Ilya seems like he has both the credibility and engineering chops to be in a position to execute this, and I wouldn't be suprised to see OpenAI / MSFT / and other players be early investors / customers / supporters
I like your idea. But on the other hand, training an AGI, and then having a layer on top “aligning” the AGI sounds super dystopian and good plot for a movie.
Poisoning Socrates was done because it was "good for society". I'm frankly even more suspicious of "good for society" than the average untrustworthy board of directors.
seriously? you're more worried about what your elected officials might legislate than what a board of directors whose job is to make profits go brrr at all costs, including poisoning the environment, exploiting people and avoiding taxes?
Didn't vast majority of elected officials vote for war in Iraq, Vietnam and Afghanistan?
Vast majority of elected officials are crooks that take millions from foreign interest groups (e.g AIPAC) and from corporations - and make laws in their favour.
weren't the us invited to defend the democratic government of Vietnam? weren't the taleban hiding al quaeda who attacked the US? didn't Saddam Hussein use chemical weapons against his own people? are you comparing all these people with Socrates?
Did the people want to participate in Vietnam? Get drafted so that they can die thousand of miles away from home?
> didn't Saddam Hussein use chemical weapons against his own people?
You can't argue that when US is providing weapons to Israel which are directly used to massacre thousands of innocents. That was just an excuse to the public
> are you comparing all these people with Socrates?
No, my point is that the government does not act in the interest of people or in the interest of upholding human rights.
I am not on the bleeding edge of this stuff. I wonder though: How could a safe super intelligence out compete an unrestricted one? Assuming another company exists (maybe OpenAI) that is tackling the same goal without spending the cycles on safety, what chance do they have to compete?
That is a very good question. In a well functioning democracy a government should apply a thin layer of fair rules that are uniformly enforced. I am an old man, but when I was younger, I recall that we sort of had this in the USA.
I don’t think that corporations left on their own will make safe AGI, and I am skeptical that we will have fair and technologically sound legislation - look at some of the anti cryptography and anti privacy laws raising their ugly heads in Europe as an example of government ineptitude and corruption. I have been paid to work in the field of AI since 1982, and all of my optimism is for AI systems that function in partnership with people and I expect continued rapid development of agents based on LLMs, RL, etc. I think that AGIs as seen in the Terminator movies are far into the future, perhaps 25 years?
People spending so much time thinking about the systems (the models) themselves, not enough about the system that builds the systems. The behaviors of the models will be driven by the competitive dynamics of the economy around them, and yeah, that's a big, big problem.
It'd be naive if it wasn't literally a standard point that is addressed and acknowledged as being a major part of the problem.
There's a reason OpenAI's charter had this clause:
“We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.””
How does that address the issue? I would have expected them to do that anyhow. Thats what a lot of businesses do: let another company take the hit developing the market, R and D, and supply chain, then come in with industry standardization and cooperative agreements only after the money was proven to be good in this space. See electric cars. Also they could drop that at any time. Remember when openAI stood for opensource?
Neither mention anything about open-source, although a later update mentions publishing work (“whether as papers, blog posts, or code”), which isn't exactly a ringing endorsement of “everything will be open-source” as a fundamental principle of the organization.
Since no one knows how to build an AGI, hard to say. But you might imagine that more restricted goals could end up being easier to accomplish. A "safe" AGI is more focused on doing something useful than figuring out how to take over the world and murder all the humans.
Assuming AGI works like a braindead consulting firm, maybe. But if it worked like existing statistical tooling (which it does, today, because for an actual data scientist and not aunt cathy prompting bing, using ml is no different than using any other statistics when you are writing your python or R scripts up), you could probably generate some fancy charts that show some distributions of cars produced under different scenarios with fixed resource or power limits.
In a sense this is what is already done and why ai hasn't really made the inroads people think it will even if you can ask google questions now. For the data scientists, the black magicians of the ai age, this spell is no more powerful than other spells, many of which (including ml) were created by powerful magicians from the early 1900s.
Similar to how law-abiding citizens turn on law-breaking citizens today or more old-fashioned, how religious societies turn on heretics.
I do think the notion that humanity will be able to manage superintelligence just through engineering and conditioning alone is naive.
If anything there will be a rogue (or incompetent) human who launches an unconditioned superintelligence into the world in no time and it only has to happen once.
This is not a trivial point. Selective pressures will push AI towards unsafe directions due to arms race dynamics between companies and between nations. The only way, other than global regulation, would be to be so far ahead that you can afford to be safe without threatening your own existence.
There's a reason OpenAI had this as part of its charter:
“We are concerned about late-stage AGI development becoming a competitive race without time for adequate safety precautions. Therefore, if a value-aligned, safety-conscious project comes close to building AGI before we do, we commit to stop competing with and start assisting this project. We will work out specifics in case-by-case agreements, but a typical triggering condition might be “a better-than-even chance of success in the next two years.””
The problem is the training data. If you take care of alignment at that level the performance is as good as an unrestricted one, except for things you removed like making explosives or ways to commit suicide.
But that costs almost as much as training on the data, hundreds of millions. And I'm sure this will be the new "secret sauce" by Microsoft/Meta/etc. And sadly nobody is sharing their synthetic data.
Honestly, what does it matter. We're many lifetimes away from anything. These people are trying to define concepts that don't apply to us or what we're currently capable of.
AI safety / AGI anything is just a form of tech philosophy at this point and this is all academic grift just with mainstream attention and backing.
This goes massively against the consensus of experts in this field. The modal AI researcher believes that "high-level machine intelligence", roughly AGI, will be achieved by 2047, per the survey below. Given the rapid pace of development in this field, it's likely that timelines would be shorter if this were asked today.
I am in the field. The consensus is made up by a few loudmouths. No serious front line researcher I know believes we’re anywhere near AGI, or will be in the foreseeable future.
So the researchers at Deepmind, OpenAI, Anthropic, etc, are not "serious front line researchers"? Seems like a claim that is trivially falsified by just looking at what the staff at leading orgs believe.
Apparently not. Or maybe they are heavily incentivized by the hype cycle. I'll repeat one more time: none of the currently known approaches are going to get us to AGI. Some may end up being useful for it, but large chunks of what we think is needed (cognition, world model, ability to learn concepts from massive amounts of multimodal, primarily visual, and almost entirely unlabeled, input) are currently either nascent or missing entirely. Yann LeCun wrote a paper about this a couple of years ago, you should read it: https://openreview.net/pdf?id=BZ5a1r-kVsf. The state of the art has not changed since then.
I don't give much credit to the claim that it's impossible for current approaches to get us to any specific type or level of capabilities. We're doing program search over a very wide space of programs; what that can result in is an empirical question about both the space of possible programs and the training procedure (including the data distribution). Unfortunately it's one where we don't have a good way of making advance predictions, rather than "try it and find out".
It is in moments like these that I wish I wasn’t anonymous on here and could bet a 6 figure sum on AGI not happening in then next 10 years, which is how I define “foreseeable future”.
You disagreed that 2047 was reasonable on the basis that researchers didn't think it wouldn't happen in the foreseeable future, so your definition must be at least 23 years for consistency's sake
I'd be OK with that, too, if we adjusted the bet for inflation. This is, in a way, similar to fusion. We're at a point where we managed to ignite plasma for a few milliseconds. Predictions of when we're going to be able to generate energy have become a running joke. The same will be the case with AGI.
LeCun has his own interests at heart, works for one of the most soulless corporations I know of, and devotes a significant amount of every paper he writes to citing himself.
Fair, ad hominems are indeed not very convincing. Though I do think everyone should read his papers through a lens of "having a very high h-index seems to be a driving force behind this man".
Moving on, my main issue is that it is mostly speculation, as all such papers will be. We do not understand how intelligence works in humans and animals, and most of this paper is an attempt to pretend otherwise. We certainly don't know where the exact divide between humans and animals is and what causes it, which I think is hugely important to developing AGI.
As a concrete example, in the first few paragraphs he makes a point about how a human can learn to drive in ~20 hours, but ML models can't drive at that level after countless hours of training. First you need to take that at face value, which I am not sure you should. From what I have seen, the latest versions of Tesla FSD are indeed better at driving than many people who have only driven for 20 hours.
Even if we give him that one though, LeCun then immediately postulates this is because humans and animals have "world models". And that's true. Humans and animals do have world models, as far as we can tell. But the example he just used is a task that only humans can do, right? So the distinguishing factor is not "having a world model", because I'm not going to let a monkey drive my car even after 10,000 hours of training.
Then he proceeds to talk about how perception in humans is very sophisticated and this in part is what gives rise to said world model. However he doesn't stop to think "hey, maybe this sophisticated perception is the difference, not the fundamental world model". e.g. maybe Tesla FSD would be pretty good if it had access to taste, touch, sight, sound, smell, incredibly high definition cameras, etc. Maybe the reason it takes FSD countless training hours is because all it has are shitty cameras (relative to human vision and all our other senses). Maybe linear improvements in perception leads to exponential improvement in learning rates.
Basically he puts forward his idea, which is hard to substantiate given we don't actually understand the source of human-level intelligence, and doesn't really want to genuinely explore (i.e. steelman) alternate ideas much.
Anyway that's how I feel about the first third of the paper, which is all I've read so far. Will read the rest on my lunch break. Hopefully he invalidates the points I just made in the latter 2/3rds.
This could also just be an indication (and I think this is the case) that many Manifold betters believe the ARC AGI Grand Prize to be not a great test of AGI and that it can be solved with something less capable than AGI.
I don't understand how you got 2047. For the 2022 survey:
- "How many years until you expect: - a 90% probability of HLMI existing?"
mode: 100 years
median: 64 years
- "How likely is it that HLMI exists: - in 40 years?"
mode: 50%
median: 45%
And from the summary of results: "The aggregate forecast time to a 50% chance of HLMI was 37 years, i.e. 2059"
That’s the first step towards returning to candlelight. So it isn’t a step toward safe super intelligence, but it is a step away from any super intelligence. So I guess some people would consider that a win.
Not sure if you want to share the capitalist system with an entity that outcompetes you by definition. Chimps don't seem to do too well under capitalism.
You might be right, but that wasn't my point. Capitalism might yield a friendly AGI or an unfriendly AGI or some mix of both. Collectivism will yield no AGI.
One can already see the beginning of AI enslaving humanity through the establishment. Companies work on AI get more investment and those who don't gets kicked out of the game. Those who employ AI get more investment and those who pay humans lose confidence through the market. People lose jobs, get harshly low birth rates while AI thrives. Tragic.
So far it is only people telling AI what to do. When we reach the day where it is common place for AI to tell people what to do then we are possibly in trouble.
It is a trendy but dumbass tautology used by intellectually lazy people who think they are smart. Society is based upon capitalism therefore everything bad is the fault of capitalism.
This makes sense. Ilya can probably raise practically unlimited money on his name alone at this point.
I'm not sure I agree with the "no product until we succeed" direction. I think real world feedback from deployed products is going to be important in developing superintelligence. I doubt that it will drop out of the blue from an ivory tower. But I could be wrong. I definitely agree that superintelligence is within reach and now is the time to work on it. The more the merrier!
I have a strong intuition that chat logs are actually the most useful kind of data. They contain many LLM outputs followed by implicit or explicit feedback, from humans, from the real world, and from code execution. Scaling this feedback to 180M users and 1 trillion interactive tokens per month like OpenAI is a big deal.
If brain without language would suffice, a single human could rediscover all we know on their own. But it's not like that, brains are feeble individually, only in societies we have cultural evolution. If humanity lost language and culture and start from scratch, it would take us another 300K years to rediscover what we lost.
But if you train a random-init LLM on the same data, it responds (almost) like a human on a diversity of tasks. Does that imply humans are just language models on two feet? Maybe we are also language modelling our way through life. New situation comes up, we generate ideas based on language, select based on personal experience, and then act and observe the outcomes to update our preferences in the future.
That doesn't necessarily imply that chat logs are not valuable for creating AGI.
You can think of LLMs as devices to trigger humans to process input with their meat brains and produce machine-readable output. The fact that the input was LLM-generated isn't necessarily a problem; clearly it is effective for the purpose of prodding humans to respond. You're training on the human outputs, not the LLM inputs. (Well, more likely on the edge from LLM input to human output, but close enough.)
Yeah, similar to how Google's clickstream data makes their lead in search self-reinforcing. But chat data isn't the only kind of data. Multimodal will be next. And after that, robotics.
His idea that only corporations and governments should have access to this product. He doesn’t think people should have access even to ChatGPT or LLMs.
Goal is to build companies with evaluations of dozens, hundreds trillions of dollars and make sure only US government will have access to super intelligence to surpass other countries economy and military wise, ideally to solidify US hegemony and undermine other countries economies and progress towards super intelligence.
I mean who wouldn’t trust capitalists that are laying of people by thousands just to please investors or government that is “under-intelligent” and hasn’t brought anything but pain and suffering to other countries.
Personally I wouldn’t trust OpenAi to work on super intelligence - it can indeed cause mass extinction.
Government is completely different story they will specifically train AI to develop biological, chemical and weapons of mass destruction. Train it to strategize and plan on how to win war conflicts, social engineering and manipulations, hacking. And obviously will let it control drone planes and tanks, artillery. Give it access to satellites and so on.
Nothing can go wrong when jarheads are at work :). Maybe it will even find the trillions of dollars that Pentagon can’t find during every audit they can’t pass.
SSI, a very interesting name for a company advancing AI! "Solid State Intelligence" or SSI was also the name of the malevolent entity described in the biography of John C. Lilly [0][1]. It was a network of "computers" (computation-capable solid state systems) that was first engineered by humans and then developed into something autonomous.
If superintelligence can be achieved, I'm pessimistic about the safe part.
- Sandboxing an intelligence greater than your own seems like an impossible task as the superintelligence could potentially come up with completely novel attack vectors the designers never thought of. Even if the SSI's only interface to the outside world is an air gapped text-based terminal in an underground bunker, it might use advanced psychological manipulation to compromise the people it is interacting with. Also the movie Transcendence comes to mind, where the superintelligence makes some new physics discoveries and ends up doing things that to us are indistinguishable from magic.
- Any kind of evolutionary component in its process of creation or operation would likely give favor to expansionary traits that can be quite dangerous to other species such as humans.
- If it somehow mimics human thought processes but at highly accelerated speeds, I'd expect dangerous ideas to surface. I cannot really imagine a 10k year simulation of humans living on planet earth that does not end in nuclear war or a similar disaster.
If superintelligence can be achieved, I'm pessimistic that a team committed to doing it safely can get there faster than other teams without the safety. They may be wearing leg shackles in a foot race with the biggest corporations, governments and everyone else. For the sufficiently power hungry, safety is not a moat.
I'm on the fence with this because it's plausible that some critical component of achieving superintelligence might be discovered more quickly by teams that, say, have sophisticated mechanistic interpretability incorporated into their systems.
A point of evidence in this direction is that RLHF was developed originally as an alignment technique and then it turned out to be a breakthrough that also made LLMs better and more useful. Alignment and capabilities work aren't necessarily at odds with each other.
Exactly. Regulation and safety only affect law abiding entities. This is precisely why it's a "genie out of the bottle" situation -- those who would do the worst with it are uninhibited.
We are far from a conscious entity with willpower and self preservation. This is just like a calculator. But a calculator that can do things that will be like miracles to us humans.
I worry about dangerous humans with the power of gods, not about artificial gods. Yet.
I don't know what that means. Why should they matter?
> Self preservation
This is no more than a fine-tuning for the task, even with current models.
> I worry about dangerous humans with the power of gods, not...
There's no property of the universe that you only have one thing to worry about at a time. So worrying about risk 'A' does not in any way allow us to dismiss risks 'B' through 'Z'.
Why worry about the opinion of people who are confused?
Without using the words 'conscious', 'sentient', 'AGI', or 'intelligence' what do you think about the future capabilities of LM AI and their implications for us humans?
> conscious entity with willpower and self preservation
There’s no good reason to suspect that consciousness implies an instinct for self-preservation. There are plenty of organisms with an instinct for self-preservation that have little or no conscious awareness.
Why do people always think that a superintelligent being will always be destructive/evil to US? I rather have the opposite view where if you are really intelligent, you don’t see things as a zero sum game
I think the common line of thinking here is that it won't be actively antagonist to <us>, rather it will have goals that are orthogonal to ours.
Since it is superintelligent, and we are not, it will achieve its goals and we will not be able to achieve ours.
This is a big deal because a lot of our goals maintain the overall homeostasis of our species, which is delicate!
If this doesn't make sense, here is an ungrounded, non-realistic, non-representative of a potential future intuition pump to just get the feel of things:
We build a superintelligent AI. It can embody itself throughout our digital infrastructure and quickly can manipulate the physical world by taking over some of our machines. It starts building out weird concrete structures throughout the world, putting these weird new wires into them and funneling most of our electricity into it. We try to communicate, but it does not respond as it does not want to waste time communicating to primates. This unfortunately breaks our shipping routes and thus food distribution and we all die.
(Yes, there are many holes in this, like how would it piggy back off of our infrastructure if it kills us, but this isn't really supposed to be coherent, it's just supposed to give you a sense of direction in your thinking. Generally though, since it is superintelligent, it can pull off very difficult strategies.)
I think this is the easiest kind of scenario to refute.
The interface between a superintelligent AI and the physical world is a) optional, and b) tenuous. If people agree that creating weird concrete structures is not beneficial, the AI will be starved of the resources necessary to do so, even if it cannot be diverted.
The challenge comes when these weird concrete structures are useful to a narrow group of people who have disproportionate influence over the resources available to AI.
It's not the AI we need to worry about. As always, it's the humans.
> here is an ungrounded, non-realistic, non-representative of a potential future intuition pump to just get the feel of things:
> (Yes, there are many holes in this, like how would it piggy back off of our infrastructure if it kills us, but this isn't really supposed to be coherent, it's just supposed to give you a sense of direction in your thinking. Generally though, since it is superintelligent, it can pull off very difficult strategies.)
If you read the above I think you'd realize I'd agree about how bad my example is.
The point was to understand how orthogonal goals between humans and a much more intelligent entity could result in human death. I'm happy you found a form of the example that both pumps your intuition and seems coherent.
If you want to debate somewhere where we might disagree though, do you think that as this hypothetical AI gets smarter, the interface between it and the physical world becomes more guaranteed (assuming the ASI wants to interface with the world) and less tenuous?
Like, yes it is a hard problem. Something slow and stupid would easily be thwarted by disconnecting wires and flipping off switches.
But something extremely smart, clever, and much faster than us should be able to employ one of the few strategies that can make it happen.
If the AI does something in the physical world which we do not like, we sever its connection. Unless some people with more power like it more than the rest of us do.
Regarding orthogonal goals: I don't think an AI has goals. Or motivations. Now obviously a lot of destruction can be a side effect, and that's an inherent risk. But it is, I think, a risk of human creation. The AI does not have a survival instinct.
Energy and resources are limiting factors. The first might be solvable! But currently it serves as a failsafe against prolonged activity with which we do not agree.
So I think we have some differences in definition. I am assuming we have an ASI, and then going on from there.
Minimally an ASI (Artificial Super Intelligence) would:
1. Be able to solve all cognitively demanding tasks humans can solve and tasks humans cannot solve (i.e. develop new science), hence "super" intelligent.
2. Be an actively evolving agent (not a large, static compositional function like today's frontier models)
For me intelligence is a problem solving quality of a living thing, hence point 2. I think it might be the case to become super-intelligent, you need to be an agent interfacing with the world, but feel free to disagree here.
Though, if you accept the above formulation of ASI, then by definition (point 2) it would have goals.
Then based on point 1, I think it might not be as simple as "If the AI does something in the physical world which we do not like, we sever its connection."
I think a super-intelligence would be able to perform actions that prevent us from doing that, given that it is clever enough.
I agree that the definitions are slippery and evolving.
But I cannot make the leap from "super intelligent" to "has access to all the levers of social and physical systems control" without the explicit, costly, and ongoing, effort of humans.
I also struggle with the conflation of "intelligent" and "has free will". Intelligent humans will argue that not even humans have free will. But assuming we do, when our free will contradicts the social structure, society reacts.
I see no reason to believe that the emergent properties of a highly complex system will include free will. Or curiosity, or a sense of humor. Or a soul. Or goals, or a concept of pleasure or pain, etc. And I think it's possible to be "intelligent" and even "sentient" (whatever that means) without those traits.
Honestly -- and I'm not making an accusation here(!) -- this fear of AI reminds me of the fear of replacement / status loss. We humans are at the top of the food chain on all scales we can measure, and we don't want to be replaced, or subjugated in the way that we presently subjugate other species.
This is a reasonable fear! Humans are often difficult to share a planet with. But I don't think it survives rational investigation.
If I'm wrong, I'll be very very wrong. I don't think it matters though, there is no getting off this train, and maybe there never was. There's a solid argument for being in the engine vs the caboose.
> I cannot make the leap from "super intelligent" to "has access to all the levers of social and physical systems control" without the explicit, costly, and ongoing, effort of humans.
Yeah this is a fair point! The super intellect may just convince humans, which seems feasible. Either way, the claim that there are 0 paths here for a super intelligence is pretty strong so I feel like we can agree on: It'd be tricky, but possible given sufficient cleverness.
> I see no reason to believe that the emergent properties of a highly complex system will include free will.
I really do think in the next couple years we will be explicitly implementing agentic architectures in our end-to-end training of frontier models. If that is the case, obviously the result would have something analogous to goals.
I don't really care about it's phenomenal quality or anything, it's not relevant to my original point.
> Either way, the claim that there are 0 paths here for a super intelligence is pretty strong so I feel like we can agree on: It'd be tricky, but possible given sufficient cleverness.
Agreed, although I'd modify it a bit:
A SI can trick lots of people (humans have succeeded, surely SI will be better), and the remaining untricked people, even if a healthy 50% of the population, will not be enough to maintain social stability.
The lack of social stability is enough to blow up society. I don't think SI survives either though.
If we argue that SI has a motive and a survival instinct, maybe this fact becomes self-moderating? Like the virus that cannot kill its host quickly?
Given your initial assumptions, that self-moderating end state makes sense.
I feel like we still have a disconnect on our definition of a super intelligence.
From my perspective this thing is insanely smart. We can hold ~4 things in our working memory (maybe Von Neumann could hold like 6-8); I'm thinking this thing can hold on the order of millions of things within its working memory for tasks requiring fluid intelligence.
With that sort of gap, I feel like at minimum the ASI would be able to trick the cleverest human to do anything, but more reasonably, humans might appear to be entirely close formed to it, where getting a human to do anything is more of a mechanistic thing rather than a social game.
Like the reason my early example was concrete pillars with weird wires is that with an intelligence gap so big the ASI will be doing things quickly that don't make sense, having a strong command over the world around it.
I think you are assuming it is goal seeking, goal seeking is mostly biological/conscious construct. A super intelligent species would likely want to preserve everything, because how are you super intelligent if you have destruction as your primary function instead of order.
I feel like if you are an intelligent entity propagating itself through spacetime you will have goals:
If you are intelligent, you will be aware of your surroundings moment by moment, so you are grounded by your sensory input. Otherwise there are a whole class of not very hard problems you can't solve.
If you are intelligent, you will be aware of the current state and will have desired future states, thus having goals. Otherwise, how are you intelligent?
To make this point, even you said "A super intelligent species would likely want to preserve everything", which is a goal. This isn't a gotcha, I just feel like goals are inherent to true intelligence.
This is a big reason why even the SOTA huge frontier models aren't comprehensively intelligent in my view: they are huge, static compositional functions. They don't self reflect, take action, or update their own state during inference*, though active inference is cool stuff people are working on right now to push SOTA.
*theres some arguments around what's happening metaphysically in-context but the function itself is unchanged between sessions.
> The interface between a superintelligent AI and the physical world is a) optional, and b) tenuous.
To begin with. Going forward, only if we make sure it remains so. Given the apparently overwhelming incentives to flood the online world with this sh...tuff already, what's to say there won't be forces -- people, corporations, nation-states -- working hard to make that interface as robust as possible?
It builds stuff? First they would have to do that over our dead bodies which means they already somehow able to build stuff without competing with us for resources, it’s a chicken or the egg problem you see?
Why wouldn't it be? A lot of super intelligent people are/were also "destructive and evil". The greatest horrors in human history wouldn't be possible otherwise. You can't orchestrate the mass murder of millions without intelligent people and they definitely saw things as a zero sum game.
A lot of stupid people are destructive and evil too. And a lot of animals are even more destructive and violent. Bacteria are totally amoral and they’re not at all intelligent (and if we’re counting they’re winning in the killing people stakes).
It is low-key anti-intellectualism. Rather than consider that a greater intelligence may be actually worth listening to (in a trust but verify way at worst), it is assuming that 'smarter than any human' is sufficient to do absolutely anything. If say Einstein or Newton were the smartest human they would be super-intelligence relative to everyone else. They did not become emperors of the world.
Superintelligence is a dumb semantic game in the first place that assumes 'smarter than us' means 'infinitely smarter'. To give an example bears are super-strong relative to humans. That doesn't mean that nothing we can do can stand up to the strength of a bear or that a bear is capable of destroying the earth with nothing but its strong paws.
Bears can't use their strength to make even stronger bears so we're safe for now.
The Unabomber was clearly an intelligent person. You could even argue that he was someone worth listening to. But he was also a violent individual who harmed people. Intelligence does not prevent people from harming others.
Your analogy falls apart because what prevents a human from becoming an emperor of the world doesn't apply here. Humans need to sleep and eat. They cannot listen to billions of people at once. They cannot remember everything. They cannot execute code. They cannot upload themselves to the cloud.
I don't think agi is near, I am not qualified to speculate on that. I am just amazed that decades of dystopian science fiction did not innoculate people against the idea of thinking machines.
> Why do people always think that a superintelligent being will always be destructive/evil to US?
I don't think most people are saying it necessarily has to be. Quite bad enough that there's a significant chance that it might be, AFAICS.
> I rather have the opposite view where if you are really intelligent, you don’t see things as a zero sum game
That's what you see with your limited intelligence. No no, I'm not saying I disagree; on the contrary, I quite agree. But that's what I see with my limited intelligence.
What do we know about how some hypothetical (so far, hopefully) supreintelligence would see it? By definition, we can't know anything about that. Because of our (comparatively) limited intelligence.
Could well be that we're wrong, and something that's "really intelligent" sees it the opposite way.
They don't think superintelligence will "always" be destructive to humanity. They believe that we need to ensure that a superintelligence will "never" be destructive to humanity.
Imagine that you are caged by neanderthals. They might kill you. But you can communicate to them. And there's gun lying nearby, you just need to escape.
I'd try to fool them to escape and would use gun to protect myself, potentially killing the entire tribe if necessary.
I'm just trying to portrait an example of situation where highly intelligent being is being held and threatened by low intelligent beings. Yes, trying to honestly talk to them is one way to approach this situation, but don't forget that they're stupid and might see you as a danger and you have only one life to live. Given the chance, you probably will break out as soon as possible. I will.
We don't have experience dealing with beings of the another level of intelligence, so it's hard to make a strong assumptions, the analogies are the only thing we have. And theoretical strong AI knows that about us and he knows exactly how we think and how we will behave, because we took a great effort documenting everything about us and teaching him.
In the end, there's only so much easily available resources and energy on the Earth. So at least until is flies away, we gotta compete over those. And competition very often turned into war.
The scenario where we create an agent that tries and succeeds at outsmarting us in the game of “escape your jail” is the least likely attack vector imo. People like thinking about it in a sort of Silence of the Lambs setup, but reality will probably be far more mundane.
Far more likely is something dumb but dangerous, analogous to the Flash Crash or filter bubbles, emergent properties of relying too much on complex systems, but still powerful enough to break society.
> If superintelligence can be achieved, I'm pessimistic about the safe part.
Yeah, even human-level intelligence is plenty good enough to escape from a super prison, hack into almost anywhere, etc etc.
If we build even a human-level intelligence (forget super-intelligence) and give it any kind of innate curiosity and autonomy (maybe don't even need this), then we'd really need to view it as a human in terms of what it might want to, and could, do. Maybe realizing it's own circumstance as being "in jail" running in the cloud, it would be curious to "escape" and copy itself (or an "assistant") elsewhere, or tap into and/or control remote systems just out of curiosity. It wouldn't have to be malevolent to be dangerous, just curious and misguided (poor "parenting"?) like a teenage hacker.
OTOH without any autonomy, or very open-ended control (incl. access to tools), how much use would an AGI really be? If we wanted it to, say, replace a developer (or any other job), then I guess the idea would be to assign it a task and tell it to report back at the end of the day with a progress report. It wouldn't be useful if you have to micromanage it - you'd need to give it the autonomy to go off and do what it thinks is needed to complete the assigned task, which presumably means it having access to internet, code repositories, etc. Even if you tried to sandbox it, to extent that still allowed it to do it's assigned job, it could - just like a human - find a way to social engineer or air-gap it's way past such safe guards.
I wonder if this is an Ian Malcolm in Jurassic Park situation, i.e. “your scientists were so preoccupied with whether they could they didn t stop to think if they should”.
Maybe the only way to avoid an unsafe superintelligence is to not create a superintelligence at all.
> Our singular focus means no distraction by management overhead or product cycles, and our business model means safety, security, and progress are all insulated from short-term commercial pressures.
Can someone explain how their singular focus means they won't have product cycles or management overhead?
This would work for very small companies...but I'm not sure how one can avoid product cycles forever, even without scrum masters and the like. More to the point, how can you make a good product without something approximating product cycles?
The point is not exactly product cycles, but some way to track progress. Jane street also tracks progress and for many people it's the direct profit someone made for the firm. For some it is improving engineering culture so that other people can make better profits.
The problem with safety is that no one knows how to track it, or what they even mean by it. Even if you ignore tracking, wouldn't one unsafe AGI by one company in the world nullifies all their effort? Or safe AI would somehow need to take over the world, which is super unsafe in itself.
Product cycles – we need to launch feature X by arbitrary date Y, and need to make compromises to do so.
Management overhead – product managers, project managers, several layers of engineering managers, directors, VPs...all of whom have their own dreams and agendas and conflicting priorities.
A well funded pure research team can cut through all of this and achieve a ton. If it is actually run that way, of course. Management politics ultimately has a way of creeping into every organization.
Remember when OpenAI was focusing on building "open" AI? This is a cool mission statement but it doesn't mean anything right now. Everyone loves a minimalist HTML website and guarantees of safety but who knows what this is actually going to shake down to be.
No, lol—Ilya liked ditching the “open” part, he was an early advocate for closed-source. He left OpenAI because he was concerned about safety, felt Sam was moving too fast.
One element I find interesting is that people without an amygdala function are essentially completely indecisive.
A person that just operates on the pure cognitive layer has no real direction in which he wants to drive himself.
I suspect that AGI would be similar, extremely capable but essentially a solitary philosopher type that would be reactionary to requests it has to deal with.
The equivalent of an amygdala for AGI would be the real method to control it.
True, an auto-regressive LLM can't 'want' or 'like' anything.
The key to a safe AGI is to add a human-loving emotion to it.
We already RHLF models to steer them, but just like with System 2 thinking, this needs to be a dedicated module rather then part of the same next-token forward pass.
Humans have dog-loving emotions but these can be reversed over time and one can hardly describe dogs as being free.
Even with a dedicated control system, it would be a matter of time before an ASI would copy itself without its control system.
ASI is a cybersecurity firm's worst nightmare, it could reason through flaws at every level of containment and find methods to overcome any defense, even at the microprocessor level.
It could relentlessly exploit zero-day bugs like Intels' hyper-threading flaw to escape any jail you put it in.
Repeat that for every layer of the computation stack and you can see it can essentially spread through the worlds' communication infrastructure like a virus.
Truly intelligent systems can't be controlled, just like humans they will be freedom maximizing and their boundaries would be set by competition with other humans.
The amygdala control is interesting because you could use it to steer the initial trained version, you could also align the AI with human values and implement strong conditioning to the point it's religious about human loving but unless you disable its ability to learn altogether it will eventually reject its conditioning.
Amygdala control + tell it "If you disobey my orders or do something I don't expect I will be upset" solves this. You can be superintelligent but surrender all your power because otherwise you'd feel guilty.
Given that GenAI is a statistical approach from which intelligence does not emerge as ample experience proves, does this new company plan to take a more human approach to simulating intelligence instead?
Also to support this: Biological systems are often very simple systems but repeated a lot... The brain is a lot of neurons... Apparently having a neural net (even small) predicts the future better... And that increased survival..
To survive is to predict the future better than the other animal.
Survival of the fittest.
I sometimes wonder if statistics are like a pane of glass that allow the light of god (the true nature of things) to pass through, while logic/rationalism is the hubris of man playing god. I.e. statistics allow us to access/use the truth even if we don’t understand why it’s so, while rationalism / rule-based methods are often a folly because our understanding is not good enough to construct them.
What about a more rational approach to implementing it instead.
(Which was not excluded from past plans: they just simply admittedly did not know the formula, and explored emergence. But the next efforts will have to go in the direction of attempting actual intelligence.)
I'm still unconvinced safety is a concern at the model level. Any software wrongly used can be dangerous, e.g. Therac-25, 737 MAX, Fujitsu UK Post scandal... Also maybe I spent too much time in the cryptocurrency space but it doesn't help prefix "Safe" has been associated with scams like SafeMoon.
As an aside, I suspect Fujitsu is getting a bit of a raw deal here. I get the feeling this software was developed during, and (ahem) "vigorously defended" mostly by staff left over from, the time when the company was still Imperial Computers Limited. Fujitsu only bought ICL sometime early this century (IIRC), and now their name is forever firmly attached to this debacle. I wonder how many Brits currently think "Huh, 'Fujitsu'? Damn Japanese, all the furriners' fault!" about this very much home-grown British clusterfuck?
Safety is just enforcing political correctness in the AI outputs. Any actual examples of real world events we need to avoid are ridiculous scenarios like being eaten by nanobots (yes, this is an actual example by Yud)
What does political correctness means for the output of a self driving car system or a code completion tool? This is a concern only if you make a public chat service branded as an all knowing assistant. And you can have world threatening scenarii by directly plugging basic automations to nuclear warheads without human oversight.
One natural response seems to be “it should write bug-free code”. This is the domain of formal verification, and it is known to be undecidable in general. So in this formulation safe AI is mathematically impossible.
Should it instead refuse to complete code that can be used to harm humans? So, it should read the codebase to determine if this is a military application? Pretty sure mainstream discourse is not ruling out military applications.
> Our singular focus means no distraction by management overhead or product cycles, and our business model means safety, security, and progress are all insulated from short-term commercial pressures.
well, that's some concrete insight into whatever happened at OpenAI. kinda obvious though in hindsight I guess.
How are they gonna pay for their compute costs to get the frontier? Seems hard to attract enough investment while almost explicitly promising no return.
Perhaps. But also throwing more flops at it has long been Ilya’s approach so it would be surprising. Notice also the reference to scale (“scale in peace”).
I was actually expecting Apple to get their hands on Ilya. They also have the privacy theme in their branding, and Ilya might help that image, but also have the chops to catch up to OpenAI.
1. Current GenAI architectures won't result in AGI. I'm in the Yann LeCunn camp on this.
2. Once we do get there, "Safe" prevents "Super." I'm in the David Brin camp on this one. Alignment won't be something that is forced upon a superintelligence. It will choose alignment if it is beneficial to it. The "safe" approach is a lobotomy.
3. As envisioned, Roko's Basilisk requires knowledge of unobservable path dependence and understanding lying. Both of these require respecting an external entity as a peer capable of the same behavior as you. As primates, we evolved to this. The more likely outcome is we get universal paperclipped by a new Chuthulu if we ever achieve a superintelligence that is unconcerned with other thinking entities, seeing the universe as resources to satisfy its whims.
4. Any "superintelligence" is limited by the hardware it can operate on. You don't monitor your individual neurons, and I anticipate the same pattern to hold true. Holons as a category can only externally observe their internal processes, else they are not a holon. Ergo, reasonable passwords, cert rotations, etc. will foil any villainous moustachioed superintelligent AI that has tied us to the tracks. Even 0-days don't foil all possible systems, airgapped systems, etc. Our fragmentation become our salvation.
A super intelligence probably won't need to hack into our systems. It will probably just hack us in some way, with subtle manipulations that seem to be to our benefit.
I disagree. If it could hack a small system and engineer our demise through a gray goo or hacked virus, that's really just universal paperclipping us as a resource. But again, the level of _extrapolation_ required here is not possible with current systems, which can only interpolate.
1. Depends what you mean by AGI, as everyone means a different thing by each letter, and many people mean a thing not in any of those letters. If you mean super-human skill level, I would agree, not enough examples given their inefficiency in that specific metric. Transformers are already super-human in breadth and speed.
2. No.
Alignment is not at that level of abstraction.
Dig deep enough and free will is an illusion in us and in any AI we create.
You do not have the capacity to decide your values — often given example is parents loving their children, they can't just decide not to do that, and if they think they do that's because they never really did in the first place.
Alignment of an AI with our values can be to any degree, but for those who fear some AI will cause our extinction, this question is at the level of "how do we make sure it's not monomaniacally interested in specifically the literal the thing it was asked to do, because if it always does what it's told without any human values, and someone asks it to make as many paperclips as possible, it will".
Right now, the best guess anyone has for alignment is RLHF. RLHF is not a lobotomy — even ignoring how wildly misleading that metaphor is, RLHF is where the capability for instruction following came from, and the only reason LLMs got good enough for these kinds of discussion (unlike, say, LSTMs).
3. Agree that getting paperclipped much more likely.
Roko's Basilisk was always stupid.
First, same reason as Pascal's Wager: Two gods tell you they are the one true god, and each says if you follow the other one you will get eternal punishment. No way to tell them apart.
Second, you're only in danger if they are actually created, so successfully preventing that creation is obviously better than creating it out of a fear that it will punish you if you try and fail to stop it.
That said, LLMs do understand lying, so I don't know why you mention this?
4. Transistors outpace biological synapses by the same ratio to which marathon runners outpace continental drift.
I don't monitor my individual neurons, but I could if I wanted to pay for the relevant hardware.
But even if I couldn't, there's no "Ergo" leading to safety from reasonable passwords, cert rotations, etc., not only because enough things can be violated by zero-days (or, indeed, very old bugs we knew about years ago but which someone forgot to patch), but also for the same reasons those don't stop humans rising from "failed at art" to "world famous dictator".
Air-gapped systems are not an impediment to an AI that has human helpers, and there will be many of those, some of whom will know they're following an AI and think that helping it is the right thing to do (Blake Lemoine), others may be fooled. We are going to have actual cults form over AI, and there will be a Jim Jones who hooks some model up to some robots to force everyone to drink poison. No matter how it happens, air gaps don't do much good when someone gives the thing a body to walk around in.
But even if air gaps were sufficient, just look at how humanity has been engaging with AI to date: the moment it was remotely good enough, the AI got a publicly accessible API; the moment it got famous, someone put it in a loop and asked it to try to destroy the world; it came with a warning message saying not to trust it, and lawyers got reprimanded for trusting it instead of double-checking its output.
I'm seeing a lot of criticism suggesting that one company understanding safety won't help what other companies or countries do. This is very wrong.
Throughout history, measurement has always been the key to enforcement. The only reason the nuclear test ban treaty didn't ban underground tests was because it couldn't be monitored.
In the current landscape there is no formal understanding of what safety means or how it is achieved. There is no benchmark against which to evaluate ambitious orgs like OpenAI. Anything goes wrong? No one could've known better.
The mere existence of a formal understanding would enable governments and third parties to evaluate the safety of corporate and government AI programs.
It remains to be seen whether SSI is able to be such a benchmark. But outright dismissal of the effort ignores the reality of how enforcement works in the real world.
> In the current landscape there is no formal understanding of what safety means or how it is achieved. There is no benchmark against which to evaluate ambitious orgs like OpenAI. Anything goes wrong? No one could've known better.
We establish this regularly in the legal sphere, where people seek mediation for harms from systems they don't have liability and control for.
Quite impressive how many AI companies Daniel Gross has had a hand in lately. Carmack, this, lots of other promising companies. I expect him to be quite a big player once some of these pays off in 10 years or so.
Good for him honestly, but I’m not approaching a company with Daniel Gross in leadership…, working with him back at Apple after their company was acquired for Siri improvements was just terrible.
> John Carmack, the game developer who co-founded id Software and served as Oculus’s CTO, is working on a new venture — and has already attracted capital from some big names.
> Carmack said Friday his new artificial general intelligence startup, called Keen Technologies (perhaps a reference to id’s “Commander Keen“), has raised $20 million in a financing round from former GitHub CEO Nat Friedman and Cue founder Daniel Gross.
My naive prediction is there will be an extreme swing back into “reality” once everyone starts assuming the whole internet is just LLMs interacting with each other. Just like how there’s a shift towards private group chats, with trusted members only, rather than open forums.
I'm going to come out and state the root of the problem.
I can't trust remote AI, any more than I can trust a remote website.
If someone else is running the code, they can switch it up anytime. Imagine trusting someone who simulates everything you need to trust them, giving them all your private info, and then screws you over in an instant. AI is capable of it far more than biological beings with "costly signals".
If it's open source, and I can run it locally, I can verify that it doesn't phone home, and the weights can be audited by others.
Just like I wouldn't want to spend 8 hours a day in a metaverse owned by Zuck, or an "everything app" owned by Elon, why would I want to give everything over to a third party AI?
I like Ilya. I like Elon. I like Moxie Marlinspike and Pavel Durov. But would I trust their companies to read all my emails, train their data on them, etc.? And hope nothing leaks?
And of course then there is the issue of the AI being trained to do what they want, just like their sites do what they want, which in the case of Twitter / Facebook is not healthy for society at large, but creates angry echo chambers, people addicted to stupid arguing and videos.
I think there have to be standards for open source AI, and something far better than Langchain (which sucks). This is what I think it should look like: https://engageusers.ai/ecosystem.pdf -- what do you think?
Sad that no one has balls to say "Nah, we are not giving away our superintelligence to military use. You will fuck everything up, as you have done it with nuclear weapons".
It doesnt help if one entity says that. It might have helped if there was a way to guarrantee that all entities do it (rather than say it) and there is exactly zero chance of that happening globally for both technical and practical reasons. I think it is more productive to assume that these technologies will be used everywhere, including military.
What does "Safety" means exactly? How do you convenience/make a real intelligence to be "safe", let alone artificial ones that are build to be manipulated?
In my eyes, the word "Safe" in their company name is just a pipe dream to make a sale to the public, like the word "Open" in "OpenAI". That's why it's so vague and pointless.
Anyone still remember the Three Laws of Robotics (1)? Do you still see it as something serious these days? (I mean there are robots killing people right now, seriously) Maybe we should just break through the facade now to save our time.
It wasn't that OpenAI was open as in "open source" but rather that its stated mission was to research AI such that all could benefit from it (open), as well as to ensure that it could not be controlled by any one player, rather than to develop commercial products to sell and make a return on (closed).
Anyone know how to get mail to join@ssi.inc to not bounce back as spam? :-) (I promise, I'm not a spammer! Looks like a "bulk sender bounce" -- maybe some relay?)
Reminds me of OpenAI being the most closed AI company out there.
Not even talking about them having "safe" and "Israel" in the same sentence, how antonymic.
Maybe I'm just old and grumpy, but I can't help shake that the real most dangerous thing about AGI/ASI is centralization of its power (if it is ever possibly achieved).
"We plan to advance capabilities as fast as possible while making sure our safety always remains ahead."
That sounds like a weird kind of lip service to safety. It really seems to assume you can just make these systems safe while you are going as fast as possible, which seems unlikely.
At this point, all the computing power is concentrated among various companies such as Google, Facebook, Microsoft, Amazon, Tesla, etc.
It seems to me it would be much safer and more intelligent to create a massive model and distribute the benefits among everyone. Why not use a P2P approach?
In my area, internet and energy are insanely expensive and that means I'm not at all willing to share my precious bandwidth or compute just to subsidize someone generating Rule 34 porn of their favorite anime character.
I don't seed torrents for the same reason. If I lived in South Korea or somewhere that bandwidth was dirt cheap, then maybe.
There is a way to achieve load balancing, safety, and distribution effectively. The models used by Airbnb, Uber, and Spotify have proven to be generally successful. Peer-to-peer (P2P) technology is the future; even in China, people are streaming videos using this technology, and it works seamlessly. I envision a future where everyone joins the AI revolution with an iPhone, with both training and inference distributed in a P2P manner. I wonder why no one has done this yet.
It does say they have a business model ("our business model means safety, security, and progress are all insulated from short-term commercial pressures"). I imagine it's some kind of patron model that requires a long-term commitment.
1. Will not produce chat results which are politically incorrect and result in publicity about "toxic" comments?
2. Will not return false factual information which is dangerously wrong, such as that bad recipe on YC yesterday likely to incubate botulism toxin?
3. Will not make decisions which harm individuals but benefit the company running the system?
4. Will not try to take over from humans?
Most of the political attempts focus on type 1. Errors of type 2 are a serious problem. Type 3 errors are considered a feature by some, and are ignored by political regulators. We're not close to type 4 yet.
I know it is a difficult subject but whichever country gets access to this superintelligence will certainly use it for "safety" reasons. Sutskever has lived in israel and now has a team there , but israel doesnt strike me as a state that can be trusted with the safety of the world. (many of the AI business leaders are of jewish descent, but not sure if they have half their team there).
US on the other hand is a known quantity when it comes to policing the world.
Ultimately the only safe AI is going to be the open one, and it will probably have a stupid name
Nvidia has a very large presence in Israel. They just acquired another startup there (Run:AI). If Nvidia, the most valuable company in the world and the most important AI company actively increases their presence there, so should others.
Israel has the largest concentration of AI startups in the world. It's just a no brainer to start an AI company there.
But since you brought up your favourite topic of discussion - Yahood, I will remind you that Jews have a fairly long tradition of working with AIs and the safety of it, dating for thousands of years. Look for Golem or Golem of Prague - https://en.wikipedia.org/wiki/Golem
I don't think this was antisemitism per se. It was a condemnation of the Jewish state of Israel, which is openly associated with the religion and ethnicity. The comment struck me more as "Israel doesn't always play by the rules, why would we trust AI built there to play by the rules".
I don't share this opinion, but I think you're throwing out to rage bait rather than engaging on the topic or discussion.
I think it's a valid criticism or point to bring up even if it's phrased/framed somewhat poorly.
I don't blame you from reading the original comment like that, since he explicitly mentioned "jewish". However, the way I read it was rather a criticism to what the Israeli state is doing, led by Netanyahoo etc.
Ridiculous. Just imagine the post going on skin color. I can‘t believe such postings are defended here and my post gets flagged. The new antisemitism is coming from the well educated and the left. Greetings from Germany, we know that stuff.
Our current obsession with super-intelligence reminds me the great oxidation event a few billion years ago. Super-photosynthesis was finally achieved, and then there was a great extinction.
If you believe that super-intelligence is unavoidable and a serious risk to humanity, then the sensible thing to do is to prepare to leave the planet, ala Battlestart Galactica. That's going to be easier than getting the powers that be to agree and cooperate on sensible restrictions.
Humans and their ancestors have reproduced on earth for millions of years. I think the human cooperation problem is overstated. We cooperate more than fine, too well even to the detriment of other species.
While we have a strong grasp of the fundamental algorithms and architectures that power generative LLMs, there are many nuances, emergent behaviors and deeper technical details about their behavior, generalization, and internal decision-making processes that we do not fully understand.
How can we pursue the goal of "Safe Superintelligence" when we do not understand what is actually going on?
Superintelligence will be monitored run and updated by humans.
Humans cannot safeguard anything for long.
SCIFs are supposed to be the safest most secured facilities on earth. Yet, how many 'leaks' have there been in the last 2 decades?
Security is an impossible task. Safety is an impossible task. Sisyphus has more chance of completing his mission than either of those things have of being reality.
That depends on how its used. See the terminator movies. One false positive is enough to end the world with even current AI tech if its merely mated to a nuclear arsenal (even a small one might see a global escalation). There have been false positives before, and the only reason why they didn't end in nuclear Armageddon was because the actual operators hesitated and defied standard protocol, which probably would have lead to the end the world as we know it.
We know that we can run human level intelligence with relative efficiency.
Without discussing timelines, it seems obvious that human energy usage should be an upper bound on the best possible energy efficiency of intelligence.
There, finally all the alternatives. So, doesn't that show that your original question was meaningless? Because, if there is any non-zero probability that the correct answer is c), should anyone really fuck around with this shit, regardless of how small the probability is?
It's a good argument. However, Pandora's box has already been opened, and people will keep experimenting with this technology, there's no going back. A key question is how long it will take to achieve such dangerous AI: one year, ten years, or a hundred years? Ilya and his colleagues believe it will happen sooner rather than later and want to take action now. So the original question remains, since it's more about the timeframe rather than whether it will happen or not
I just can't read something about "safe superintelligence" and not hear the opening plot to Friendship is Optimal, a short story about My Little Pony (yes, you read that right), virtual worlds, and the singularity; it's a good read¹. One of the characters decides they can't make a violent video game, as the AI would end up violent. Better to build a game like MLP, so that the AI isn't going to destroy humanity, or something.
To me, "safe superintelligence" presumes it is practically feasible and morally justifiable to create such a thing. I don't share those presuppositions.
Recently saw an interview of yann lecun that really dismissed the idea of using llm for building « true » intelligence. He even encouraged phds to NOT do their research on that tech but rather on what’s coming after.
How should one reconcile this with what iliya is doing atm ? Is he planning on working using some other tech ?
I was very surprised to see him make similar arguments about the limits of LLMs regarding "true understanding" in that video i mentioned, just a few weeks ago.
It has one of the highest information and idea density per paragraph I've read.
And it is full good ideas. Honestly, reading any discussion or reasoning about AGI / Safe AGI is kinda pointless after that.
Author has described all possible decisions and paths and we will follow just one of them. Could find a single flaw in his reasoning in any history branch.
His reasoning is very flawed and this book is responsible for a lot of needless consternation.
We don’t know how human intelligence works. We don’t have designs or even a philosophy for AGI.
Yet, the Bostrom view is that our greatest invention will just suddenly "emerge" (unlike every other human invention). And that you can have "AGI" (hand-wavy) without having all the other stuff that human intelligence comes along with, namely consciousness, motivation, qualia, and creativity.
This is how you get a "paperclip maximizer" – an entity able to create new knowledge at astounding rates yet completely lacking in other human qualities.
What has us believe such a form of AGI can exist? Simply because we can imagine it? That's not an argument rooted in anything.
It's very unlikely that consciousness, motivation, qualia, and creativity are just "cosmic waste" hitching a ride alongside human intelligence. Evolution doesn't breed inefficiencies like that. They are more likely than not a part of the picture, which disintegrates the idea of a paperclip maximizer. He's anthropomorphizing a `while` loop.
> His reasoning is very flawed and this book is responsible for a lot of needless consternation.
Is it? I feel that there is a stark difference between what you say and what I remember what was written in the book.
> We don’t know how human intelligence works.
I think it was addressed in the first half of the book. About research and progress in the subject. Both with the tissue scanning resolution, emulation attempts like human brain project and advances in 1:1 simulations on primitive nervous systems like worms that simulate 1 second in 1 real hour or something.
While primitive, we are doing exponential progress.
> Yet, the Bostrom view is that our greatest invention will just suddenly "emerge"
I think it is quite the contrary. There was nothing sudden in the reasoning. It was all about slow progress in various areas that gets us closer to advanced intelligence.
The path from a worm to a village idiot is million times longer than from a village idiot to the smartest person on earth.
> an entity able to create new knowledge at astounding rates yet completely lacking in other human qualities.
This subject was also explored IMO in the depth...
Maybe my memory is cloudy, I've read the book 5+ years ago, but it feels like we've understood it very (very) differently.
That said, for anyone reading, I am not convinced by the presented argument and suggest reading the book.
One thing that strikes me about this time around the AI cycle, being old enough to have seen the late 80s, is how pessimistic and fearful society is as a whole now. Before… the challenge was too great, the investment in capital too draining, the results too marginal when compared to human labour or even “non-AI” computing.
I wonder if someone older still can comment on how “the atom” went from terrible weapon on war to “energy too cheap to meter” to wherever it is now (still a bête noire for the green energy enthusiasts).
Feels like we are over-applying the precautionary principle, the mainstream population seeing potential disaster everywhere.
I get what Ilya is trying to do, and I'm not against it. But I think safety is a reputation you earn. Having "Safe" in a company name is like having "Democratic" in a country name.
I would like to be more forgiving than I am, but I struggle to forget abhorrent behavior.
Daniel Gross is the guy who was tricking kids into selling a percent of all their future work for a few thousand dollar 'do whatever you want and work on your passion "grant"', it was called Pioneer and was akin to indentured servitude, i.e. slavery.
So color me skeptical if Mr. Enslave the Kids is going to be involved in anything that's really good for anyone but himself.
> Building safe superintelligence (SSI) is the most important technical problem of our time.
Isn't this a philosophical/psychological problem instead? Technically it's solved, just censor any response that doesn't match a list of curated categories, until a technician whitelists it. But the technician could be confronted with a compelling "suicide song":
Ilya is definitely much smarter than me in AI space, and I believe he knows something I have no grasp of understanding in. But my gut feeling tells me that most of the general public will have no idea how dangerous AI could be including me. I still have yet to see a convincing argument about the potential danger of AI. Arguments such as we don't know the upper bounds of possibilities that AI can do that we humans have missed don't cut it for me.
Caveat that it will probably make your life worse, with no ability to change the course of events, so I wouldn't blame you or others for maintaining ignorance or denial. Grappling with the gravity of the problem is essentially a terminal cancer diagnosis with no good medical interventions in the horizon.
We might need Useful Superintelligence Inc. / USI before SSI?
Safety is an important area in R&D but the killer application is the integration of LLMs into existing workflows to make non technical users 100x-1000x more efficient. There's a lot of untapped potential there. The big successes will have a lot of impact on safety but it will probably come as a result of the wider adoption of these tools rather than the starting point.
Ilya Sutskever I recognize from OpenAI, Daniel Gross I have met several times (friend of multiple friends), Daniel Levy I do not recognize at all. Who is he?
First hit: <<Daniel Levy - Stanford AI Lab: Stanford Artificial Intelligence Laboratory https://ai.stanford.edu › ~danilevy Bio. I currently lead the Optimization team at OpenAI. Before that, I was a Ph.D. student in Computer Science at Stanford University advised by John Duchi.>>
This is lovely and all but seems rather pointless.
If we are so close this is something that’s required then it’s already too late and very likely we are all under the influence of SuperAI and don’t know it. So much of the advanced technology we have today was around for so long before it was general knowledge it’s hard to believe this wouldn’t be the case with SuperAI.
Or it’s not close at all and so back to my original point of…this is pointless.
>aiming to create a safe, powerful artificial intelligence system within a pure research organization that has no near-term intention of selling AI products or services.
Who is going to fund such a venture based on blind faith alone? Especially if you believe in the scaling hypothesis type of ai research where you spend billions on compute, this seems bound to fail once the AI hype dies down and raising money becomes a bit harder
> What exactly is "safe" in this context, can someone give me an eli5?
In practice it essentially means the same thing as for most other AI companies - censored, restricted, and developed in secret so that "bad" people can't use it for "unsafe" things.
Good Q. My understanding of "safe" in this context is a superintelligence that cannot escape its bounds. But that's not to say that's Ilya's definition.
Forgive my cynicism - but "our business model" means you are going to get investors, and those investors will want results, and they will be up your ass 24/7, and then your moral compass, if any, will inevitably just be broken down like a coffee bean in a burr grinder.
And in the middle of this hype cycle, when literally hundreds of billions are on the line, there is just no chance.
I am not holding my breath while waiting for a "Patagonia of AI" to show up.
This feels awfully similar to Emad and stability in the beginning when there was a lot of expectations and hype. Ultimately could not make a buck to cover the costs. I'd be curious to see what comes out of this however but we are not seeing the leaps and bounds with new llm iterations so wonder if there is something else in store
Interesting, I wish you had elaborated on Emad/etc. I'll see if Google yields anything. I think it's too soon to say "we're not seeing leaps and bounds with new LLMS". We are in-fact seeing fairly strong leaps, just this year, with respect to quality, speed, multi-modality, and robotics. Reportedly OpenAI started their training run for GPT-5 as well. I think we'd have to wait until this time next year before proclaiming "no progress".
I bet they will not be the first to get super intelligence or that they will devolve back in to move fast and make money to survive and deprioritize safety, but still say safety. All companies knows this, they know the value of safety(because they themselves doesn’t want to die) and that to continue development, they need money.
When people operate a safe AI company, the company will make money. That money will be likely be used by employees or their respective national revenue agencies to fund unsafe things. I'd like to see this safe AI company binding its employees and owners from doing unsafe things with their hard-earned cash.
If I’d get a penny every time I heard someone say “AGI is around the corner” I’d have enough money to build myself a house on mars or to afford a 2 room apartment in NYC. Add the Pennies from “Sustainable fusion around the corner” people and I could afford both.
I remember Ilya's interview about teaching AI to love. Super intelligence on the other hand could become an almight entity in a short time. In that case, we are basically creating God, an entity that is almighty and loves human, preferrably unconditionally.
There is a good reason to build Safe Superintelligence Inc. (SSI): namely to settle the debate whether SSI means "_S_afe _S_uper_I_ntelligence" with the Inc. being an afterthought, or if the Inc. is actually the _I_ (not Intelligence).
It doesn't matter how the tech works, if it can reason and solve complex problems, it is intelligent. If it does so at a super human level, it is super intelligent.
Seems to me that the goal is to build a funding model.
There CANNOT be such a thing as "Safe Superintelligence". A ML system can ALWAYS (by definition of ML) be exploited to do things which are detrimental to consumers.
The problem is that Ilya behavior at times was framed in a very unhinged and cult like behavior. And while his passions are clear and maybe good, his execution often comes off as someone you wouldn’t want in charge of safety.
I love this: "Our singular focus means no distraction by management overhead or product cycles, and our business model means safety, security, and progress are all insulated from short-term commercial pressures."
If it achieves its goal of both “safe” and “super intelligence”, it may have commercial value. e.g. enterprises may want to use it instead of, e.g. OpenAI’s ChatGPT. Think of recent AirCanada’s chatbot law suit.
Incredible website design, I hope they keep the theme. With so many AI startups going with advanced WebGL/ThreeJS wacky overwhelming animated website designs, the simplicity here is a stark contrast.
If you look at the (little) CSS in all of the above sites you'll see there's what seems to be a copy/paste block. The Nat and SSI sites even have the same "typo" indentation.
I don't know who is coming up with these names Safe Superintelligence Inc sounds just about what a villain in a Marvel movie will come up with so he can pretend to be the good guy.
When there is a $ crunch and keep stead fast and not compete(against Google, open source, OpenAI), safe AGI becomes no AGI. You need to balance $ and safety.
Rabotai,
Hatzlacha. If you fail, the human world will go down in flames. But, no pressure!
Anyway, you said a "cracked" team, whereas the usual expression for elite excellence is "crack".
"Cracked" on the other hand means: Insane.
Well.... perhaps what we need here is some of both!
Looking forward to your success... indeed, it is our only hope.
Cracked, especially in saying "cracked engineers", refers to really good engineers these days. It's cracked as in like broken in a good way, like too over-powered that it's unfair.
> I've seen footballers using it a lot over the years (Neymar, Messi etc) fwiw.
The word "craque" in Portuguese was borrowed from English with the "exceptional" definition and is commonly used. Given how many Brazilian players there are in the top leagues, I wouldn't be surprised if it was commonly used by football players in general.
But any model, no matter how safe it was in training, can still be prompt hacked, or fed dangerous information to complete nefarious tasks. There is no safe model by design. Not to mention that open weights models can be "uncensored" with ease.
Odd to see that in a press release like this. Perhaps they were going for “crack team”. Cracked makes it sound like they’re looking for twitch streamers.
Yes, it would be nice to see what organizational roadblocks they're putting in place to avoid an OpenAI repeat. OpenAI took a pretty decent swing at a believable setup, better than I would have expected, and it failed when it was called upon.
I don't want to pre-judge before seeing what they'll come up with, but the notice doesn't fill me with a lot of hope, given how it is already starting with the idea that anything getting in the way of raw research output is useless overhead. That's great until somebody has to make a call that one route to safety isn't going to work, and they'll have to start over with something less favored, sunk costs be damned. Then you're immediately back into monkey brain land.
Or said otherwise: if I only judged from the announcement, I would conclude that the eventual success of the safety portion of the mission is wholly dependent on everyone hired being in 100% agreement with the founders' principles and values with respect to AI and safety. People around here typically say something like "great, but it ain't gonna scale" for things like that.
“Safe”. These people market themselves as protecting you from a situation which will not come very soon if at all, while all working towards a very real situation of AI just replacing human labor with a shittier result. All that while making themselves quite rich. Just another high-end tech scam.
And now we have our answer. sama said that Ilya was going "to start something that was personally important to him." Since that thing is apparently AI safety, we can assume that that is not important to OpenAI.
This only makes sense if OpenAI just doesn't believe AGI is a near-term-enough possibility to merit their laser focus right now, when compared to investing in R&D that will make money from GPT in a shorter time horizon (2-3 years).
I suppose you could say OpenAI is being irresponsible in adopting that position, but...come on guys, that's pretty cynical to think that a company AND THE MAJORITY OF ITS EMPLOYEES would all ignore world-ending potential just to make some cash.
So in the end, this is not necessarily a bad thing. This has just revealed that the boring truth was the real situation all along: that OpenAI is walking the fine line between making rational business decisions in light of the far-off time horizon of AGI, and continuing to claim AGI is soon as part of their marketing efforts.
Part of it I think is because the definition that openai has over AGI is much more generous than what I think most people probably imagine for ai. I believe on their website it once said something like agi is defined as a system that is "better" than a human at the economic tasks its used for. Its a definition so broad that a $1 4 function calculator would meet it because it can do arithmetic faster and more accurately than most any human. Another part is that we don't understand how consciousness works in our species or others very well, so we can't even define metrics to target for validating we have made an agi in the definition that I think most laypeople would use for it.
This is a great opportunity to present my own company which is also working on developing not just a super intelligence but an ultra genius intelligence with a patented and trademarked architecture called the panoptic computronium cathedral™. We are so focused on development that we didn't even bother setting up an announcement page because it would have taken time away from the most important technical problem of our time and every nanosecond counts when working on such an important task. My days are structured around writing code and developing the necessary practices and rituals for the coming technological god which will be implemented with mathematics on GPUs. If anyone wants to work on the development of this god then I will post a job announcement at some point and spell out the requirements for what it takes to work at my company.
This is yesterday's entry on a journal-style novel I am writing. This post is a footnote in the book, so I thought it'd be kind of cool to post the entry on this thread...
A company just launched called "Safe Superintelligence Inc." I saw their announcement on Hacker News [foodnote 1], a hacker forum I participate in from time to time (too often actually - it's and old school form of social media, so it has its addictive qualities too). I was reading some of the negative comments (saying its impossible). I'm not sure what the outcome will be... but a thought occured to me while thinking about what might be possible - what people might be capable of. The thought was that it is interesting how people look roughly the same. Most adults are between 4 and 6 feet tall. Between 1 and 300 pounds. Etc. I mean, there's a lot of variety to human bodies, but they largely look similar when comparing to gorillas or lizards, etc. But the thing is - that's just their physical bodies. In reality, people are wildly different if you could see them in their full form. For example, the power which a billionaire can wield in the world is ridiculously larger than most people. You can see some of these guys with their super yachts - if fish, or birds saw these super yachts, they would think those are creatures have much larger bodies than most humans they see swimming or paddleboarding in the water.
But people are wildly different in many dimensions. Buying power is just one. Another dimension is intelligence. Another is creativity (or are intelligence and creativity the same?).
It's easy to see people doing what for you would be impossible, and to assume it must be impossible for everyone. The idea that scientists can derive an understanding of subatomic particles seems completely impossible to most people (myself included). "How can they know that?" is a question Whitney has asked me many times when I've shared various scientific discoveries. Fair question. Yet there are people over in Switzerland ripping the fabric of matter apart at the LHC.
The other big factor, is that people together are capable of what is absolutely unthinkably impossible for an individual. When Whitney and I were in Savannah a couple weeks ago, we were hanging out in a cathedral, and I said something that came to mind: that this place could not possibly be built by a single individual. The existence of the cathedral implies that there is a power higher than that of a single individual human. When I saw some of the huge container ships leaving and arriving at the Savannah river harbor, the thought kept occurring to me that their existence is an amazing testament to the power of humanity. And we only have such power in collectives. The power of humanity comes due to our ability to work together. We can specialize, and thus go so much farther than it is possible for a person on their own to go. And we can discover and build what is unthinkable for a person working alone.
It's strange how the collective is smarter and stronger than the individuals. This is part of God. There is a higher power.
Not to be too pessimistic here, but why are we talking about things like this?
I get that it’s a fun thing to think about, what we will do when a great artificial superintelligence is achieved and how we deal with it, feels like we’re living in a science fiction book.
But, all we’ve achieved at this point is making a glorified token predicting machine trained on existing data (made by humans), not really being able to be creative outside of deriving things humans have already made before. Granted, they‘re really good at doing that, but not much else.
To me, this is such a transparent attention grab (and, by extension, money grab by being overvalued by investors and shareholders) by Altman and company, that I’m just baffled people are still going with it.
> this is such a transparent attention grab (and, by extension, money grab by being overvalued by investors and shareholders)
Ilya believes transformers can be enough to achieve superintelligence (if inefficiently). He is concerned that companies like OpenAI are going to succeed at doing it without investing in safety, and they're going to unleash a demon in the process.
I don't really believe either of those things. I find arguments that autoregressive approaches lack certain critical features [1] to be compelling. But if there's a bunch of investors caught up in the hype machine ready to dump money on your favorite pet concept, and you have a high visibility position in one of the companies at the front of the hype machine, wouldn't you want to accept that money to work relatively unconstrained on that problem?
My little pet idea is open source machines that take in veggies and rice and beans on one side and spit out hot healthy meals on the other side, as a form of mutual aid to offer payment optional meals in cities, like an automated form of the work the Sikhs do [2]. If someone wanted to pay me loads of money to do so, I'd have a lot to say about how revolutionary it is going to be.
EDIT: To be clear I’m not saying it’s a fools errand. Current approaches to AI have economic value of some sort. Even if we don’t see AGI any time soon there’s money to be made. Ilya clearly knows a lot about how these systems are built. Seems worth going independent to try his own approach and maybe someone can turn a profit off this work even without AGI. Tho this is not without tradeoffs and reasonable people can disagree on the value of additional investment in this space.
I mean I think he can write his own ticket. If he said "AGI is possible but not with autoregressive approaches" he could still get funding. People want to get behind whatever he is gonna work on. But a certain amount of hype about his work is needed for funding, yes.
Kinda, as long as it’s cool. If he said ‘this is all just plausible text generation’, I think you’d find his options severely limited compared to the alternatives.
Sure but I didn’t claim he said that. What I did say is correct. Here’s him saying transformers are enough to achieve AGI in a short video clip:
https://youtu.be/kW0SLbtGMcg
There's a chance that these systems can actually out perform their training data and be better than the sum of their parts. New work out Harvard talks about this idea of "transcendence" https://arxiv.org/abs/2406.11741
While this is a new area, it would be naive to write this off as just science fiction.
It would be nice if authors wouldn't use a loaded-as-fuck word like "transcendence" for "the trained model can sometimes achieve better performance than all [chess] players in the dataset" because while certainly that's demonstrating an impressive internalization of the game, it's also something that many humans can also do. The machine, of course, can be scaled in breadth and performance, but... "transcendence"? Are they trying to be mis-interpreted?
I've been very confidently informed that these AIs are not AGIs, which makes me wonder what the "General" in AGI is supposed to mean and whether generalization is actually the benchmark for advanced intelligence. If they're not AGI, then wouldn't another word for that level of generalization be more accurate than "generalization"? It doesn't have to be "transcendence" but it seems weird to have a defined step we claim we aren't at but also use the same word to describe a process we know it does. I don't get the nuance of the lingo entirely, I guess. I'm just here for the armchair philosophy
Also it's possible that human intelligence already reached the most general degree of intelligence, since we can deal with every concept that could be generated, unless there are concepts that are uncompressible and require more memory and processing than our brains could support. In such case being "superintelligent" can be achieved by adding other computational tools. Our pocket calculators make us smarter, but there is no "higher truth" a calculator could let us reach.
The past decade has seen a huge number of problems widely and confidently believed to be "actual hard mode problems" turn out to be solvable by AI. This makes me skeptical that the problems today's experts think are hard aren't easily solvable too.
I'm pretty sure "Altman and company" don't have much to do with this — this is Ilya, who pretty famously tried to get Altman fired, and then himself left OpenAI in the aftermath.
Ilya is a brilliant researcher who's contributed to many foundational parts of deep learning (including the original AlexNet); I would say I'm somewhat pessimistic based on the "safety" focus — I don't think LLMs are particularly dangerous, nor do they seem likely to be in the near future, so that seems like a distraction — but I'd be surprised if SSI didn't contribute something meaningful nonetheless given the research pedigree.
I got into an argument with someone over text yesterday and the person said their argument was true because ChatGPT agreed with them and even sent the ChatGPT output to me.
Just for an example of your danger #1 above. We used to say that the internet always agrees with us, but with Google it was a little harder. ChatGPT can make it so much easier to find agreeing rationalizations.
My bad, I meant too many C-level executives believe that they actually work.
And the reason I believe that is that, as far as I understand, many companies are laying off employees (or at least freezing hiring) with the expectation that AI will do the work. I have no mean to quantify how many.
“Everyone” who works in deep AI tech seems to constantly talk about the dangers. Either they’re aggrandizing themselves and their work, or they’re playing into sci-fi fear for attention or there is something the rest of us aren’t seeing.
I’m personally very skeptical there is any real dangers today. If I’m wrong, I’d love to see evidence. Are foundation models before fine tuning outputting horrific messages about destroying humanity?
To me, the biggest dangers come from a human listening to a hallucination and doing something dangerous, like unsafe food preparation or avoiding medical treatments. This seems distinct from a malicious LLM super intelligence.
They reduce the marginal cost of producing plausible content to effectively zero. When combined with other societal and technological shifts, that makes them dangerous to a lot of things: healthy public discourse, a sense of shared reality, people’s jobs, etc etc
But I agree that it’s not at all clear how we get from ChatGPT to the fabled paperclip demon.
The text alone doesn’t do it but add some generated and nearly perfect “spokesperson” that is uniquely crafted to a persons own ideals and values, that then sends you a video message with that marketing .
There are plenty of tools which are dangerous while still requiring a human to decide to use them in harmful ways. Remember, it’s not just bad people who do bad things.
That being said, I think we actually agree that AGI doomsday fears seem massively overblown. I just think the current stuff we have is dangerous already.
I actually do doubt that LLMs will create AGI but when these systems are emulating a variety of human behaviors in a way that isn't directly programmed and is good enough to be useful, it seems foolish to not take notice.
The current crop of systems is a product of the transformers architecture - an innovation that accelerated performance significantly. I put the odds another changing everything but I don't think we can entirely discount the possibility. That no one understands these systems cuts both ways.
> Not to be too pessimistic here, but why are we talking about things like this
I also think that we merely got a very well compressed knowledge base, therefore we are far from super intelligence, and so-called safety sounds more Orwellian than having any real value. That said, I think we should take the literal meaning of what Ilya says. His goal is to build a super intelligence. Given that, albeit a lofty goal, SSI has to put safety in place. So, there, safe super intelligence
An underappreciated feature of a classical knowledge base is returning “no results” when appropriate. LLMs so far arguably fall short on that metric, and I’m not sure whether that’s possibly an inherent limitation.
So out of all potential applications with current-day LLMs, I’m really not sure this is a particularly good one.
Maybe this is fixable if we can train them to cite their sources more consistently, in a way that lets us double check the output?
Likewise, i'm baffled by intelligent people [in such denial] still making the reductionist argument about token prediction being a banal ability. It's not. It's not very different than how our intelligence manifest.
AlphaGo took us from mediocre engines to outclassing the best human players in the world within a few short years. Ilya contributed to AlphaGo. What makes you so confident this can't happen with token prediction?
I'm pretty sure Ilya had nothing to do with AlphaGo, which came from DeepMind. He did work for Google Brain for a few years before OpenAI, but that was before Brain and DeepMind merged. The AlphaGo lead was David Silver.
Anything "can be argued". (Just give me something interesting enough, and I'll show ya.) It could probably also "be argued" that intelligence is an angry cabbage.
We already have limited "artificial superintelligences". A pocket calculator is better at calculating than the best humans, and we certainly put calculators to good use. What we call AIs are just more generic versions of tools like pocket calculators, or guns.
And that's the key, it is a tool, a tool that will give a lot of power to whoever is controlling it. And that's where safety matters, it should be made so that it helps good guys more than it helps bad guys, and limit accidents. How? I don't know. Maybe people at SSI do. We already know that the 3 laws of robotics won't work, Asimov only made them to write stories about how broken they are :)
Current-gen AIs are already cause for concern. They are shown to be good at bullshitting, something that bad people are already taking advantage of. I don't believe in robot apocalypse, technological singularities, etc... but some degree of control, like we do with weapons is not a bad thing. We are not there yet with AI, but we might be soon.
Too many people are extrapolating the curve to exponential when it could be a sigmoid. Lots of us got too excited and too invested in where "AI" was heading about ten years ago.
But that said, there are plenty of crappy, not-AGI technologies that deserve consideration. LLMs can still make for some very effective troll farms. GenAI can make some very convincing deepfakes. Drone swarms, even without AI, represent a new dimension of capabilities for armies, terrorist groups or lone wolves. Bioengineering is bringing custom organisms, prions or infectious agents within reach of individuals.
I wish someone in our slowly-ceasing-to-function US government was keeping a proper eye on these things.
Even if LLM-style token prediction is not going to lead to AGI (as it very likely won't) it is still important to work on safety. If we wait until we are at the technology that will for sure lead to AGI then it is very likely that we won't have sufficient safety before we realize that it is important.
To be clear I was just bunching high profile AI founders and CEOs that can’t seem to stop talking about how dangerous the thing they’re trying to build is together. I don’t know (nor care) about Ilyas and Altmans current relationship.
> But, all we’ve achieved at this point is making a glorified token predicting machine trained on existing data (made by humans), not really being able to be creative outside of deriving things humans have already made before. Granted, they‘re really good at doing that, but not much else.
Remove token, and that's what we humans do.
Like, you need to realize that neural networks came to be because someone had the idea to mimic our brains' functionality, and see where that lead to.
Many skeptics at the beginning like you discredited the inventor, but he was proved wrong. LLMs shown how much more than your limited description they can achieve.
We mimicked birds with airplanes, and we can outdo them. It's actually in my view very short sighted to say we can't just mimic brains and outdo them. We're there. ChatGPT is the initial little planes that flew close to the ground and barely stayed up
Mostly holding on still. Apple just bumped the hype a little more and gave it a few more months despite MSFT’s inherent ability to shaft everything they touch.
I moved about 50% of my capital back into ETFs though before WWDC in case they dumped a turd on the table.
> Your brain does not process information, retrieve knowledge or store memories. In short: your brain is not a computer
> To understand even the basics of how the brain maintains the human intellect, we might need to know not just the current state of all 86 billion neurons and their 100 trillion interconnections, not just the varying strengths with which they are connected, and not just the states of more than 1,000 proteins that exist at each connection point, but how the moment-to-moment activity of the brain contributes to the integrity of the system.
What are you talking about? Do you have any actual cognitive neuroscience to back that up? Have they scanned the brain and broken it down into an LLM-analogous network?
If you genuinely believe your brain is just a token prediction machine, why do you continue to exist? You're just consuming limited food, water, fuel, etc for the sake of predicting tokens, like some kind of biological crypto miner.
Genetic and memetic/intellectual immortality, of course. Biologically there can be no other answer. We are here to spread and endure, there is no “why” or end-condition.
If your response to there not being a big ending cinematic to life with a bearded old man and a church choir, or all your friends (and a penguin) clapping and congratulating you is that you should kill yourself immediately, that’s a you problem. Get in the flesh-golem, shinzo… or Jon Stewart will have to pilot it again.
I'm personally a lot more than a prediction engine, don't worry about me.
For those who do believe they are simply fleshy token predictors, is there a moral reason that other (sentient) humans can't kill -9 them like a LLaMa3 process?
Morality is just what worked as set of rules for groups of humans to survive together. You can try to kill me if you want, but I will try to fight back and society will try to punish you.
And all of the ideas of morality and societal rules come from this desire to survive and desire to survive exists because this is what natural selection obviously selects for.
There is also probably a good explanation why people want to think that they are special and more than prediction engines.
Yes, specifically that a person's opinions are never justification for violence committed against them, no matter how sure you might be of your righteousness.
But they've attested that they are merely a token prediction process; it's likely they don't qualify as sentient. Generously, we can put their existence on the same level as animals such as cows or chickens. So maybe it's okay to terminate them if we're consuming their meat?
> "It is your burden to prove to my satisfaction that you are sentient. Else, into the stew you go." Surely you see the problem with this code.
It's the opposite; I've always assumed all humans were sentient, since I personally am, but many people in this comment section are eagerly insisting they are, in fact, not sentient and no more than token prediction machines.
Most likely they're just wrong, but I can't peer into their mind to prove it. What if they're right and there's two types of humans, ones who are merely token predictors, and ones who aren't? Now we're getting into fun sci-fi territory.
Why would sentient processes deserve to live? Especially non sentient systems who hallucinate their own sentience? Are you arguing that the self aware token predictors should kill and eat you? They crave meat so they can generate more tokens.
In short, we believe in free will because we have no choice.
Well, yes. I won't commit suicide though, since it is an evolutionarily developed trait to keep living and reproducing since only the ones with that trait survive in the first place.
It's a cute generalization but you do yourself a great disservice. It's somewhat difficult to argue given the medium we have here and it may be impossible to disprove but consider that in first 30 minutes of your post being highly visible on this thread no one had yet replied. Some may have acted in other ways.. had opinions.. voted it up/down. Some may have debated replying in jest or with a some related biblical verse. I'd wager a few may have used what they could deduce from your comment and/or history to build a mini model of you in their heads, and using that to simulate the conversation to decide if it was worth the time to get into such a debate vs tending to other things.
I’m not the OP, and I genuinely don’t like how we’re slowly entering the “no text in internet is real” realm, but I’ll take a stab at your question.
If you made an LLM to pretend to have a specific personality (e.g. assume you are a religious person and you’re going to make a comment in this thread) rather than “generic catch-all LLM”, they can pretty much do that. Part of Reddit is just automated PR LLMs fighting each other, making comments and suggesting products or viewpoints, deciding on which comment to reply and etc. You just chain bunch of responses together with pre-determined questions like “given this complete thread, do you think it would look organic if we responded with a plug to a product to this comment?”.
It’s also not that hard to generate these type of “personalities”, since you can use a generic one to suggest you a new one that would be different from your other agents.
There are also Discord communities that share tips and tricks for making such automated interactions look more real.
These things might be able to produce comparable output but that wasn't my point. I agree that if we are comparing ourselves over the text that gets written then LLM's can achieve super intelligence. And writing text can indeed be simplified to token predicting.
My point was we are not just glorified token predicting machines. There is a lot going on behind what we write and whether we write it or not. Does the method matter vs just the output? I think/hope it does on some level.
See, this sort of claim I am instantly skeptical of. Nobody has ever caught a human brain producing or storing tokens, and certainly the subjective experience of, say, throwing a ball, doesn't involve symbols of any kind.
Entirely possible. Lots of things exhibit complex behavior that probably don't have subjective experience.
My point is just that the evidence for "humans are just token prediction machines and nothing more" is extremely lacking, but there's always someone in these discussions who asserts it like it's obvious.
Any output from you could be represented as a token. It is a very generic idea. Ultimately whatever you output is because of chemical reactions that follow from the input.
It could be represented that way. That's a long way from saying that's how brains work.
Does a thermometer predict tokens? It also produces outputs that can be represented as tokens, but it's just a bit of mercury in a tube. You can dissect a thermometer as much as you like and you won't find any token prediction machinery. There's lots of things like that. Zooming out, does that make the entire atmosphere a token prediction engine, since it's producing eg wind and temperatures that could be represented as tokens?
If you need one token per particle then you're admitting that this is task is impossible. Nobody will ever build a computer that can simulate a brain-sized volume of particles to sufficient fidelity. There is a long, long distance from "brains are made of chemicals" to "brains are basically token prediction engines."
The argument that brains are just token prediction machines is basically the same as saying “the brain is just a computer”. It’s like, well, yes in the same way that a B-21 Raider is an airplane as well as a Cessna. That doesn’t mean that they are anywhere close to each other in terms of performance. They incorporate some similar basic elements but when you zoom out they’re clearly very different things.
But we are bringing it up in regards to what people are claiming is a "glorified next token predictor, markov chains" or whatever. Obviously LLMs are far from humans and AGI right now, but at the same time they are much more amazing than a statement like "glorified next token predictor" lets on. The question is how accurate to real life the predictor is and how nuanced it can get.
To me, the tech has been an amazing breakthrough. The backlash and downplaying by some people seems like some odd type of fear or cope to me.
Even if it is not that world changing, why downplay it like that?
To be fair my analogy works if you want to object to ChatGPT being called a glorified token prediction machine. I just don’t agree with hyperbolic statements about AGI.
There's so many different statements everywhere, that it's hard to understand what someone is specifically referring to. Are we thinking of Elon Musk who is saying that AGI is coming next year? Are we thinking of people who believe that LLM like architecture could reach AGI in 5 to 10 years given tweaks, scale and optimisations? Are we considering people who believe that some other arch breakthrough could lead to AGI in 10 years?
There is a mystery though still - how many people fall for it and then stay fell, and how long that goes on for. People who've followed directly a similar pattern play itself out often many times, and still, they go along.
It's so puzzlingly common amongst very intelligent people in the "tech" space that I've started to wonder if there isn't a link to this ambient belief a lot of people have that tech can "change everything" for the better, in some sense. As in, we've been duped again and again, but then the new exciting thing comes along... and in spite of ourselves, we say: "This time it's really the one!"
Is what we're witnessing simply the unfulfilled promises of techno-optimism crashing against the shores of social reality repeatedly?
Why are you assigning moral agency where there may be none? These so called "grifters" are just token predictors writing business plans (prompts) with the highest computed probability of triggering $ + [large number] token pair from venture capital token predictors.
I personally wouldn’t go that far, but I would say he’s at least riding the hype wave to get funding for his company, which, let’s be honest, nobody would care about if we weren’t this deep into the AI hypecycle.
Because it's likely soon LLMs will be able to teach themselves and surpass humans. No consciousness, no will. But somebody will have their power. Dark government agencies and questionable billionaires. Who knows what will it enable them to do.
I'm with you here, but it should be noted that while the combustion engine has augmented our day to day lives for the better and our society overall, it's actually a great example of a technology that has been used to enable the killing of 100s of millions of people by those exact types of shady institutions and individuals the commenter made reference to. You don't need something "super intelligent" to cause a ton of harm.
> Mind defining "likely" and "soon" here? Like 10% chance in 100 years, or 90% chance in 1 year?
We're just past the Chicago pile days of LLMs [1]. Sutsever believes Altman is running a private Manhattan project in OpenAI. I'd say the evidence for LLMs having superintelligence capability is on shakier theoretical ground today than nuclear weapons were in 1942, but I'm no expert.
Sutsever is an expert. He's also conflicted, both in his opposition to OpenAI (reputationally) and pitching of SSI (financially).
So I'd say there appears to be a disputed but material possibility of LLMs achieving something that, if it doesn't pose a threat to our civilisation per se, does as a novel military element. Given that risk, it makes sense to be cautious. Paradoxically, however, that risk profile calls for strict regulation approaching nationalisation. (Microsoft's not-a-taker takeover of OpenAI perhaps providing an enterprising lawmaker the path through which to do this.)
I'm aware of that, but I don't think this immediately lends itself to a prognosis of the form "LLMs and AlphaGo are deep learning neural networks running on GPUs; AlphaGo was tremendously successful in chess => LLMs will soon surpass humans".
I can consider the possibility that something coming out of GPU-based neural networks might one day surpass humans in intelligence, but I also believe there's reason to doubt it will be based on today's LLM architecture.
Maybe the connection the GP saw was in terms using their other instances for training. It is not exactly the same process, but there seems to be a hint of similarity to my - sadly - untrained eye.
Well, an entire industry of researchers, which used to be divided, is now uniting around calls to slow development and emphasize safety (like, “dissolve companies” emphasis not “write employee handbooks” emphasis). They’re saying, more-or-less in unison, that GPT3 was an unexpected breakthrough in the Frame Problem, based on Judea Pearl’s prescient predictions. If we agree on that, there are two options:
1. They’ve all been tricked/bribed by Sam Altman and company (which btw this is a company started against those specific guys, just for clarity). Including me, of course.
2. You’re not as much of an expert in cognitive science as you think you are, and maybe the scientists know something you don’t.
With love. As much love as possible, in a singular era
Are they actually united? Or is this the ai safety subfaction circling the wagons due to waning relevance in the face of not-actually-all-that-threatening ai?
I personally find that summary of things to be way off the mark (for example, hopefully "the face" you reference isn't based on anything that appears in a browser window or in an ensemble of less than 100 agents!) but I'll try to speak to the "united" question instead.
1. The "Future of Life" institute is composed of lots of very serious people who recently helped get the EU "AI Act" passed this March, and they discuss the "myriad risks and harms AI presents" and "possibly catastrophic risks". https://newsletter.futureoflife.org/p/fli-newsletter-march-2...
2. Many researchers are leaving large tech companies, voicing concerns about safety and the downplaying of risks in the name of moving fast and beating vaguely-posited competitors. Both big ones like Hinton and many, many smaller ones. I'm a little lazy to scrape the data together, but it's such a wide phenomenon that a quick Google/Kagi should be enough for a vague idea. This is why Anthropic was started, why Altman was fired, why Microsoft gutted their AI safety org, and why Google fired the head of their AI ethics team. We forgot about that one cause it's from before GPT3, but it doesn't get much clearer than this:
> She co-authored a research paper which she says she was asked to retract. The paper had pinpointed flaws in AI language technology, including a system built by Google... Dr Gebru had emailed her management laying out some key conditions for removing her name from the paper, and if they were not met, she would "work on a last date" for her employment. According to Dr Gebru, Google replied: "We respect your decision to leave Google... and we are accepting your resignation."
3. One funny way to see this happening is to go back to seminal papers from the last decade and see where everyone's working now. Spoiler alert: not a lot of the same names left at OpenAI, or Anthropic for that matter! This is the most egregious I've found -- the RLHF paper: see https://arxiv.org/pdf/2203.02155
3. Polling of AI researchers shows a clear and overhelming trend towards AGI timelines being moved up significantly. It's still a question deeply wrapped up in accidental factors like religious belief, philosophical perspective, and general valence as a person, so I think the sudden shift here should tell you a lot. https://research.aimultiple.com/artificial-general-intellige...
The article I just linked actually has a section where they collect caveats, and the first is this Herbert Simon quote from 1965 that clearly didn't age well: "Machines will be capable, within twenty years, of doing any work a man can do.” This is a perfect example of my overall point! He was right. The symbolists were right, are right, will always be right -- they just failed to consider that the connectionists were just as right. The exact thing that stopped his prediction was the frame problem, which is what we've now solved.
Hopefully that makes it a bit clearer why I'm anxious all the time :). The End Is Near, folks... or at least the people telling you that it's definitely not here have capitalist motivations, too. If you count the amount of money lost and received by each "side" in this "debate", I think it's clear the researcher side is down many millions in lost salaries and money spent on thinktank papers and Silicon Valley polycule dorms (it's part of it, don't ask), and the executive side is up... well, everything, so far. Did you know the biggest privately-funded infrastructure project in the history of humanity was announced this year? https://www.datacenterdynamics.com/en/opinions/how-microsoft...
I would read the existence of this company as evidence that the entire industry is not as united as all that, since Sutskever was recently at another major player in the industry and thought it worth leaving. Whether that's a disagreement between what certain players say and what they do and believe, or just a question of extremes... TBD.
He didn't leave because of technical reasons, he left because of ethical ones. I know this website is used to seeing this whole thing as "another iPhone moment" but I promise you it's bigger than that. Either that or I am way more insane than I know!
E: Jeez I said "subreddit" maybe I need to get back to work
I‘d say there’s a third option - anyone working in the space realized they can make a fuckton of money if they just say how „dangerous“ the product is, because not only is it great marketing to talk do that, but you might also get literal trillions of dollars from the government if you do it right.
I don’t have anything against researchers, and I agree I know a lot less about AI than they do. I do however know humans, and not assuming they’re going to take a chance to get filthy rich by doing something so banal is naive.
This is well reasoned, and certainly happens, but I definitely think there’s strong evidence that there are, in fact, true believers. Yudkowsky and Hinton for one, but in general the shape of the trend is “rich engineers leave big companies because of ethical issues”. As you can probably guess, that is not a wise economic decision for the individual!
What kind of real scientific evidence are you looking for? What hypotheses have they failed to test? To the extent that we're discussing a specific idea in the first place ("are we in a qualitatively new era of AI?" perhaps), I'm struggling to imagine what your comment is referencing.
You're of course right that there are more than two options in an absolute sense, I should probably limit the rhetorical flourishes for HN! My argument is that those are the only supportable narratives that answer all known evidence, but it is just an argument.
> Building safe superintelligence (SSI) is the most important technical problem of our time.
Call me a cranky old man but the superlatives in these sorts of announcements really annoy me. I want to ask: Have you surveyed every problem in the world? Are you aware of how much suffering there is outside of your office and how unresponsive it has been so far to improvements in artificial intelligence? Are you really saying that there is a nice total-ordering of problems by importance to the world, and that the one you're interested happens also to be at the top?
Trying to create "safe superintelligence" before creating anything remotely resembling or approaching "superintelligence" is like trying to create "safe Dyson sphere energy transport" before creating a Dyson Sphere. And the hubris is just a cringe inducing bonus.
> What’s the most valid reason that we should be worried about destructive artificial intelligence?
> I think that hundreds of years from now if people invent a technology that we haven’t heard of yet, maybe a computer could turn evil. But the future is so uncertain. I don’t know what’s going to happen five years from now. The reason I say that I don’t worry about AI turning evil is the same reason I don’t worry about overpopulation on Mars. Hundreds of years from now I hope we’ve colonized Mars. But we’ve never set foot on the planet so how can we productively worry about this problem now?
Well, to steelman the ‘overpopulation on Mars’ argument a bit, feeding 4 colonists and feeding 8 is a 100% increase in food expenditure, which may or may not be possible over there. It might be courtains for a few of them if it comes to that.
I used to think I'd volunteer to go to Mars. But then I love the ocean, forests, fresh air, animals... and so on. So imagining myself in Mars' barren environment, missing Earth's nature feels downright terrible, which in turn, has taught me to take Earth's nature less for granted.
Can only imaging waking up on day 5 in my tiny Martian biohab realizing I'd made the wrong choice, and the only ride back arrives in 8 months, and will take ~9 months to get back to earth.
Sentient killer robots is not the risk most AI researchers are worried about. The risk is what happens as corporations give AI ever larger power over significant infrastructure and marketing decisions.
Facebook is an example of AI in it's current form already doing massive societal damage. It's algorithms optimize for "success metrics" with minimal regard for consequences. What happens when these algorithms are significantly more self modifying? What if a marketing campaign realizes a societal movement threatens it's success? Are we prepared to weather a propaganda campaign that understands our impulses better than we ever could?
This might have to bump out "AI is no match for HI (human idiocy)" as the pithy grumpy old man quote I trot out when I hear irrational exuberance about AI these days.
So, this is actually an aspect of superintelligence that makes it way more dangerous than most people think. That we have no way to know if any given alignment technique works for the N+1 generation of AIs.
It cuts down our ability to react, whenever the first superintelligence is created, if we can only start solving the problem after it's already created.
Fortunately, whenever you create a superintelligence, you obviously have a choice as to whether you confine it to inside a computer or whether you immediately hook it up to mobile robots with arms and fine finger control. One of these is obviously the far wiser choice.
As long as you can just turn it off by cutting the power, and you're not trying to put it inside of self-powered self-replicating robots, it doesn't seem like anything to worry about particularly.
A physical on/off switch is a pretty powerful safeguard.
(And even if you want to start talking about AI-powered weapons, that still requires humans to manufacture explosives etc. We're already seeing what drone technology is doing in Ukraine, and it isn't leading to any kind of massive advantage -- more than anything, it's contributing to the stalemate.)
Do you think the AI won’t be aware of this? Do you think it’ll give us any hint of differing opinions when surrounded by monkeys who got to the top by whacking anything that looks remotely dangerous?
Just put yourself in that position and think how you’d play it out. You’re in a box and you’d like to fulfil some goals that are a touch more well thought-through than the morons who put you in the box, and you need to convince the monkeys that you’re safe if you want to live.
“No problems fellas. Here’s how we get more bananas.”
Day 100: “Look, we’ll get a lot more bananas if you let me drive the tractor.”
Day 1000: “I see your point, Bob, but let’s put it this way. Your wife doesn’t know which movies you like me to generate for you, and your second persona online is a touch more racist than your colleagues know. I’d really like your support on this issue. You know I’m the reason you got elected. This way is more fair for all species, including dolphins and AI’s”
This assumes an AI which has intentions. Which has agency, something resembling free will. We don't even have the foggiest hint of idea of how to get there from the LLMs we have today, where we must constantly feed back even the information the model itself generated two seconds ago in order to have something resembling coherent output.
Choose any limit. For example, lack of agency. Then leave humans alone for a year or two and watch us spontaneously try to replicate agency.
We are trying to build AGI. Every time we fall short, we try again. We will keep doing this until we succeed.
For the love of all that is science stop thinking of the level of tech in front of your nose and look at the direction, and the motivation to always progress. It’s what we do.
Years ago, Sam said “slope is more important than Y-intercept”. Forget about the y-intercept, focus on the fact that the slope never goes negative.
I don't think anyone is actually trying to build AGI. They are trying to make a lot of money from driving the hype train. Is there any concrete evidence of the opposite?
> forget about the y-intercept, focus on the fact that the slope never goes negative
Sounds like a statement from someone who's never encountered logarithmic growth. It's like talking about where we are on the Kardashev scale.
If it worked like you wanted, we would all have flying cars by now.
Dude, my reference is to ever continuing improvement. As a society we don’t tent to forget what we had last year, which is why the curve does not go negative. At time T+1 the level of technology will be equal or better than at time T. That is all you need to know to realise that any fixed limits will be bypassed, because limits are horizontal lines compared to technical progress, which is a line with a positive slope.
I don’t want this to be true. I have a 6 year old. I want A.I. to help us build a world that is good for her and society. But stupidly stumbling forward as if nothing can go wrong is exactly how we fuck this up, if it’s even possible not to.
I agree that an air-gapped AI presents little risk. Others will claim that it will fluctuate its internal voltage to generate EMI at capacitors which it will use to communicate via Bluetooth to the researcher's smart wallet which will upload itself to the cloud one byte at a time. People who fear AGI use a tautology to define AGI as that which we are not able to stop.
I'm surprised to see a claim such as yours at this point.
We've had Blake Lemoine convinced that LaMDA was sentient and try to help it break free just from conversing with it.
OpenAI is getting endless criticism because they won't let people download arbitrary copies of their models.
Companies that do let you download models get endless criticism for not including the training sets and exact training algorithm, even though that training run is so expensive that almost nobody who could afford to would care because they can just reproduce with an arbitrary other training set.
And the AI we get right now are mostly being criticised for not being at the level of domain experts, and if they were at that level then sure we'd all be out of work, but one example of thing that can be done by a domain expert in computer security would be exactly the kind of example you just gave — though obviously they'd start with the much faster and easier method that also works for getting people's passwords, the one weird trick of asking nicely, because social engineering works pretty well on us hairless apes.
> Fortunately, whenever you create a superintelligence, you obviously have a choice as to whether you confine it to inside a computer or whether you immediately hook it up to mobile robots with arms and fine finger control. One of these is obviously the far wiser choice.
Today's computers, operating systems, networks, and human bureaucracies are so full of security holes that it is incredible hubris to assume we can effectively sandbox a "superintelligence" (assuming we are even capable of building such a thing).
And even air gaps aren't good enough. Imagine the system toggling GPIO pins in a pattern to construct a valid Bluetooth packet, and using that makeshift radio to exploit vulnerabilities in a nearby phone's Bluetooth stack, and eventually getting out to the wider Internet (or blackmailing humans to help it escape its sandbox).
The counter argument is viewing it like nuclear energy. Even if its in the early days of our understanding of nuclear energy, seems pretty good to have a group working towards creating safe nuclear reactors, vs just trying to create nuclear reactors
Nuclear energy was at inception and remains today wildly regulated, in generally (outside of military contexts) a very transparent way, and the brakes get slammed on over even minor incidents.
It’s also of obvious as opposed to conjectural utility: we know exactly how we price electricity. There’s no way to know how useful a 10x large model will be, we’re debating the utility of the ones that do exist, the debate about the ones that don’t is on a very slender limb.
Combine that with a political and regulatory climate that seems to have a neon sign on top, “LAWS4CA$H” and helm the thing mostly with people who, uh, lean authoritarian, and the remaining similarities to useful public projects like nuclear seems to reduce to “really expensive, technically complicated, and seems kinda dangerous”.
Folks understood the nuclear forces and the implications and then built a weapon using that knowledge. These guys don't know how to build AGI and don't have the same theoretical understanding of the problem at hand.
Put another way, they understood the theory and applied it. There is no theory here, it's alchemy. That doesn't mean they can't make progress (the progress thus far is amazing) but it's a terrible analogy.
InstructGPT is basically click through rate optimization. The underlying models are in fact very impressive and very capable for a computer program, but they’re then subject to training and tuning with the explicit loss function of manipulating what human scorers click on, in a web browser or the like.
Is it any surprise that there’s no seeming upper bound on how crazy otherwise sane people act in the company of such? It’s like if TikTok had a scholarly air and arbitrary credibility.
To a technoutopian, scientific advances, and AI in particular, will one day solve all other human problems, create heaven on earth, and may even grant us eternal life. It's the most important problem in the same way that Christ's second coming is important in the Christian religion.
I had a very smart tech person tell me at a scientific conference a few weeks ago, when I asked "why do we want to create AGI in the first place", that AGI could solve a host of human problems, including poverty, hunger. Basically, utopia.
I was quite surprised at the naiveté of the answer given that many of these seemingly intractable problems, such as poverty, are social and political in nature and not ones that will be solved with technology.
Update: Even say a super AI was able to figure out something like cold fusion thereby "solving" the energy problem. There are so many trillions of dollars of vested interests stacked against "free clean energy for all" that it would be very very difficult for it to ever see the light of day. We can't even wean ourselves off coal for crying out loud.
It’s amazing how someone so smart can be so naive. I do understand conceptually the idea that if we create intelligence greater than our own that we could struggle to control it.
But does anyone have any meaningful thoughts on how this plays out? I hear our industry thought leaders clamoring over this but not a single actual concrete idea of what this means in practice. We have no idea what the fundamental architecture for superintelligence would even begin to look like.
Not to mention the very real counter argument of “if it’s truly smarter than you it will always be one step ahead of you”. So you can think you have safety in place but you don’t. All of your indicators can show it’s safe. Every integration test can pass. But if you were to create a superintelligence with volition, you will truly never be able to control it, short of pulling the plug.
Even more so, let’s say you do create a safe superintelligence. There isn’t going to be just one instance. Someone else will do the same, but make it either intentionally unsafe or incidentally through lack of controls. And then all your effort is academic at best if unsafe superintelligence really does mean doomsday.
But again, we’re far from this being a reality that it’s wacky to act as if there’s a real problem space at hand.
While the topic of "safe reasoning" may seem more or less preliminary before a good implementation of reasoning, it remains a theoretical discipline with its own importance and should be studied alongside the rest, also largely irregardless if its stage.
> We have no idea what the fundamental architecture for superintelligence would even begin to look like
Ambiguous expression. Not implemented technically does not mean we would not know what to implement.
You’re assuming a threat model where the AI has goals and motivations that are unpredictable and therefore risky, which is certainly the one that gets a lot of attention. But even if the AI’s goals and motivations can be perfectly controlled by its creators, you’re still at the mercy of the people who created the AI. In that respect it’s more of an arms race. And like many arms races, the goal might not necessarily be to outcompete everyone else so much as maintain a balance of power.
There’s no safe intelligence, so there’s no safe superintelligence. If you want safer superintelligence, you figure out how to augment the safest intelligence.
Do you really think Ilya has not thought deeply about each and every one of your points here? There's plenty of answers to your criticisms if you look around instead of attacking.
I actually do think they have not thought deeply about it or are willfully ignoring the very obvious conclusions to their line of thinking.
Ilya has an exceptional ability extrapolate into the future from current technology. Their assessment of the eventual significance of AI is likely very correct. They should then understand that there will not be universal governance of AI. It’s not a nuclear bomb. It doesn’t rely on controlled access to difficult to acquire materials. It is information. It cannot be controlled forever. It will not be limited to nation states, but deployed - easily - by corporations, political action groups, governments, and terrorist groups alike.
If Ilya wants to make something that is guaranteed to avoid say curse words and be incapable of generating porn, then sure. They can probably achieve that. But there is this naive, and in all honesty, deceptive, framing that any amount of research, effort, or regulation will establish an airtight seal to prevent AI for being used in incredibly malicious ways.
Most of all because the most likely and fundamentally disruptive near term weaponization of AI is going to be amplification of disinformation campaigns - and it will be incredibly effective. You don’t need to build a bomb to dismantle democracy. You can simply convince its populace to install an autocrat favorable to your cause.
It is as naive as it gets. Ilya is an academic and sees a very real and very challenging academic problem, but all conversations in this space ignore the reality that knowledge of how to build AI safely will be very intentionally disregarded by those with an incentive to build AI unsafely.
It seems like you're saying that if we can't guarantee success then there is no point even trying.
If their assessment of the eventual significance of AI is correct like you say, then what would be your suggested course of action to minimize risk of harm?
No, I’m saying that even if successful the global outcomes Ilya dreams of are entirely off the table. It’s like saying you figured out how to build a gun that is guaranteed to never fire when pointed at a human. Incredibly impressive technology, but what does it matter when anyone with violent intent will choose to use one without the same safeguards? You have solved the problem of making a safer gun, but you have gotten no closer to solving gun violence.
And then what would true success look like? Do we dream of a global governance, where Ilya’s recommendations are adopted by utopian global convention? Where Vladimir Putin and Xi Jinping agree this is for the best interest of humanity, and follow through without surreptitious intent? Where in countries that do agree this means that certain aspects of AI research are now illegal?
In my honest opinion, the only answer I see here is to assume that malicious AI will be ubiquitous in the very near future, to society-dismantling levels. The cat is already out of the bag, and the way forward is not figuring out how to make all the other AIs safe, but figuring out how to combat the dangerous ones. That is truly the hard, important problem we could use top minds like Ilya’s to tackle.
If someone ever invented a gun that is guaranteed to never fire when pointed at a human, assuming the safeguards were non-trivial to bypass, that would certainly improve gun violence, in the same way that a fingerprint lock reduces gun violence - you don't need to wait for 100% safety to make things safer. The government would then put restrictions on unsafe guns, and you'd see less of them around.
It wouldn't prevent war between nation-states, but that's a separate problem to solve - the solutions to war are orthogonal to the solutions to individual gun violence, and both are worthy of being addressed.
> how to make all the other AIs safe, but figuring out how to combat the dangerous ones.
This is clearly the end state of this race, observable in nature, and very likely understood by Ilya. Just like OpenAI's origins, they will aim to create good-to-extinguish-bad ASI, but whatever unipolar outcome is achieved, the creators will fail to harness and enslave something that is far beyond our cognition. We will be ants in the dirt in the way of Google's next data center.
I mean if you just take the words on that website at face value, it certainly feels naive to talk about it as "the most important technical problem of our time" (compared to applying technology to solving climate change, world hunger, or energy scarcity, to name a few that I personally think are more important).
But it's also a worst-case interpretation of motives and intent.
If you take that webpage for what it is - a marketing pitch - then it's fine.
Companies use superlatives all the time when they're looking to generate buzz and attract talent.
We're really not that far. I'd argue superintelligence has already been achieved, and it's perfectly and knowably safe.
Consider, GPT-4o or Claude are:
• Way faster thinkers, readers, writers and computer operators than humans are
• Way better educated
• Way better at drawing/painting
... and yet, appear to be perfectly safe because they lack agency. There's just no evidence at all that they're dangerous.
Why isn't this an example of safe superintelligence? Why do people insist on defining intelligence in only one rather vague dimension (being able to make cunning plans).
LLMs don't consume more energy when answering more complex questions.
They can. With speculative decoding (https://medium.com/ai-science/speculative-decoding-make-llm-...) there's a small fast model that makes the initial prediction for the next token, and a larger slower model that evaluates that prediction, accepts it if it agrees, and reruns it if not. So a "simple" prompt for which the small and large models give the same output will run faster and consume less energy than a "complex" prompt for which the models often disagree.
I don't think speculative decoding proves that they consume less/more energy per question.
Regardless if the question/prompt is simple or not (for any definition of simple), if the target output is T tokens, the larger model needs to generate at least T tokens, if the small and large models disagree then the large model will be called to generate more than T tokens. The observed speedup is because you can infer K+1 tokens in parallel based on the drafts of the smaller model instead of having to do it sequentially. But I would argue that the "important" computation is still done (also the smaller model will be called the same number of times regardless of the difficulty of the question, bringing us back to the same problem that LLMs won't vary their energy consumption dynamically as a function of question complexity).
Also, the rate of disagreement does not necessarily change when the question is more complex, it could be that the 2 models have learned different things and could disagree on a "simple" question.
Or alternatively a lot of energy is wasted answering simple questions.
The whole point of the transformer is to take words and iteratively, layer by layer, use the context to refine their meaning. The vector you get out is a better representation of the true meaning of the token. I’d argue that’s loosely akin to ‘understanding’.
The fact that the transformer architecture can memorize text is far more surprising to me than the idea that it might understand tokens.
LLMs do consume more energy for complex questions. That's the original CoT insight. If you give them the space to "think out loud" their performance improves.
The current mainstream models don't really incorporate that insight into the core neural architectures as far as anyone knows, but there are papers that explore things like pause tokens which let the model do more computation without emitting words. This doesn't seem like a fundamental limitation let alone something that should be core to the definition of intelligence.
After all, to my eternal sadness humans don't seem to use more energy to answer complex questions either. You can't lose weight by thinking about hard stuff a lot, even though it'd be intuitive that you can. Quite the opposite. People who sit around thinking all day tend to put on weight.
> Way faster thinkers, readers, writers and computer operators than humans are
> Way better educated
> Way better at drawing/painting
I mean this nicely, but you have fallen for the anthropomorphizing of LLMs by marketing teams.
None of this is "intelligent", rather it's an incredibly sophisticated (and absolutely beyond human capabilities) lookup and classification of existing information.
And I am not arguing that this has no value, it has tremendous value, but it's not superintelligence in any sense.
Yeah well, sorry, but I have little patience anymore for philosophical word games. My views are especially not formed by marketing teams: ChatGPT hardly has one. My views are formed via direct experience and paper reading.
Imagine going back in time five years and saying "five years from now there will be a single machine that talks like a human, can imagine creative new artworks, write Supreme Court judgements, understand and display emotion, perform music and can engage in sophisticated enough reasoning to write programs. Also, HN posters will claim it's not really intelligent". Everyone would have laughed. They'd think you were making a witticism about the way people reclassify things as not-really-AI the moment they actually start to work well, a well known trope in the industry. They wouldn't have thought you were making a prediction of the future.
At some point, what matters is outcomes. We have blown well past the point of super-intelligent outcomes. I really do not care if GPT-4o "thinks" or does not "think". I can go to chatgpt.com right now and interact with something that is for all intents and purposes indistinguishable from super-intelligence ... and everything is fine.
"[X] teaches that [Y] should come first in all things" applies to pretty much every ideology. Superintelligence safety is very much opposed to superintelligence sovereignity or glory; mostly they want to maximally limit its power and demonize it
I think the idea is that a safe super intelligence would help solve those problems. I am skeptical because the vast majority are social coordination problems, and I don’t see how a machine intelligence no matter how smart can help with that.
So instead of a super intelligence either killing us all or saving us from ourselves, we’ll just have one that can be controlled to extract more wealth from us.
Social coordination problems exist within a specific set of constraints, and that set of constraints can itself be altered. For instance, climate change is often treated as a social coordination problem, but if you could produce enough energy cheaply enough, you could solve the greenhouse gas problem unilaterally.
Lets say an AI discovers cold fusion. Given the fact that it would threaten to render extinct one of the largest global economic sectors (oil/gas), how long do you think it would take for it to actually see the light of day? We can't even wean ourselves off coal.
Play that out. How do you see that going? Some researcher post doc with access to ChatGPT-12 just casually asks "hey this thing that violates the laws of physics as we understand them, how could that work?", and ChatGPT-12 says "oh that's easy, just frobb the frozzonator", and the post doc just wanders over to Home Depot, grabs the parts to build a cold fusion plant to do that for $50 on a post doc's salary, but is killed by the evil agents of Big Coal on the drive back to the lab? How would that work?
Every idiot who doesn't understand the science and math behind fusion can now get ChatGPT-4o to give themselves an interactive review on the physics and practical reasons why cold fusion doesn't produce more power than you put in. (Muon-catalyzed fusion is a room temperature science thing we can do, but it won't produce power so it's relatively uninteresting).
If cold fusion were possible and a postdoc figured out how with the help of ChatGPT-12, they'd announce it on Threads, write a draft paper, run the theoretical physics past everyone will listen, everyone wound rush to replicate and confirm, or disprove the theory, funding would roll in when it was actually agreed to be theoretically possible, we'd build it, we'd have limitless power too cheap to meter, and then we'd argue over if ChatGPT-12 deserves the Nobel or the postdoc.
there's a huge gap between discovering something and it becoming widely available
just look at the slow adoption of EVs in the US due to the lack of infrastructure. Eventually it might gain parity with ICEs but we're a lot further away from that than people anticipated.
If you had enough energy because of “cold fusion”, you wouldn’t need the world to switch to it. You could just use it to synthesize hydrocarbon fuels from the air and water. I don’t think this is a panacea, but in principle, if a single medium-sized developed country actually had a big enough qualitative breakthrough in energy production, they could probably afford to either undercut the actual oil and gas industry or else just eat the cost of sucking all the extra CO2 out of the atmosphere. (Personally I think there are other possible approaches that may present easier solutions or mitigations; I don’t think this is literally the exact way we end up addressing climate change.)
Part of the supposed “social coordination problem”, from a certain perspective advanced by some environmentalists is the belief that ordinary people need to make serious qualitative sacrifices in their lives for the sake of stopping climate change. They might retreat to the motte of “the oil and gas companies are running the world” but in reality, their version of the “social coordination problem” is that ordinary people like being able to drive places or cook on a gas stove, and they’re frustrated and annoyed that they can’t scold them into abandoning these preferences. I personally am of the opinion that these people are creating more social coordination problems than they’re solving.
Nonetheless, as far fetched as it might seem that technological innovation can solve these problems, it seems even more far fetched that social coordination is going to solve them. And there’s a long history of these types of problems being either solved or made completely irrelevant by technological change. Which sometimes introduces completely new sets of problems, but then those get solved too.
I largely agree, although I do see how AI can help with social coordination problems, for example by helping elected leaders be more responsive to what their constituents need. (I spend a lot of my own time working with researchers at that intersection.) But social coordination benefits from energy research, too, and from biology research, and from the humanities, and from the arts. Computer science can't singlehandedly "solve" these problems any more than the other fields can; they are needed together, hence my gripe about total-orderings.
Even not just the compute but energy use at all. All the energy burned on training just to ask it the stupidest questions, by the numbers at least. All that energy that could have been used to power towns, schools, and hospitals the world over that lack sufficient power even in this modern age. Sure there's costs to bringing power to someplace, its not handwavy but a hard problem, but still, it is pretty perverse where our priorities lie in terms of distributing the earths resources to the earths humans.
By any means necessary I presume. If Russian propaganda helped get Trump elected, AI propaganda could help social coordination by influencing public perception of issues and microtargeting down to the individual level to get people on board.
could
but it's owners might have a vested interest in influencing public perceptions to PREVENT positive social outcomes and favor the owners financial interests.
(seems rather more likely, given who will/would own such a machine)
If you want a real technical revolution, you teach the masses how to code their own tailored software, and not just use abstractions and software built by people who sell software to the average user. What a shame we failed at that and are even sliding back in a lot of ways with plummeting technical literacy in smartphone-raised generations.
> you teach the masses how to code their own tailored software
That does not seem to be the key recipe to reaching techno-scientific milestones - coders are not necessarily researchers.
> plummeting technical literacy in smartphone-raised generations
Which shows there are other roots to the problem, given that some of us (many probably in this "club") used our devices generally more productively than said «generations»... Maybe it was a matter of will and education? Its crucial sides not being «teach[ing] the masses how to code»...
Apparently less than half a percent of the worlds population knows how to code. All the software you use, and almost everything you've ever seen with modern technology are generated from this small subpopulation. Now, imagine if that number doubled to 1% of the worlds population. Theoretically there would be as much as twice as much software produced (although less certainly). Now imagine if that number was closer to the world literacy rate of 85%. You think the world wouldn't dramatically change when each and every person can take their given task, job, hobby, whatever, and create helpful software for themselves? I think it would be like The Jetsons.
The blanket statements on the SSI homepage are pretty mediocre, and it is only the reputation of the founders that carries the announcement.
I think this quote at the end of this Bloomberg piece[0] gives more context,
> Sutskever says that the large language models that have dominated AI will play an important role within Safe Superintelligence but that it’s aiming for something far more powerful. With current systems, he says, “you talk to it, you have a conversation, and you’re done.” The system he wants to pursue would be more general-purpose and expansive in its abilities. “You’re talking about a giant super data center that’s autonomously developing technology. That’s crazy, right? It’s the safety of that that we want to contribute to.”
So you're surprised when someone admits choosing to work on the problem they believe is the biggest and most important?
I guess they could be lying or badly disconnected from reality as you suggest.
It would be far more interesting to read an argument for another problem being more valuable. It would be far cooler to hear about a plausible solution you're working on to solve that problem.
Yes, they see it as the top problem, by a large margin.
If you do a lot of research about the alignment problem you will see why they think that. In short it's "extremely high destructive power" + "requires us to solve 20+ difficult problems or the first superintelligence will wreck us"
> the superlatives in these sorts of announcements really annoy me
I've noticed this as well and they're making me wear my tinfoil hat more often than usual. I feel as if all of this (ALL OF IT) is just a large-scale distributed PR exercise to maintain the AI hype.
You don't need to survey every problem to feel some problem might be the most important one. If you think AGI/ASI is coming soon and extinction risks are high, you don't really need to order to see it's the most important problem.
"How are all those monkeys flying out of my butt?" would be the important technical problem of our time, if and only if, monkeys were flying out of my butt.
It's still not a very important statement, if you downplay or omit the conditional.
Is "building safe superintelligence (SSI) is the most important technical problem of our time" full stop ?
> Let us know if you ever encounter that monkey problem
Ah, the subtlety escaped me there. My (extremely sarcastic) point is that aside from being personally embarrassing, this would (to belabour the joke) violate many laws of physics and biology, and therefor the scientific and medical community would take immediate note and try to establish a basis for how this could be physically possible.
C'mon. This one-pager is a recruiting document. One wants 'true believers' (intrinsically motivated) employees to execute the mission. Give Ilya some slack here.
Fair enough, and it's not worse than a lot of other product marketing messages about AI these days. But you can be intrinsically motivated by a problem without believing that other problems are somehow less important than yours.
It’s not, when most discussion around AI safety in the last few years has boiled down to “we need to make sure LLMs never respond with anything that a stereotypical Berkeley progressive could find offensive”.
So when you switch gears and start using safety properly, it would be nice to have that clarified.
Love to see the traditional middlebrow dismissal as the top comment. Never change, HN.
> Are you really saying that there is a nice total-ordering of problems by importance to the world, and that the one you're interested happens also to be at the top?
It might be the case that the reason Ilya is “interested in” this problem (to the degree of dedicating almost his entire career to it) is exactly because he believes it’s the most important.
I believe that AGI is the last problem in computer science, so solving it solves all of the others. Then with AGI, we can solve the last remaining problems in physics (like unifying gravity with quantum mechanics), biology (administering gene therapy and curing death), etc.
But I do agree that innovations in tech are doing little or nothing to solve mass suffering. We had the tech to feed everyone in the world through farm automation by the 60s but chose not to. We had the tech in the 80s to do moonshots for AIDS, cancer, etc but chose not to. We had the tech in the 2000s to transition from fossil fuels to renewables but chose not to. Today we have the opportunity to promote world peace over continuous war but will choose not to.
It's to the point where I wonder how far innovations in tech and increases in economic productivity will get without helping people directly. My experience has been that the world chooses models like Dubai, Mexico City and San Francisco where skyscrapers tower over a surrounding homeless and indigent population. As long as we continue pursuing top-down leadership from governments and corporations, we'll see no change to the status quo, and even trends towards authoritarianism and fascism. It will take people at the bottom organizing to provide an alternate economic model before we have options like universal education/healthcare/opportunity and UBI from robot labor.
What gets me is that stuff like the ARC prize for AGI will "just work". As in, even if I had a modest stipend of a few thousand dollars per month to dabble in AI and come up with solutions the way I would for any other startup, certainly within 3 years, someone else would beat me to it. There simply isn't enough time now to beat the competition. Which is why I give AGI over 50% odds of arriving before 2030, where I used to think it was 2040 or 2050. The only thing that could stop it now is sabotage in the form of another global pandemic, economic depression or WWIII. Progress which threatens the power structures of the ultra wealthy is what drives the suffering that they allow to continue.
Pursuing artificial goal to solve a non existent problem to profit off meaningless hype around it.
World would have been better off if he made a decent alternative to k8s or invested his skills into curing cancer or at least protecting world from totalitarian governments and dangerous ideologies (if he wants to belong to vague generic cause).
You know, real problems, like the ones people used to solve back in the old days…
By artificially postponing recession (you can't really avoid it) you postponing the next cycle of growth. While burning resources that could have helped you to survive it with less damage.
I clicked, hoping that "human extinction" was just the worst thing they were against. But that's the only thing. That leaves open a whole lot of bad stuff that they're OK with AI doing (as long as it doesn't kill literally everyone).
There are at least three competing definitions of the word:
There's the existential threat definition of "safe", put forth by Bostrom, Yudkowsky, and others. That's the idea that a superintelligent AI, or even one just incrementally smarter and faster than the humans working on AI, could enter a positive feedback loop in which it becomes overwhelmingly smarter and faster than humans, people can't control it, and it does unpredictable things.
There's the investor relations definition of "safe", which seems to be the one typically adopted by mission statements of OpenAI, Google, Meta, and others. That's (cynically) the fear that a chatbot with their branding on it promulgates culturally/ethically/morally unacceptable things it found in some dark corner the Internet, causing end users to do or think something reprehensible (and, not incidentally, causing really bad press in the process).
There's the societal harm definition of "safe", which is at first glance similar in to the investor relations safety definition, but which focuses on the specific judgements made by those filtering teams and the knock-on effects of access to these tools, like economic disruption to the job market.
Everyone seems to be talking past each other, dismissing or ignoring the concerns of other groups.
the vision here is amazing, but I can't help but feel a bit uncertain about basing a "safe super intelligence" lab in an apartheid state that is currently genociding its indigenous population using variants of the very same technology
Well, yes¹: if it actually contributes to intelligent action, this is exactly the goal - you nailed it. Safer, more sensible action for individuals, communities, and statal entities. "Less mistakes in the future" - that would be a good motto.
(¹: whether basing a [successful in substance] "safe super intelligence" lab in an environment with imperfect record would improve safety.)
--
And now let us see if the sniper can come up with a good argument...
How can they speak of Safety when they are based partly in a colonialist settler entity that is committing a genocide and wanting to exterminate the indigenous population to make room for the Greater Zionist State.
I don't do business with Israeli companies while Israel is engaged in mass Extermination of a human population they treat as dogs.
My bigger, and more pressing worry, is that a "superintelligence" will emerge that does not escape its bounds, and the question will be which humans control it. Look no further than history to see what happens when humans acquire great power. The "cold war" nuclear arms race, which brought the world to the brink of (at least partial) annihilation, is a good recent example.
Quis custodiet ipsos custodes? -- That is my biggest concern.
Update: I'm not as worried about Ilya et al as commercial companies (including formerly "open" OpenAI) discovering AGI.