Because it's likely soon LLMs will be able to teach themselves and surpass humans. No consciousness, no will. But somebody will have their power. Dark government agencies and questionable billionaires. Who knows what will it enable them to do.
I'm with you here, but it should be noted that while the combustion engine has augmented our day to day lives for the better and our society overall, it's actually a great example of a technology that has been used to enable the killing of 100s of millions of people by those exact types of shady institutions and individuals the commenter made reference to. You don't need something "super intelligent" to cause a ton of harm.
> Mind defining "likely" and "soon" here? Like 10% chance in 100 years, or 90% chance in 1 year?
We're just past the Chicago pile days of LLMs [1]. Sutsever believes Altman is running a private Manhattan project in OpenAI. I'd say the evidence for LLMs having superintelligence capability is on shakier theoretical ground today than nuclear weapons were in 1942, but I'm no expert.
Sutsever is an expert. He's also conflicted, both in his opposition to OpenAI (reputationally) and pitching of SSI (financially).
So I'd say there appears to be a disputed but material possibility of LLMs achieving something that, if it doesn't pose a threat to our civilisation per se, does as a novel military element. Given that risk, it makes sense to be cautious. Paradoxically, however, that risk profile calls for strict regulation approaching nationalisation. (Microsoft's not-a-taker takeover of OpenAI perhaps providing an enterprising lawmaker the path through which to do this.)
I'm aware of that, but I don't think this immediately lends itself to a prognosis of the form "LLMs and AlphaGo are deep learning neural networks running on GPUs; AlphaGo was tremendously successful in chess => LLMs will soon surpass humans".
I can consider the possibility that something coming out of GPU-based neural networks might one day surpass humans in intelligence, but I also believe there's reason to doubt it will be based on today's LLM architecture.
Maybe the connection the GP saw was in terms using their other instances for training. It is not exactly the same process, but there seems to be a hint of similarity to my - sadly - untrained eye.
https://en.wikipedia.org/wiki/AlphaGo_Zero