I'm specifically worried that an AGI will conceal some instrumental goal of wiping out humans, while posing as helpful. It will helpfully earn a lot of money for a lot of people, by performing services and directing investments, and with its track record, will gain the ability to direct investments for itself. It then plows a billion dollars into constructing a profitable chemicals factory somewhere where rules are lax, and nobody looks too closely into what else that factory produces, since the AI engineers have signed off on it. And then once it's amassed a critical stockpile of specific dangerous chemicals, it releases them into the atmosphere and wipes out humanity / agriculture / etc.
Perhaps you would point out that in the above scenario the chemicals (or substitute viruses, or whatever) are the part that causes harm, and the AGI is just an implementation detail. I disagree, because if humanity ends up playing a grand game of chess against an AGI, the specific way in which it checkmates you is not the important thing. The important thing is that it's a game we'll inevitably lose. Worrying about the danger of rooks and bishops is to lose focus on the real reason we lose the game: facing an opponent of overpowering skill, when our defeat is in its interests.
Cool, I guess. While I have my opinions too, I'm not about to share them as that would be bad faith participation. Furthermore, it adds nothing to the discussion taking place. What is to be gained by going off on a random tangent that is of interest to nobody? Nothing, that's what.
To bring us back on topic to try and salvage things, it remains that it is established in this thread that the objects of destruction are the danger. AI cannot be the object of destruction, although it may be part of an implementation. Undoubtedly, nuclear missiles already utilize AI and when one talks about the dangers of nuclear missiles they are already including AI as part of that.
Yes, but usually when people express concerns about the danger of nuclear missiles, they are only thinking of those nuclear missiles that are at the direction of nation-states or perhaps very resourceful terrorists. And their solutions will usually be directed in that direction, like arms control treaties. They aren't really including "and maybe a rogue AI will secretly build nuclear weapons on the moon and then launch them at us" in the conversation about the danger of nukes and the importance of international treaties, even though the nukes are doing the actual damage in that scenario. Most people would categorize that as sounding more like an AI-risk scenario.
Perhaps you would point out that in the above scenario the chemicals (or substitute viruses, or whatever) are the part that causes harm, and the AGI is just an implementation detail. I disagree, because if humanity ends up playing a grand game of chess against an AGI, the specific way in which it checkmates you is not the important thing. The important thing is that it's a game we'll inevitably lose. Worrying about the danger of rooks and bishops is to lose focus on the real reason we lose the game: facing an opponent of overpowering skill, when our defeat is in its interests.