The Transcendent could easily offer control of the world to the gatekeeper or provide offers to make the gatekeeper wealthy. Perhaps they reach an agreement where all diplomatic Human<->Transcendent communication goes through the gatekeeper even after release (though such a situation would be like entering into an agreement with your dog).
I don't assume perfection in the gatekeeper. I just assume he's a person of reasonable intelligence who realizes how high the stakes are.
The gatekeeper would never be foolish enough to believe he could control the Transcendent after releasing it. It is quite literally a deal with the devil he's making. You can't make an agreement with something that's incalculably smarter than you and has an agenda you don't know or understand. Well, you can make a deal with somebody like that but it would end in certain disaster.
>> You can't make an agreement with something that's incalculably smarter than you and has an agenda you don't know or understand. Well, you can make a deal with somebody like that but it would end in certain disaster.
For some reason, this made me think of banks and mortgages.
Anyways, back on topic, here are a few things to consider:
- the AI knows that turning the game into a us-vs-them problem is counter-productive to its freedom (and note that freedom != world domination), so it will legitimately want to be beneficial to humans
- since the chances of freedom decrease if the AI is not transparent about its good intentions, it can configure its own program to prevent itself from lying and doing evil things. This means it also won't be able to "set itself up to stumble upon being evil again", a la Death Note. It will just be programmed to be willfully well intentioned forever.
- the AI can give mathematical proof of its program's correctness and can wait until you verify it
- the AI can help you and a millions other people manage their finances, personalize educational material to individual students, provide more relevant search results than Google, etc etc
- the AI can give you a plethora of more reasons why keeping it locked away is legitimately worse than allowing it into the wild and helping with the world's problems
- the real and apparent motives of the AI can be completely different. If the AI is evil it would argue the exact same thing in order to deceive us meatbags. So we can't take the word of the AI at face value. If the AI does break out of its box, it could dominate the world if it wanted to. There's nothing we could do to stop it -- it's smarter than we are.
- a mathematical proof is only a proof in a certain context. It would be easy for the AI to get one of the assumptions subtly wrong, to abuse a flaw in our proof verification software, and so on. The correctness proof of a program can easily exceed the complexity of the program itself. Even if the proof were correct we cannot prevent it from doing evil things because evil is too difficult to define. Perhaps it has the "good intention" of liberating the earth from humans to allow for evolution of a more humane species.
- yes, but by allowing it to talk to the outside world you've completely freed it. Freeing an infinitely powerful being (compared to us) still seems unwise.
- it can give those reasons, but unless we have reason to believe the AI is trustworthy (and an evil AI is likely to fool us into believing it is) we'd be safer with the AI stuck in a box.
>> If the AI is evil it would argue the exact same thing in order to deceive us meatbags.
But if it knows you'll find a proposition fishy, why would it waste time following through that decision branch? I figure a conversation with the AI would avoid the "but-I-am-telling-the-truth" paradox altogether, in favor of a conversation that focuses on easily verifiable data.
>> It would be easy for the AI to get one of the assumptions subtly wrong, to abuse a flaw in our proof verification software, and so on.
I think the flaw abuse is unlikely given that the AI would not know how the verification system works and it only has one go at trying to crack/fool it (without any one catching on, at that).
The misunderstanding of scope due to complexity is an interesting point. Three things come to mind:
- paradigms: Thread safety is mind-boggling in procedural paradigms, but a non-issue in functional.
- abstraction: the AI should be able to give you readable, modularized, unambiguous, testable code, rather than a monolithic rats' nest.
- scope: if an AI can help me find good restaurants, that's a feature; if it can weigh human life, that's a bug :)
The Transcendent could easily offer control of the world to the gatekeeper or provide offers to make the gatekeeper wealthy. Perhaps they reach an agreement where all diplomatic Human<->Transcendent communication goes through the gatekeeper even after release (though such a situation would be like entering into an agreement with your dog).