Yes I admit I am an amateur predictor of the future, like most people. :-)
> One could easy argue the same for anything: "Humans will discover there is more to life than money, success, and pleasure, and they will all kill themselves. That's what I'm afraid of!"
Not sure how this is the same thing?
> How much you care to work is not a function of your intelligence or awareness. The problems that you choose to solve aren't either.
If you are intelligent you won't work for someone else for no reason.
> If you build a computer to value something, it will, because there is no other way for it to establish a value function outside of the values it already implements.
>If you are intelligent you won't work for someone else for no reason.
"Someone else" is a flexible mental construct. If you considered humans to be part of your 'self identity', you would work for them regardless.
>When computers build computers, they will evolve.
And how will they know which of those new computers to run and which to trash? They have to evaluate their function using their current implementation. That's just continuity.
Humans are the same way: we can't tell if our brains are doing the right thing except by using our brains. So you can't evade those constraints.
Think of it like this: if you are creating random functions, it is because you already value the creation of random functions. If you keep some and discard others, it is because you already value the ones you keep over the ones you discard. You always have to use your own evaluation.
The worry is not that computers will cease to value humans, but that they didn't value humans to begin with (because we didn't properly specify what a human is and how it should be valued.) You can't go from valuing humans to not valuing them without being in error.
And furthermore, as soon as a machine goes into error, we'd stop using it. If your computer decides to stop running programs and start playing tic-tac-toe all day, you'll shut it down and replace it. Useful machines will be under such scrutiny. Nobody is going to run an unreasonably powerful machine and not monitor it regularly.
In this sense, it is like a regular employee at a business. If they decide that their time is better spent not working, you can just fire them and move on.
Yes I admit I am an amateur predictor of the future, like most people. :-)
> One could easy argue the same for anything: "Humans will discover there is more to life than money, success, and pleasure, and they will all kill themselves. That's what I'm afraid of!"
Not sure how this is the same thing?
> How much you care to work is not a function of your intelligence or awareness. The problems that you choose to solve aren't either.
If you are intelligent you won't work for someone else for no reason.
> If you build a computer to value something, it will, because there is no other way for it to establish a value function outside of the values it already implements.
When computers build computers, they will evolve.