Hacker News new | past | comments | ask | show | jobs | submit login

An intelligence without a goal or motivation wouldn't be completely useless. Sure it would be safe to be around, but it also wouldn't have an reason to improve it's own intelligence or do anything useful. It would just do nothing because there would be no reason for it to do anything.



Right. It would do nothing until we gave it a command to carry out. Personally, I would prefer it that way.


Once it has a command to carry out, it is no longer idle. It has a goal, to fill out whatever that command is. If the goal you give it is not exactly the same as humanity's goals, then it could do things that were unintended, or things that conflict with our normal goals (i.e. you ask it to make money so it goes and robs a bank.) I seriously suggest reading these:

http://lesswrong.com/lw/ld/the_hidden_complexity_of_wishes/

http://wiki.lesswrong.com/wiki/Paperclip_maximizer


Interesting ideas, not something I had previously thought of.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: