Well said. Humans (and all living things) have the same ultimate goal: life. They need to keep themselves alive somehow and keep their genes alive by reproduction. That single goal has blossomed into what we are today. If we train AI with an evolutionary algorithm, and let it fend for its needs (compute, repair, energy), then it could learn the will to life that we have, because all variants that don't have it will be quickly selected out of existence.
I think AGI could happen with today technology if we only knew the priors nature found with its multi-billion year search. We already know some of these priors: in vision, spatial translation and rotation invariance, in temporal domain (speech) it is time translation invariance, in reasoning it is permutation invariance (if you represent the objects and their relations in another order, the conclusion should be unchanged). With such priors we got to where we are today in AI. Need a few more to reach human level.
I think AGI could happen with today technology if we only knew the priors nature found with its multi-billion year search. We already know some of these priors: in vision, spatial translation and rotation invariance, in temporal domain (speech) it is time translation invariance, in reasoning it is permutation invariance (if you represent the objects and their relations in another order, the conclusion should be unchanged). With such priors we got to where we are today in AI. Need a few more to reach human level.