i think if AI figures took their "alignment" concept and really pursued it down to its roots -- digging past the technological and into the social -- they could do some good.
take every technological hurdle they face -- "paperclip maximizers", "mesa optimizers" and so on -- and assume they get resolved. eventually we're left with "we create a thing which perfectly emulates a typical human, only it's 1000x more capable": if this hypothetical result is scary to you then exactly how far do you have to adjust your path such that the result after solving every technical hurdle seems likely to be good?
from the outside, it's easy to read AI figures today as saying something like "the current path of AGI subjects the average human to ever greater power imbalances. as such, we propose <various course adjustments which still lead to massively increased power imbalance>". i don't know how to respond productively to that.
take every technological hurdle they face -- "paperclip maximizers", "mesa optimizers" and so on -- and assume they get resolved. eventually we're left with "we create a thing which perfectly emulates a typical human, only it's 1000x more capable": if this hypothetical result is scary to you then exactly how far do you have to adjust your path such that the result after solving every technical hurdle seems likely to be good?
from the outside, it's easy to read AI figures today as saying something like "the current path of AGI subjects the average human to ever greater power imbalances. as such, we propose <various course adjustments which still lead to massively increased power imbalance>". i don't know how to respond productively to that.