We’re already training models based on ML models of what human feedback would be for a response. I see no difference between that and training models on inputs that a model estimates are desirable to humans.
This is classic “processor design will hit a wall when they get too complex for humans to understand every gate” criticism: it assumes limits on a tech advance, when the limits themselves are being wiped out by the advance.
Sole humans can still trace every gate on a modern chip, or even the gate's components, and how exactly they work, to some explicit specification. It's completely deterministic and a human can fully understand all details. Otherwise it wouldn't be possible to debug it, or manufacture it in the first place.
OTOH we can't track or understand the results of AI. It spits out results but we can't understand how exactly it deduced those results, usually. That's the unfortunate current state actually.
So the analogy doesn't hold.
But this seems not even relevant. Because no level of technological progress will invalidated the old "garbage in, garbage out" principle. If you make a loop out of it, there's more or less only one thing to expect…
An AI would need to know how to teach itself "reasonable things". But to be able to do so it would need human level intelligence.
Growing AI is likely like growing children: Children can't teach themself in the beginning. You need to teach them some basics before they (maybe) learn how to teach themself. AI is not even near to "know" any basics. So it will need a lot of well thought out human input before it can proceed to teach itself.
We’re already training models based on ML models of what human feedback would be for a response. I see no difference between that and training models on inputs that a model estimates are desirable to humans.
This is classic “processor design will hit a wall when they get too complex for humans to understand every gate” criticism: it assumes limits on a tech advance, when the limits themselves are being wiped out by the advance.