I agree with your broad point however I think the proof example is off.
Finding a proof is the hard part. Verifying a proof is the easier part. So the AI could search for likely proofs and we can use other mechanisms to reliably verify them. This doesn’t require high accuracy in the search stage to be reliable overall.
Finding a proof is the hard part. Verifying a proof is the easier part. So the AI could search for likely proofs and we can use other mechanisms to reliably verify them. This doesn’t require high accuracy in the search stage to be reliable overall.