Why not define additional metrics around what you would spot check and then prod...

alok-g 3 months ago | parent | context | favorite | on: Goodhart’s law isn’t as useful as you might think ...

Why not define additional metrics around what you would spot check and then produce a consolidated figure of merit as the metric to optimize. Aren't we doing that with AI training and general optimization (including regularization).