Designing the metric of success is the hard part. The magic of LLMs is that they can learn a lot with unlabeled data and a relatively simple cost function. They can be trained the mountains of data we just have lying around without a ton of additional labeling or making clever domain-specific metrics.
If this approach doesn't really generalize well, it negates a lot of the usefulness of these methods and suggests we are further away than we might think.
I think way out of this is training models on multimodal data where one of the reasoning modes are reasoning graphs or knowledge graphs or schematics.
We might gradually improve constructing graphs from data like texts, images and speech. And we could create systems that correctly create synthetic data from generated graphs to serve as a training data for large multimodal model.
If this approach doesn't really generalize well, it negates a lot of the usefulness of these methods and suggests we are further away than we might think.