Baidu's Improving Retrieval Augmented Language Model with Self-Reasoning

[a-s-k-af]

https://arxiv.org/abs/2407.19813

[kaspermarstal]

Can anyone explain what is gained by training a model? Why not use the foundational LLM for the relevance, evidence, and trajectory processes?

[a-s-k-af → kaspermarstal]

I assume you are referring to fine tuning a model here?

[Tostino → a-s-k-af]

You could also just continue pre-training of an existing foundation model. Would still be cheaper by not starting from zero.

[a-s-k-af → Tostino]

The amount of accuracy while doing fine tuning or distillation is usually better than pre-training an existing model, not to mention the graph against the cost.