ACL 2023 Tutorial: Retrieval-Based Language Models and Applications

[TalktoCrystal]

https://acl2023-retrieval-lm.github.io/

[cschmidt]

While on the subject of ACL 2023 tutorials, here are two I went to:

Complex Reasoning in Natural Language https://wenting-zhao.github.io/complex-reasoning-tutorial/

Generating Text from Language Models https://rycolab.io/classes/acl-2023-tutorial/

the second one in particular was very well taught. I didn't catch the Retrieval based talk, since it conflicted with the text generation one.

[TalktoCrystal → cschmidt]

This is useful

[monkeydust]

Were using retrieval-based system for Q&A for a complex product. Feeding in user guides, release notes even transcribed some tutorial videos. Its quick to create a 'wow' style demo but putting into production is challenge with hallucinations and testing/benchmarks hard to do given the stochastic nature of the output each time your run with same input (temp>0). Curious on pointers if anyone is doing something similar.

[sharemywin → monkeydust]

I'm curious what LLM your using. I would think some of the open source ones could be trained to use the facts given with much more accuracy.

[monkeydust → sharemywin]

Openai Gpt4. retrieval works best given the data source although debating if we should explore tuning as a next step

[jerpint]

If anyone has any good pointers/references/benchmarks on how to evaluate such systems that would be really helpful

[TalktoCrystal → jerpint]

yes, the market is asking for this.