Show HN: How to use LLMs to generate accurate SQL for real-world data

[zainhoda]

https://github.com/vanna-ai/vanna/blob/main/papers/ai-sql-accuracy-2023-08-17.md

Hey HN,

We are Zain and Ashish, founders of Vanna AI. We recently embarked on an experiment to see if large language models (specifically LLMs) could help in generating SQL queries for real-world datasets. We initially started this project as a web app but realized that it was most useful and had broadest applicability as a Python package since you can then incorporate it into an existing workflow (Jupyter notebook, Slackbot, etc).

We've had some good success with customer datasets but we've generally heard a lot of skepticism so we decided to write a paper about the methodology we're using and how various LLMs compare.

Let us know if you have any questions or requests. The underlying Python package is open source. There is a server component to store and retrieve metadata but by next week there will be a fully open-source and locally runnable version.

Cheers!

[chuckhend]

This looks awesome, great work team! A few questions :)

How complex were the 'static examples' that were used? Can you share the examples of the three that were used in the tests?

Were the "contextually relevant sql" ran in addition to or isolated from the "static examples"?

[zainhoda → chuckhend]

Thanks! You can see the 3 static examples in this notebook here: https://github.com/vanna-ai/research/blob/main/notebooks/tra...

The set of larger SQL queries includes the initial static set but is brought into the prompt only if it's contextually relevant to the question. I know that's a confusing statement so let me know if that makes sense.

[tamiral]

really interesting work ! looking forward to exploring this with some of my datasets. Keep up the great work!