mpmisko's comments

On: Show HN: I put PubMed in a vector DB

[mpmisko]

GPT-based search engines usually use some sort of a database to retrieve context for the LLM to summarize first. This is what people refer to as RAG these days: https://blogs.nvidia.com/blog/what-is-retrieval-augmented-ge....

Some of these GPT engines maintain their own vector DB to do semantic search, others are directly hooked into Bing / Google. So pubmedisearch.com would be one component of a GPT-based engine. We actually have a GPT-based engine here: https://medisearch.io/.

On: Show HN: I put PubMed in a vector DB

[mpmisko]

Lots of annoying edge cases as you can imagine, nothing particularly glamorous.

On: Show HN: I put PubMed in a vector DB

[mpmisko]

Done! Let me know if you have other feedback.

On: Show HN: I put PubMed in a vector DB

[mpmisko]

Thanks! Looks quite relevant

On: Show HN: I made a spaced repetition tool to master...

[mpmisko]

Training for multiple epochs is a bit like that :)

On: Show HN: I put PubMed in a vector DB

[mpmisko]

We use pinecone and it is not ideal, looking at https://turbopuffer.com/ now. They look quite promising :)

[yumraj → mpmisko]

Did you compare pinecone against pgvector with Postgres? Self hosted of course

[cchance → yumraj]

Isn't it funny how the best Choice somehow always comes back to Postgres in the end XD (for most)

[yumraj → cchance]

Yes, that’s where I’m these days. I don’t even think of venturing outside of Postgres these days, except for say things like Redis etc. where there are mature and established options for specific use cases.

[mpmisko → yumraj]

Will definitely check pgvector, thanks for the pointer.

[cchance → mpmisko]

What kinda dimensions did you keep it relatively low to keep costs down?

On: Show HN: I put PubMed in a vector DB

[mpmisko]

1. We cover all the articles on PMC. The exact cost is hard to estimate because we did a lot of iterations.

2. We do weight those ... it is a lot of trial and error and you have to have good & exhaustive benchmarks.

On: Show HN: I put PubMed in a vector DB

[mpmisko]

Yes

On: Show HN: I put PubMed in a vector DB

[mpmisko]

Glad you like it! I did this as a mini-project within our startup MediSearch (https://medisearch.io/) & the search pipeline is custom tuned for the problem.

On: Show HN: I put PubMed in a vector DB

[mpmisko]

Hi, it currently does not support search by PMID. But you can find the paper included in the results here (5th place):

https://pubmedisearch.com/share/Do%20some%20individuals%20wi...