On: Wikipedia search-by-vibes through millions of page...

[pizza]

FWIW sentence-transformers truncates the input to at most 256 tokens by default; you might just be embedding the first paragraph or so.

[lsb → pizza]

I average the embeddings of every 512 bytes of the page text

[theolivenbaum → lsb]

That might actually be making things worse for longer articles. It probably would be better would be to index them separately and aggregate back to article level post query

[pizza → lsb]

Ah ok, that makes sense