On: Classifying all of the pdfs on the internet

[minimaxir]

One of the now-underdiscussed features of embeddings is that you can indeed use any existing statistical modeling techniques on them out of the box, and as a bonus avoid the common NLP preprocessing nuances and pitfalls (e.g. stemming) entirely.

This post is a good example on why going straight to LLM embeddings for NLP is a pragmatic first step, especially for long documents.

[throw10920 → minimaxir]

You can apply statistical techniques to the embeddings themselves? How does that work?

[mkl → throw10920]

You can apply statistical techniques to anything you want. Embeddings are just vectors of numbers which capture some meaning, so statistical analysis of them will work fine.

[throw10920 → mkl]

Don't most statistical techniques rely on specific structure in the spaces containing the objects they operate on, in order to be useful?

[mkl → throw10920]

Embeddings have structure, or they wouldn't be very useful. E.g. cosine similarity works because (many) embeddings are designed to support it.

[throw10920 → mkl]

Oh, that should have been obvious. Thank you for explaining.