This is interesting work. We at Plasticity (YC S17) have open sourced something similar called Magnitude (https://github.com/plasticityai/magnitude) a few months ago for querying vector embeddings quickly with low memory usage using SQLite and a standard universal file format (.magnitude) between word2vec, fastText, and GloVe. We also added features for out-of-vocabulary lookups, bulk queries, concatenating embeddings from multiple models, etc. It may also be of interest to folks looking at this.
We also have a paper on Magnitude we will be presenting at EMNLP 2018 and ELMo support is coming soon!
It depends what you are optimizing for. It's a little slower for random disk reads, but it doesn't matter given we utilize an in-memory LRU cache and most text has a Zipf-ian distribution which will hit the cache very often even with a small cache size. (see my comment elsewhere on this thread)
Developers are more familiar with SQLite, it's bundled with Python, it is easy to add to nearly every language, and it has specialized features we use like the Full Text Search module for out of vocabulary lookups.
Thanks! Still working on it, but it will embed the ELMo model in the .magnitude file, wrap the ELMo model interface code so that it is standardized with Magnitude's features (querying, concatenating, iterating, etc.) and will use ELMo's OOV method natively.
I have tried plasticity (even submitted a PR*) before putting together this repo but found it horrendously slow in comparison. It really does depend though; as plasticity utilise an in-mem cache if you're querying on many similar words repeatedly, that is likely to be faster.
Note that the original word2vec binary format is extremely bad for low-memory use. It stores the words and vectors interleaved (and words are obviously of a variable length). Of course, you could circumvent this problem by building a separate (sorted) index file.
However, newer formats, such as the fastText binary format, store the embedding matrix contiguously. In such formats you can just memory-map the embedding matrix and you only have to load the vocabulary into memory. This is even simpler than the approach described here and in the Delft README, you don't have any serialization/deserialization overhead [1], you can let the OS decide how much to cache in memory, and you have one dependency less (mmap is in POSIX [2]).
[1] Of course, if you have a system with different endianness, you have to do byte swapping.
The low memory footprint in LMDB is interesting. There were also some exciting papers recently that used compression (vector quantization) [0]. We're still evaluating whether to add those to Gensim [1].
But the embedding matrix will not be stored contiguously, which has benefits, especially when frequent words are stored together (which happens in some implementations, because they sort the vocabulary by frequency), due to page-granularity of mapping, etc. Also, in this particular implementation:
By default, LMDB Embeddings uses pickle to serialize the vectors to bytes (optimized and pickled with the highest available protocol). However, it is very easy to use an alternative approach such as msgpack
You can also speed up the loading of embeddings by using BPE (byte pair encoding) to segment words into a smaller dictionary of char ngrams, and learning ngram embeddings instead of words.
You can replace a list of 500K words with 50K ngrams, and it also works on unseen words and agglutinative languages such as German. It's interesting that it can both join together frequent words or split into pieces infrequent words, depending on the distribution of characters. Another advantage is that the ngram embedding size is much smaller, thus making it easy to deploy on resource constrained systems such as mobile phones.
Neural Machine Translation of Rare Words with Subword Units
Edit: this work uses a different byte-layout + parallel reader that heaves the word vecs into memory as compressed byte arrays. Load time is seconds (haven’t benchmarked with current ssds). Memory footprint is on the order of the size of your word vecs (memory is cheap for me, but could easily be extended to support mem-mapping if memory resources are scarce).
Very nice work! :) Are there any benchmarks available? I'm curious how this compares to caching frequent word vectors (Zipf's law helps here) and disk-seeking the rest.
It actually utilizes an LRU cache (configurable cache size argument on the constructor) so you can utilize an in-memory LRU cache as you query words off-disk using SQLite indexed disk-seeks. And you're right, due to Zipf-ian properties of most text, you can see gains even with a small in-memory cache size :).
I think rocks gives better read+ write performance together whereas lmdb is heavily skewed to read performance in our experience -- which is mirrored by the benchmark.
It's not as clearcut as that. RocksDB gives better write performance up to record sizes of about 1/4 page size. Above that size, LMDB write performance is superior.
We also have a paper on Magnitude we will be presenting at EMNLP 2018 and ELMo support is coming soon!