tantivy has two dictionaries FST and SSTable. We added SSTable in tantivy because it works great with object storage, while FST does not. With some metadata we can download only the required parts and not the whole dictionary.
SStable does not support Regex queries, it would require a full load and scan, which would be very expensive.
Your best bet currently would be to make it work with tokenizing, which is way more efficient anyways.
Hey PSeitz, Meilisearch CEO here. Sorry to hear that you failed to index a low volume of data. When did you last try Meilisearch? We have made significant improvements in the indexing speed. We have a customer with hundreds of gigabytes of raw data on our cloud, and it scales amazingly well. https://x.com/Kerollmops/status/1772575242885484864
Frankly, I'm okay with Meillisearch for instant search because y'all are clear about analytics choices, offer understandable FOSS Rust, and have a non-AGPL license. If/when we make some money, I'm in favor of $upporting and consulting of tools used to keep them alive out of self-interest.
But instead of this, I would prefer some way to just hand it JSON and for it to just index all the fields...
for comparison, this is my meilisearch SDK code:
fun createCustomers() {
val client = Client(Config("http://localhost:7700", "password"))
val index = client.index("customers")
val customers = transaction {
val customers = Customer.all()
val json = customers.map { CustomerJson.from(it) }
Json.encodeToString(ListSerializer(CustomerJson.serializer()), json)
}
index.addDocuments(customers, "id")
}
> It would definitely compress much better than roaring bitmaps. In terms of performance, it depends on the access patterns. If very sparse (large jumps), PEF would likely be faster, if dense (visit a large fraction of the bitmap), it'd be slower.
Just for clarification you mean the access pattern is sparse and not the data itself? How is that relevant?
I did part of the investigation and implementation, but didn't look much into elias fano coding.
The select for the dense codec is already really fast (with popcount), there's not much room for improvement on the instruction side (https://godbolt.org/z/dq7WeE66Y), except implicitly by touching less memory.
The sparse codec with its binary search should be easy to beat though. Partitioned Elias-Fano indexes may be a superior choice in contrast to the sparse codec in terms of rank and compression, and probably less so for select and code complexity.
Roaring bitmaps and similar data structures get their speed from decoding together consecutive groups of elements, so if you do sequential decoding or decode a large fraction of the list you get excellent performance.
EF instead excels at random skipping, so if you visit a small fraction of the list you generally get better performance. This is why it works so well for inverted indexes, as generally the queries are very selective (otherwise why do you need an index?) and if you have good intersection algorithms you can skip a large fraction of documents.
I didn't follow the rest of your comment, select is what EF is good at, every other data structure needs a lot more scanning once you land on the right chunk. With BMI2 you can also use the PDEP instruction to accelerate the final select on a 64-bit block: https://github.com/facebook/folly/blob/main/folly/experiment...
>Roaring bitmaps and similar data structures get their speed from decoding together consecutive groups of elements, so if you do sequential decoding or decode a large fraction of the list you get excellent performance.
EF instead excels at random skipping, so if you visit a small fraction of the list you generally get better performance. This is why it works so well for inverted indexes, as generally the queries are very selective (otherwise why do you need an index?) and if you have good intersection algorithms you can skip a large fraction of documents.
There's no sequential decoding in our variant, every access is independent.
The roaring bitmap variant is used only for the optional index (1 docid => 0 or 1 value) in the columnar storage (DocValues), not for the inverted index. Since this is used for aggregation, some queries may be a full scan.
The inverted index in tantivy uses bitpacked values of 128 elements with a skip index on top.
> I didn't follow the rest of your comment, select is what EF is good at, every other data structure needs a lot more scanning once you land on the right chunk. With BMI2 you can also use the PDEP instruction to accelerate the final select on a 64-bit block
The select for the sparse codec is a simple array index access, that is hard to beat (https://github.com/quickwit-oss/tantivy/blob/main/columnar/s...). Compression is not good near the 5k threshold though.
PDEP is currently deactivated since ryzen before Zen3 was really slow.
Creation speed is also quite important, do you know how "Partitioned Elias-Fano" performs there?
Indeed if you are colorblind that may be an issue. I'll address it If I get to it.
Otherwise it assigns names to colors, no idea how you could be confused by that.
SStable does not support Regex queries, it would require a full load and scan, which would be very expensive.
Your best bet currently would be to make it work with tokenizing, which is way more efficient anyways.
prefix queries are supported btw