On: Searching 20 GB/sec: Systems Engineering Before Al...

[ddorian43]

Look at their pricing. The best gb/$ public plan, is 500$ month for (worst case scenario) 300gb of data at all time.

With that price, you can keep most of the data in memory-ssd and don't care to use fancy-algorithm.

But what if the pricing was 10x lower ?

[jcampbell1 → ddorian43]

300 GB of ram, is going to cost ~ $3000/month.

Reading 300GB from an SSD is going to be way too slow. It will take minutes per search. In a simple test on linode, the SSD seems to read at about 1GB/s.

Their pricing seems like a bargain to me.

[Wilya → jcampbell1]

300GB of ram might cost $3000/month on AWS. There are a few dedicated server providers that offer 384GB or even 512GB configs for less than 1000$/month.

[herf → Wilya]

lz4 typically compresses logs 6:1 (note: the "high compression" mode allows fast searching.)