On: MySQL is bazillion times faster than MemSQL

[ChikkaChiChi]

Simple solution:

[1] Someone create a project on GitHub [2] Upload an amply sized CSV full of data [3] Publish list of queries needed to "gauge" speed [4] Allow anyone to post their metrics to see how well tuned any database can be in comparison

The only other criteria would be to ask them to all use the same form factor. EC2 makes the most sense.

[aphyr → ChikkaChiChi]

Shared environments with virtualized IO are not good places to run benchmarks.

[ChikkaChiChi → aphyr]

I couldn't agree more, but its the closest to standardized testing we can get that everyone can use.

I HATE the idea personally, but I cannot think of a better way to start comparing apples to apples.

[aphyr → ChikkaChiChi]

I'd opt for the TPC workloads on standardized hardware: http://www.tpc.org/tpcc/results/tpcc_price_perf_results.asp

[btilly → ChikkaChiChi]

http://www.tpc.org/ already does something like this, but much more rigorously.

[jacques_chester → btilly]

The DBT suite benchmarks are open source versions of the TPC suite.