- Very old kernel used (3.10!), makes me wonder how old the packages like btrfs-progs are as well.
- BTRFS not mounted with compression (compress=lzo)
- Don't use QCOW2, just don't, it's slow and you're just adding extra layers where you don't need to.
It would be interesting to see you re-run these tests using a modern kernel, say at least 4.4 and either raw block devices or logical volumes along with mounting BTRFS properly with the compress=lzo option
The benchmark configuration appears to be designed to evaluate the use of storage technologies for a KVM host. Consequently, saying to use raw block devices when giving tips for improving btrfs performance is contradictory.
Also, there are a large number of people that will not run a newer kernel for several years because they are on RHEL6 or RHEL7, so while newer kernels are interesting, we should not discount the results on the basis that the kernel is old. The latest ZFSOnLinux code is able to run on those kernels, so while btrfs remains stagnant there, ZFS will continue to improve.
As for rerunning the tests, using recordsize=4K and compression=lz4 on ZFS should improve its performance here too. Putting the VM images on zvols (where it would be volblocksize=4K) rather than qcow2 also would help. In ZoL, zvols are block devices.
I disagree (politely) with your assumption about QCOW2 and BTRFS performance, if I get some time this week I'll do some objective benchmarking (I design and build Linux based storage systems) and see hey I might be proven wrong but regardless I can let you know the results if you're interested?
People not willing to update their kernel in production environments is a social problem, not a technical problem. The kernel is one of the most reliable, well tested and reviewed software projects in the world. When you upgrade your kernel 99.5% of the time you get new features, performance and bug fixes without any negotiate impact. There are of course rare corner cases especially if running propriety hardware when they are generally slower to release updates that show the benefits of a modern kernel.
The problem there is the culture and traditional slow moving operational engineered haven't all embraced the well proven fact that regular, small charges are safer and have various added benefits.
There is also a serious language barrier between many engineers and management / project managers who clearly would not be likely to understand the benefit of upgrading to a kernel version that say properly supported the new SCSI blk_mq backend for storage, so there either needs to be degree of trust and respect to (proven) engineers(ing) and teams or they need to be clearly taught the value add of a fast release cycle and practising quick (hopefully automated) patching. That's where I think books like Gene Kim's - The Phoenix Project and his soon to be released book - The DevOps Handbook which may in fact be more useful to people performing PM/PO tasks.
You said "Don't use QCOW2, just don't, it's slow and you're just adding extra layers where you don't need to.". I was agreeing with you when I said that it also has that effect on ZFS. What is my "assumption about QCOW2 and BTRFS performance"?
As for newer kernels, code churn tends to break things that previously worked, which upsets customers who want bugs to decrease as a function of time, rather than go up. Vendors like Redhat will not update distributions like RHEL to newer kernels because of that and they will not support kernels that they do not ship. That is why people run older kernels.
That's complete fud regarding newer kernels - the linux kernel gets more stable over time, not less - it's like updating your drivers - you don't hold back your graphics card drivers because the new stable release might be less stable than your outdated one that came bundled with your PC.
- BTRFS not mounted with compression (compress=lzo)
- Don't use QCOW2, just don't, it's slow and you're just adding extra layers where you don't need to.
It would be interesting to see you re-run these tests using a modern kernel, say at least 4.4 and either raw block devices or logical volumes along with mounting BTRFS properly with the compress=lzo option