I've been using btrfs since about 2011, and I've stopped using ext4 / xfs / zfs everwhere since about 2014.
From 2012-2014 it was mostly breakage every other month. From 2014-2016, it was semi-annual issues.
For the last ~18 months I have had ~30 machines running btrfs with no issues, some servers, some personal computers. The release notes are boring, the bugs are boring, and to me its definitely in a state I would strongly consider trusting it with any workload.
I worry that btrfs is just going to remain doomed. It wasn't stable half a decade ago, so it - for some reason - cannot be more stable now. But it has seen so much work put into it to make it as mature as it is now, and in my experience it is pretty damn mature now. All I want to see is another year and a half of perfect stability before I would start arguing to drop zfs entirely.
Are you running BTRFS with its built in RAID? That's been the biggest blocker for me. There have been numerous RAID bugs that have caused data-loss and I believe at least one of them is still unpatched.
My main issue isn't actually a single thing that's wrong -- it's the completely and utterly haphazard way many of the features in btrfs have been "designed"[1]. Some of these problem they've had seem, to me at least, to be a fundamental lack of a coherent design. That does not bode well for stability, even 10(?) years after its first version.
bcachefs seems to have a much more coherent design.
[1] "Oh, yeah, I don't know how to handle this code path yet, let's stick a BUG_ON in there! I'm sure we'll figure something out later."
As far as I know they consider RAID0, 1, and 10 to be stable. Last time I used it rebuilds were substantially slower than ZFS or mdraid. Rebuild performance seems to one of a few issues that BTRFS has had trouble solving. RAID 5 and 6 were declared stable last year, only to have that retracted when some fatal flaw was discovered that would apparently cause data loss if you needed to rebuild.
It's mostly true but the issues as far as I know are:
- RAID 1 with more than 2 disks is not what you think it is: the data will be mirrored but only once, no matter how many disks you have (meaning if you have a mirror with 3 disks, you only have 2 copies of your data). Because in BTRFS lingo, RAID 1 means '2 copies of the data' https://btrfs.wiki.kernel.org/index.php/FAQ#What_are_the_dif... which is not what people expect from RAID 1 with more than 2 disks
I've been using btrfs since about 2014 on CentOS7. I only use it with mirroring+compression. No snapshots. I mostly went with it for data scrubbing and compression.
My experience has been mixed, but haven't had any data loss. There was a bug for a while regarding free space so occasionally the system would seem to be full but wasn't ....and it was a real pain to correct it.
I now have a cron job that does a monthly btrfs balance along with a mount -oremount,clear_cache. I also run the latest kernels from http://elrepo.org instead of the CentOS7 kernels so that I get the latest patches.
ZFS is not GPL? I dunno, I like feeling safe when I do kernel upgrades knowing even if for whatever reason my ZFS module doesn't compile/work under the new kernel, I won't be left without a root FS. I have been running BTRFS for 5 years with very little issues, and enjoy not having to compile a new dkms module with every kernel.
ZFS with DKMS is a disaster, at least in my experience. Honestly, I can't recommend ZoL unless you're running a distro with relatively stable kernel releases that don't change substantially or that happens to be supported by ZoL with binary packages. ZoL on Arch was... trying at times. It worked great, but my paranoia meant that I ended up adding the kernel to IgnorePkg to force manual kernel updates (mostly for my own memory). But then, it also meant having to build all of the ZFS packages (including SPL) tied to that specific version. This usually meant waiting until the AUR packages were updated as I figured that indicated someone else must have tested ZFS on that specific kernel version.
I remembered thinking DKMS might solve the problem, but I ended up having to use recovery media just to get an environment to reinstall an older kernel and let DKMS do its thing after a botched update started provoking panics. I suspect a version mismatch based on the errors but never investigated it beyond fixing the problem and moving to the prebuilt modules. Things may have changed, but the Arch ZFS+DKMS packages were a bit flaky and required some manual modification just to boot (should've taken this as a warning!).
Granted, it was my fault entirely for being a bit too enthusiastic with ZFS on Arch. To be honest, if I were to use it again, it would be on FreeBSD. Not Linux. I recognize it's fine for other people, but in my use case it wasn't.
It is more like Arch Linux is a disaster. Upgrading the kernel package replaces the current one! Come on, any distribution worth it's salt just installs new versions alongside and you can select any of them in the boot screen. This is a ridiculous packaging policy regardless of ZFS or any other DKMS modules.
And I acknowledged that using it on a different distro would be more advisable, although I still stand by my claim that FreeBSD is far more appropriate for ZFS.
I will, however, agree that having no fallback to the prior kernel version is a problem. In practice, it's never caused me much trouble except when I do something stupid like using ZFS from the AUR. initrd generation has historically seemed to be more problematic under Arch, but I'd argue that's mostly fixed with install hooks.
In all honesty, it was probably more the fault of the zfs-dkms packages than it was either the kernel packaging policy or ZoL+DKMS itself (for reasons I elaborated on in my original post).
But, that's also what you get when you use packages from the AUR or using a distro like Arch for something that really only benefits from a wider installation base (like Ubuntu does, for instance).
I know you acknowledged that using it on a different distro would be more advisable, I just wanted to vent about more broad issue of their packaging policy. Sorry if it wasn't clear.
I do agree. There are circumstances where Arch's packaging is brain dead (they only recently, within the last 2 years or so, started validating packages against signatures!). I use it for a number of applications, and as my desktop OS among others. However, I'll freely admit at least part of my choice is perhaps the fault of masochistic tendencies. After all, I migrated to Arch from Gentoo, and I used Gentoo for years! :)
In all honesty, I've been bit more by the initrd and mkinitcpio's failings than the lack of a fallback kernel. That's mostly fixed with packaging hooks that essentially guarantee it will run, but it's still a problem with the ZFS packages and may require running it manually (which is annoying). However, that wasn't always the case, and sometimes the generated initrd would be missing something important. You can imagine what happened next.
Indeed. Arch is good for a few things, but sometimes stability isn't one of them when it comes to unsupported packages. ;)
I'm not sure I'd be brave enough to run ZoL again, but given Ubuntu's FAR wider install base and availability of binary packages, it's the better option if you have to choose.
My personal preference would be to stick with ZFS on FreeBSD. Performance is probably better.
I'm actually surprised by this, but I'd wager that you also didn't use the DKMS AUR packages. I also suspect you wait for the zfs-linux (etc) packages to match the kernel version before updating. Or you manually bump the kernel version and build it, hoping for the best (edgy!).
I considered it, but I have some problems with the Arch LTS release cycle. If I were to choose an LTS kernel, why not just dump Arch and go with an Ubuntu LTS, which has better long term support?
The other problem is that at the time, the ZFS packages for LTS were pinned at a version that had a known issue with arc_reclaim encountering a deadlock essentially causing the file system to become unresponsive after a substantial transfer (think rsync).
Now, obviously, it wouldn't be that difficult to modify the PKGBUILD to pull a newer version of ZFS, but there's a point in time where the maintenance required to update starts to outweigh whatever benefit you can glean from the LTS kernel.
That's not the case now since the LTS packages appear to be at v0.6.5.9, which has the fixes, but I don't remember this being true about a year ago.
BTRFS lets you make CoW copies of files. You can even retroactively merge the blocks that store identical files. BTRFS also makes it not a giant pain to remove a file from snapshots.
ZFS does seem to work better overall, but I wouldn't call either filesystem great at this point in time.
It's a little black magic -- didn't have time to completely research (this is a home NAS system.)
I believe there was a bug in the clear space cache. This could cause the system to think it didn't have free blocks... you'd have to mount another device to create more space in order to rebalance and fix.
Eventually I saw a bug fix report about a corruption in the cache... I never investigated to see if my current kernel has the fix.
I'm using BTRFS on many VMs running Ubuntu servers, and I find that expanding the virtual hard-disk, on Proxmox, without stooping the VM it's far trivial.
I think you missed a key word there so the meaning is lost. Is it "far from trivial" or "far more trivial"? You also surely meant stopping, not stooping.
I purposefully ran btrfs on a malfunctioning drive for over a year (kernel 3.12, only metadata dup), much more reliable than ext4 which would lock up the entire filesystem on read/write failure and often go read-only, with btrfs the only visible signs of malfunction were dmesg and the scrub log.
Also been using it as / since 2012 with no issues.
It is ok for a single disk FS. It is no ZFS though which is its largest problem, people keep marketing it as "linux awnser to ZFS"
No, it is not.
Maybe some day, but today it is no ZFS. I love zfs..
Further the ZFS utilities are far easier to use and understand. zfs and zpool commands well documented, and intuitive. btrfs utilities are are not, IMO
I am fine with using btrfs as a replacment for ext on my OS drive, but for my large data arrays of multiple disks it s ZoL all the way
Likewise, I've run it on hundreds of machines over the past three years without issue. I do continue to use EXT4 for database hosts as it far outperforms BTRFS with PostgreSQL from what I've seen.
How is the speed now? I never had data issues but I definitely had speed issues. Btrfs was really slow compared to ext4/xfs back when I tried it. And I mean orders of magnitude slow. I had an application that did a lot disk access and switching off of btrfs brought the runtime down from a week to just hours. I want to like btrfs, but after that I just can't trust it for high disk load situations.
I had a bad experience with in around 2014-15 on Ubuntu. 3 different laptops in my house suddenly stopped working (didn't boot at all) within the time-span of a year, 1 laptop faced the problem multiple times. In all cases, I had to format the root partition and reinstall Linux. All 3 had btrfs for / in common.
I moved back to EXT4 and it never happened again since then.
From 2012-2014 it was mostly breakage every other month. From 2014-2016, it was semi-annual issues.
For the last ~18 months I have had ~30 machines running btrfs with no issues, some servers, some personal computers. The release notes are boring, the bugs are boring, and to me its definitely in a state I would strongly consider trusting it with any workload.
I worry that btrfs is just going to remain doomed. It wasn't stable half a decade ago, so it - for some reason - cannot be more stable now. But it has seen so much work put into it to make it as mature as it is now, and in my experience it is pretty damn mature now. All I want to see is another year and a half of perfect stability before I would start arguing to drop zfs entirely.