Just the other day I voiced [1] a concern on a ZFS thread about not enough modern filesystems [2] shipping as defaults (or being recommended as such), despite some of these filesystems having been around for many years.
In some ways I think we actually need simpler file systems more than fancier ones. Here's an exercise: create a USB flash drive that can be both read and written on any system it is plugged into, by any user, without using FAT.
Can't be done. It's 2018 and FAT is still apparently the best we can do for universally usable filesystem.
And don't go trying to pin this on Windows not supporting ext or something stupid like that, because an ext filesystem cannot by default be both read and written universally even on systems that have full ext support, because they bake in permissions and the default umask botches this simple use case.
I realize this is sort of tangential to what you're saying, but I think a big problem we have in tech communities is that we focus too much on fancy interesting things and not enough on stuff that gets things done simply.
UDF has a lot of issues in actual implementation. For instance MacOS will only support UDF on an unpartitioned disk, while Windows will only support it on a partitioned one (IIRC was true for 7, may not be anymore). Additionally, since it has posix permissions, I expect it will have the same multi-user issue on posix systems that ext does, though I haven't tested this.
I'm pretty sure the default is still UFS, but FreeBSD allows you to use ZFS as the root filesystem. I've been using this for a while and it's been incredibly solid. It's a first class feature that's supported by the bootloader and the GENERIC kernel. My experience with it has been uneventful, which is exactly how a filesystem should be.
"TrueOS has been using OpenZFS as its exclusive file system for several years, ensuring advanced OpenZFS functionality is heavily tested and 100% production-ready."
With DragonFly's architecture being designed with multiprocessing and clustering in mind, how does it fare (relative to Linux and other BSDs) in modern environments with many-core CPUs, horizontally-scalable architectures, reliance on VM-based workloads, and containerization?
I would think it would really shine, but haven't seen comparisons - or much market uptake.
Wouldn't significant advantages result in lower costs (e.g., fewer CPUs needed because of higher performance per core due to lower OS overhead) and therefore economics would drive greater use? If it hasn't, why not?
I'm not an expert on DragonFly, but from what I know it's design differences aren't really that radical. It differs substantially from FreeBSD, but it's not like a fundamental re-imagining optimized for SMP. Performance is a bit better in some places, and about the same in others.
I do think DragonFly is neat and quite interesting, but I suspect it didn't gain as much momentum because the community behind BSD is already small and reluctant to fracture without stronger reasons. It's worth noting that this is not like spinning off a new Linux distro where you're ultimately just repackaging different variations of the same basic OS - DragonFly is effectively an entirely different OS since the kernel is significantly different.
Thanks, I took a look through that and most (all?) of the benchmarks seemed to be fairly traditional tasks, some not even highly threaded.
I guess I'm wondering if complex, multi-box, cloud-scale workloads have any advantage on Dragonfly - or if the optimizations for clustering are in such a narrow band of use cases that modern architectures have passed it by just through brute force.
There's an irony here, in that Matt Dillon was an ex-Amiga hacker (IIRC, some of Dragonfly's architectural choices were inspired by the Amiga). Anyway, the Amiga's linear memory, co-processor-driven architecture was ridiculously more elegant than the PC's segmented memory, CPU-bound approach. Didn't matter though, a huge investment in working around the PC's limitations led to its market dominance.
So, if massive technical accommodations for less efficient OS architectures end up marginalizing Dragonfly, it would be a bit of history repeating itself.
And I'm sure the irony wouldn't be lost on Mr. Dillon.
I did a little testing about ~4 years ago and found most of the speed increases were in the netstack, but I didn't do any computational testing either.
As for your last question, I think largely due to the fact that it's just not mainstream yet.
Anyone know what the state of GPU support actually is? The Supported Hardware page [1] only says "Intel GPUs are supported up to the Skylake generation (2016 machines)", but this seems to be outdated. At the very least, Matt Dillon himself reported good OpenGL performance with a Kaby Lake machine [2].
A few years ago I wanted to try this as a local Minecraft server but it was 64 bit only and the spare machine I had was 32 bit. Glad this post came up because now I've got a 64 bit machine sitting in my basement I had forgotten about until now.
From the discussion I've heard, it would be challenging even to port HAMMER2 to other BSDs, so I wouldn't expect to see it on Linux any time very soon.
Not at all, originally the whole purpose of DragonFly was a test ground for its scheduler. It moved on to being much more though, refactoring the entire source tree and rewriting parts of the system. The whole purpose of DragonFly is DragonFly.
Dragonfly also has lightweight kernel threads and some other different architectural decisions. Not just HAMMER. But you are right, it will never reach Linux.
You're not wrong. Interview from 2004 [0]: "DragonFly split off from FreeBSD-5 over major architectural differences, not anything else. We really do feel that FreeBSD-5 is taking the wrong approach to SMP and building something that is so complex that it will ultimately not be maintainable. We think we have a better way."
But their original architectural decisions were done with this goal in mind [1]: "The ultimate goal of the DragonFly project at its inception was to provide native clustering support in the kernel. This type of functionality requires a sophisticated cache management framework for filesystem namespaces, file spaces and VM spaces." It's not difficult to see how HAMMER came to be three to four years later, given their stated design goals.
While the likelihood of a collision (whether "accidental" or malicious) is probably still relatively low even with MD5, what reasonable, compelling reason is there to still be using MD5 -- especially when much stronger hash algorithms such as SHA256 and SHA512 are available?
Generate both SHA256 and SHA512 hashes or maybe SHA512 and some other "unrelated" algorithm (if you want to really play it safe), dump them into a checksums.txt (or whatever) file, then PGP/GPG sign that file (with a widely distributed and certified/signed public key, of course) and you can effectively eliminate any chances of a collision whatsoever.
It seems that the benefits of switching would greatly outweigh the costs associated with doing so (unless this would require some major code changes to your processes/pipelines/etc.).
Rather than "vaguely possible" MD5 collisions are now so easy you can make new ones at home on an ordinary PC in a few seconds, and you can do chosen prefix collision (thus enabling you to pick a "format" and collide within that) in a few hours. Cryptography never gets _less_ broken, only more broken with time.
Now, in this particular scenario (a publisher tells us the MD5 of an image, and we can check it to see we got the image) collision isn't so important. We have to trust you anyway, so we may as well trust you to not collide the hash too. But MD5 is no longer even proof against second pre-image attacks, albeit the best known are as yet impractical. This is really bad news.
MD5 has been known to be irrevocably broken since 2004, and had been expected to fall since the mid to late 1990s). DragonFly BSD was only started in 2003. Why use MD5? Imagine if in 2003 you'd decided to build a 16-bit OS, or one that doesn't do TCP/IP, because after all, in the mid-to-late 1990s that might have seemed fine too...
The malicious case would be a bad actor distributing phony files that appear to check out because the flaws have been found that would allow someone to do so.
In practice today this requires that the bad actor collaborates with the real distributor.
MD5 fails to _collision_. A collision is when you can find two different things with the same hash. But being able to collide the hash is NOT the same as being able to find a second pre-image, which is what you'd need in order to get "phony files that appear to check out" if the person who originally issued the MD5 checksums didn't collaborate with you.
There have been practical demonstrations of using padding in several formats to generate valid files that collision with an original one, but with entirely different contents.
This means that it is possible that someone could download the real image, introduce some rootkit, and then tinker that (for instance, by adding a hidden file with carefuly crafted content) until the resulting md5 is the same as that of the original image. Then hack the server and upload the modified image in place of the original one, and everyone who installs Dragonfly is now their minion.
If you use a stronger hash (which is not harder for anyone than using md5), then this attack vector becomes impossible. So... even if it is a remote possibility, just use the stronger hash because it is just a dominant strategy (it has upsides, yet 0 downsides).
There have been no practical demonstrations of colliding the MD5 of an arbitrary file (ie. pre-image attack), only situations where two files are created specifically with the intended purpose of creating a collision. This is precisely what the post you replied to said but you seem to have not understood that there's a distinction.
Yes, it is possible that the DragonFly developers could collude to create two ISOs with the same MD5, one good and one malicious. No, it is not possible that random, evil ne'erdowells could replace the ISO with one with the same MD5, unless the DragonFly developers have conspired with them to make that possible.
If you don't trust the DragonFly developers not to collide the MD5s, you probably shouldn't trust them with the code running in your kernel anyway.
The issue is that if these files are distributed elsewhere by 3. parties, it is trivial for those 3. parties change and compromise the files, but still make the files produce the same MD5 sum.
If you think a preimage attack against MD5 is "trivial" you should demonstrate it. People would be very interested in this because no one has managed to do it yet.
Creating two files with the same MD5 is a very different beast from creating a file with the same MD5 as an arbitrary pre-existing file. These third parties would need to have colluded with the DragonFly developers to make what you're proposing possible.
This is a promising move to the contrary.
[1] https://news.ycombinator.com/item?id=16798244
[2] in some loose, but recognizable definition of modern, featuring checksumming, CoW, deduplication, snapshots...