My recommendation to the authors would be to use NFSv4 instead of NFSv3. No need to mess around with that separate MOUNT protocol. Its semantics are also a lot closer to POSIX.
Very nice. The mount protocol is pretty minimal, though different OS clients seem to use it differently. The NFSv3 chattiness is definitely a thing, but since its all localhost its not "too bad". v4 will be nice and is something I will love to try to implement. Its quite a bit more complex though.
Another possibility I am interested in is SMB3. (SMB4 is rather nasty). Similarly, supported by most operating systems, and does not look too horrifying to implement (The protocol documentation is really really hard to read though). And has the advantage that it does not require Windows Pro edition.
This is awesome. NFS is very much undervalued as a cross-platform interface for talking to a (not necessarily remote) filesystem.
~20 years ago I was using SFS (Self-certifying file system), which was a SSH-like Internet-usable TOFU filesystem, using NFS as the "backend" for talking to the host OS. The site has since disappeared, but is archived at https://web.archive.org/web/20080330201843/http://www.fs.net....
> XetHub has the world’s first natively cross-platform, user-mode filesystem implementation, allowing you to mount arbitrarily large datasets on your machine.
Not really world's first. CERN has developed EOS (https://eos-web.web.cern.ch/) for many years, and even though it's not available natively on Windows, it is available on Linux and macOS. EOS uses FUSE, though, not NFS.
> This enables you to, in just a few seconds, locally mount ~660 GB of Llama 2 models or write DuckDB queries to analyze large parquet files and scan just the data you need.
If you mount all instances of EOS at CERN on your machine with the FUSE client, that in principle mounts hundreds of PB of data from LHC experiments, although much of this data requires special permissions to be accessed. However, there's also a lot of open data. See https://opendata.cern.ch/.
The client is available on Windows via a samba gateway. There's also a sync client for CERNBox for Windows, and even Android and iOS (https://cernbox.web.cern.ch/cernbox/downloads/). The server is not available on Windows, though. However, I'd say that among physicists Windows is definitely the least popular OS.
Gluster did this about a decade ago, though not in Rust. Actually twice - home-grown NFSv3 and Ganesha-based NFSv4. At Facebook we used the NFSv3 version to run dozens of clusters serving tens or perhaps hundreds of thousands of machines. My own involvement with that piece was minimal (I was a maintainer of Gluster overall for a while) but it worked pretty well for those users who needed a real mountable filesystem with real files instead of vaguely file-like objects that let you down every time you try to step beyond whole-file read and write.
Nice! I did look at Ganesha when I was exploring other NFS server implementation I could perhaps borrow and simplify. It looked great, but pretty much all implementations are too "real-filesystem" for me to convert. I needed something which I can make a very simple virtual filesystem interface over, and so wrote my own.
Thanks! I'm no longer a maintainer, or even working in tech, but still glad to share my experiences if folks might find them useful. I'm Obdurodon most places, @hachyderm.io on Mastodon, if you want to hit me up any time.
Anyway, interesting idea. As another comment pointed out, I wish there was an easy way to mount NFS and 9p shares as a user.
I've been toying with 9p+Wireguard lately, as an alternative to SMB/Webdav/sftp/etc for sharing files with friends over the Internet. Wireguard+NFS works well enough, but it's relatively hard to integrate in userspace. Toying with rust-9p[1], I obtained acceptable performance. I still have to experiment with putting Wireguard in userspace as well (probably with Tailscale's implementation), but I come back to the difficulty of mounting 9p and NFS shares as a user. I also wish it was simpler to export directories over both protocols, on an unprivileged port, with access rights limited by those of the user.
Regarding the article content, I find 9p to be even simpler and probably more ubiquitous than NFS, though I don't know if MacOS and Windows natively support it.
If you haven't seen it, Cloudflare also has an implementation of Wireguard in userspace, called Boringtun [0], written in Rust and successfully deployed to millions of iOS and Android devices running the 1.1.1.1 app.
For reference, another Rust project that depends on Boringtun is Onetun [1], which uses it to encrypt packets sent over a virtual smoltcp interface. I imagine you could follow a similar approach to integrating rust-9p with Boringtun, and you wouldn't need to leave the Rust ecosystem (whereas you might face more obstacles integrating it with Tailscale's wgengine, which is written in Go).
> Anyway, interesting idea. As another comment pointed out, I wish there was an easy way to mount NFS and 9p shares as a user.
Add e.g. foo.bar.com:/path /net/foo.bar.com nfs to /etc/fstab with options=user ?
> Regarding the article content, I find 9p to be even simpler and probably more ubiquitous than NFS, though I don't know if MacOS and Windows natively support it.
Windows certainly does not. MacOS I don't know, but if you mean "natively" as "stock install without third-party software", then probably not.
Actually, scratch that, comments below suggest that WSL2 uses 9P for file sharing with the host, so it's possible that a version of Windows with the WSL2 bits installed will support 9P.
Have you run any POSIX compliance checks against it? In particular, how are you handling assignment of inode numbers (fileid in NFS speak) to "virtual" files and making sure those stay reasonably consistent? (I vaguely remember this being a bit of a tripwire)
Also did you look at NFSv4 / what made you decide to go with NFSv3? My (superficial) impression was that NFSv4 did a lot of simplifying and throwing out legacy stuff (e.g. rpcbind), but I don't know too many details on this…
I tested by uh... building a "mirrorfs" and compiling this project in it :-) (cargo does stress it pretty hard). I have not looked for POSIX compliance testers. Is there one you can point me to?
NFSv4 did throw out a lot of the legacy stuff, but also added a bunch of other state information like locks and delegation which is quite a bit more annoying to implement. Technically cos this is meant for "localhost-mounting-only", delegation should be easy to build (since we are expecting only exactly 1 mount). But v3 had far fewer APIs to implement, and so is a good starting point.
Note that there’s absolutely no requirement to support more advanced features such as locking and delegations. Even if a client offers to accept a delegation, a server may just deny/ignore it as part of OPEN.
There's https://github.com/google/file-system-stress-testing but in this case I believe it would mostly stress-test the NFS client rather than your code… and I have never tried to use it, no idea if it even does anything useful.
I know from lore that there are compliance-test suites for the POSIX file system API, but I believe those are commercial products :(
If I may ask, is this mirrorfs available somewhere? The demofs from nfsserve seems to create and server a pure in-memory fs tree if I read the sources correctly.
NFSv4 is a completely different protocol and basically doesn't share anything with v2/v3. While technically it's still implemented using Sun RPC, it only has two procedures: NULL and COMPOUND. Version 3 is actually an RPC protocol, with a different procedure for each filesystem operation (READ, WRITE, CREATE, ...). One of the genius moves that Sun made with NFS was to distribute the protocol files (rpcgen) which autogenerate stubs (in C) making it easy to implement servers and clients. Nothing like that exists for v4.
Which is also what you want, right? The issue is that with compound calls there is some state that’s carried over between operations (current/saved file handle), so you’d need to implement that yourself anyway.
Well yeah that's the difference, it doesn't generate the state machine for you so you "just" need to implement it yourself. For v3, rpcgen spits out a working function that you link into your program and you're done (on the client side anyway). Much easier.
> You simply need to implement the vfs::NFSFileSystem trait.
Wait, what......
.. So not only is it an easy-to-run Rust NFS server and client (I gave up on NFS some years ago because I couldn't figure out how to run Samba) but it's also meant to use as a _substitute_ for FUSE in stacks like SSHFS or an app wanting to serve debug data as files?
Sweet! I've been monkeying around with writing an HFS interpreter (in rust ofc). Unfortunately 'fuser' is less than ideal on a Mac. This could be an interesting alternative.
Edit: Oh. It's async. The other bit that I liked from fuser: the default implementations in the trait cut down on the boiler plate. Setting the RO mount option got you defaults that DTRT.
I don't think this includes a client, in fact the post argues that not having to implement the client because it already exists in the kernel was a major benefit of the approach…
I have not looked at WebDAV closely. I was looking for something that is already supported in all the major operating systems without additional libraries. I don't believe WebDAV is commonly supported out of the box?
WebDAV is supported by Windows, MacOS and Linux. On all platforms you can mount it without admin/sudo (`net use` on Windows, `mount_webdav` on macOS and `gio mount` on Linux). Here is official Go package[1] using which you can implement your own WebDAV server (you only need to implement your FS interface).
I believe you can install an NFS client on windows home if you just go to programs and features > add a feature > NFS client, but I don't have a way to check.
The fact you just need to impl one trait is amazing. I will definitely be testing this out
This is clever, considering the constraints. We use a fork of catfs write now to have a read through cache over NFS, but we could, in theory use a similar flow as here to do it and this will be a good example. Nice. Part of the problem, though, is that we currently use extended attributes, and I think you need NFSv4 plus a modern Linux to get those across NFS.
Doing some quick googling, I see https://lwn.net/Articles/353831/ which sounds like an xattr protocol extension to nfsv3. I do not know the extent to which it is supported though...
NFSv4 is quite a bit more complicated and is also stateful which makes it a bit more challenging to implement. NFSv3 is a good starting point, and is still supported on all the major platforms. Certainly, extending with a v4 implementation would be great future work.
Also keep in mind that NFSv4 is mostly adds necessary complexity. Not accidental complexity.
For example, NFSv4 supports accessing files that have been unlinked, but still have one or more open file descriptors. NFSv3 does not. It needs to resort to a technique named ‘silly renames’.
You will have to throw away almost all of your code for NFSv4, it's a completely different protocol. One of the advantages of 4.x is that you don't need the portmapper or mount protocols, you just need to know the hostname/port of the server.
You can run NFSv3 without the portmapper, and for most servers (like unfs3) the mount protocol is served by the same process as the filesystem, so it isn't two daemons -- just one.
Funnily enough, "the end of time" for NFSv3 is in about 15-83 years, depending on whether the implementation uses signed or unsigned 32-bit integers for timestamps: https://lwn.net/Articles/717076/
It's widely implemented but not adopted. Most vendors implemented v4 to tick a box for some RFQ. I don't think any of the commercial fileserver vendors publish numbers but I used to work in that space and internally customer adoption of v4 was always a single digit percent. When I was on the customer side in HPC none of the supercomputer clusters were using v4 (I'm sure there are exceptions that I wasn't aware of).
That’s completely incorrect. NFSv4 was a quantum leap in performance; greatly reducing network traffic via COMPOUND and dramatically improving write performance via delegation.
All my file access is over NFSv4 (in a mix of 10 GbE and 100 GbE).
Having GP’s crate expand to support NFSv4 would be welcome but a nontrivial bit of work.
Then again, if you're using file locking on a distributed filesystem at non-trivial scale you're in for a big world of hurt anyway. Source: I was a developer and maintainer of exactly such a beast for over a decade.
NFS seems to be a good fit for their use cases, however I don't believe that NFS supports an `inotify` like interface which makes it less useful for usecases like virtual filesystems for developer environments.
Maybe newer NFS revisions allow for the NFS server to notify clients of changes like this, which would be really open things up here.
Very good point. Is there anything out there that's like NFS but also supports inotify? By "like NFS" I mean a remote filesystem protocol (not a block-level protocol) that works across platforms and can support nearly all filesystem features.
I guess these are used as a soft mount, considering that if you terminate the application or it crashes a hard mount would just hang indefinitely. That's probably fine for a read-only snapshot, most problems with soft mounts are around writes.
I don't know that its considered a "design mistake". It is not without flaws, but it does make it extremely simple to implement. Its a very acceptable point as a tradeoff between simplicity and performance. Will certainly love to implement NFS4, or even SMB3 at some point though.
FUSE is used to present a file system interface of something that isn’t a file system. E.g. present an LDAP directory as a file system.
Writing FUSE drivers is pretty frustrating, though, and not well supported by all OSs.
My takeaway is that this project discards FUSE in favour of making an NFS server that can be easily extended to do the same thing: present non-file system systems as mountable nfs remote file systems.
It’s a neat idea, and does avoid any extra kernel drivers, etc. A safer abstraction at the cost of adding a network stack and NFS’s sometimes wonky effects (especially with hard mounts).
It’s part of Buildbarn, a distributed build cluster for Bazel, but it can also easily be used outside that context.
Details: https://github.com/buildbarn/bb-adrs/blob/master/0009-nfsv4....
My recommendation to the authors would be to use NFSv4 instead of NFSv3. No need to mess around with that separate MOUNT protocol. Its semantics are also a lot closer to POSIX.