Here's what I wish I'd known before using Vagrant:
1. How to properly do cross-platform, high-performance two-way synced folders between host and guest. Most providers only support one-way syncing. Virtualbox has shared folders, but their performance is pretty lousy and they have issues with relative symlinks. In fact, I still don't fully know what the correct setup is for a dev environment where the files are edited on the host and the guest immediately picks up on them...
2. This idea that with vagrant you'll never again say "It works on my machine" is a lie. So many inconsistencies between Vagrant on Windows, Linux and macOS. Internet connection sharing, line endings issues, symlinking issues, ...
1. On macOS, host FS volumes are two magnitudes slower. There are a series of hacks that try to mitigate this problem and progress has been made, but it’s still too slow for a dev environment. Linux doesn’t have this problem. I don’t know about Windows.
2. Also on macOS, the host networking bridge doesn’t work the same way it works on Linux. There’s a different hostname that’s used to access the host from within the container that’s inconsistent with Docker on Linux.
Docker is working through these issues, but they’ve been ongoing for years.
I’ve written about Docker dev macOS environments at bradgessler.com, but have given up mostly because FS performance is so poor. Now I spin up what I call “Dockerish” dev environments where my service dependencies run in Docker and the app runs on my host.
Regarding 1), Docker implemented a cached volume mount option to mitigate the performance issues on MacOS [1], but last time I tried it, file system events stopped working. I'm not sure if this is an inherent limitation or if it's something specific to my environment. I'd appreciate if someone more knowledgeable could clarify.
As for Windows, it doesn't have the same fs performance issues on volume mounts, but AFAIK file system events have never worked there [2].
My current reluctant compromise to make volume mounts work seamlessly across platforms is to use polling instead of relying on file system events. That way I can turn on caching on Docker for Mac, and everything just works. The downside is of course polling is inefficient, but I've found when you take care to exclude all non-source directories from the polling, it's usually not as bad as you'd think.
Given Docker for Mac controls an internal VM instance to actually run the containers, is there much benefit over just using Docker in a Linux VM?
It's pretty easy these days to use tools like Vagrant to spin up the VM then run a Docker Compose file or whatever to get your development environment ready, and would result in identical cross platform environments.
I think it's easier for most people to get started using Docker for Mac, but yeah – I run Debian headless in a VM, SSH in and use that for Docker. Way less magic, and I have full control of the underlying OS to test out new kernels etc.
I also like that my underlying code is also within an encrypted VM disk, not laying around on my host mac os.
Do you use a GUI editor in the VM? I have a similar setup (Mac hosting a headless VM running Docker) and can't stand performance when using X-Forwarding and XQuartz
Nope, neovim and tmux is all I use regularly. I tried to proxy X11 to quartz and it was way too slow.
When I use Reason I'll sometimes mount over SSHFS to edit using neovim in visual studio, though right now I'm building native iOS apps with it so it's actually not in my VM so that xcode can build faster
I love VSCode, but moved back to AWS+VIM for dev because the FS is just too slow in Windows and I am forced to use a Windows machine for work. VNC is too slow with 4k monitors as well.
I hesitate to call it the correct setup because I think only a small number of people use it, but the only solution I've found for Virtualbox that 1. does 2-way sync between host & guest 2. Propagates fs events (so you can run an auto-reloading service like webpack-dev-server on the guest that picks up changes on your host) and 3. is not significantly slower at disk reads & writes, is using Unison which is basically a 2-way rsync.
There's this Vagrant extension [0] that works pretty well on mac at least (which I actually forked and pushed to rubygems a few years ago when I discovered it because it had been abandoned and no longer worked)
Unfortunately the Vagrant team seems to have no interest in having all 3 of the properties I mentioned just work out of the box, I'm not sure why. Maybe it's just a lack of resources at Hashicorp, I'm not sure how much dev time they put into Vagrant these days since it's otherwise very mature and they have such a large number of open source projects.
This looks very interesting - unison is a great tool. I rember fighting to get plan9 networking working with qemu/kvm and Linux, and never quite figuring out how to connect the dots. On paper it appears 9p should be a perfect fit for the use-case. (However, I just found [1] - maybe I'll give it another shot).
But it seems kind of obvious that the functionality needed to get this stuff to work, is an entire network filesystem stack. It seems the obvious choices would be to build into VirtualBox either nfs, or webdav - or leverage host/external support for nfs, cifs, or something like OpenAFS. With sshfs as another pragmatic option. Maybe in the future ipfs will be a viable option too.
But I think cifs is still the "mostly works, with/without complex auth, cross platform" - with webdav a distant second.
I used your vagrant-unison2 plugin for developer environments at Airbnb for a time, although we've now moved on to a more complex, home-grown Unison wrapper with Watchman for filesystem watches.
>1. How to properly do cross-platform, high-performance two-way synced folders between host and guest. Most providers only support one-way syncing. Virtualbox has shared folders, but their performance is pretty lousy and they have issues with relative symlinks. In fact, I still don't fully know what the correct setup is for a dev environment where the files are edited on the host and the guest immediately picks up on them...
I set up a TCP server, one light process watches file changes and the other side reacts to it. (Rails). It was one of my silent victories that the entire company used, but I had to do because I was the guinnea pig on the vagrant migration.
I've had much better results mounting host directories via sshfs than I ever have with shared folders. Cygwin sshd works on Windows; there are probably faster ones, but I've never felt the need to bother.
I've been meaning to look into vagrant-sshfs but the fact it requires extra setup (quite a bit of it on Windows) has put me off. Our vagrant setup right now is "Install Vagrant, Virtualbox, run this one command and you're done".
We get around the shared folder performance problem using NFS. Adds one extra step to setup (mount a network share), but we've never had performance issues with it.
Not so familiar with vagrant on mac (though my time is coming), but having used loopback KVM's on rhel I can say that fiddling with mount options can drastically improve stability / performance (though still much slower)
e.g. tcp mounts, getting read/write blocks matched up btw/client server and sized to be digestable but big enough to move data, etc.
also, nfs is mainly only suited for 'NAS-like' operations - things like rdbms's do waay better on iscsi or eating the vdisk performance.
last I messed with macos nfsd (which has been a while), it a way happier with smaller blocksizes (e.g. 8-64k range) - modern linuces will attempt 1MB which is too much for the older 4.4BSD based code
another thing to look at is timeouts / backoffs - it's easy to kill performance by setting these things too agressively so that the system double-chokes when it gets bogged down..
I used NFS when using vagrant shared folders a lot a couple of years ago. Didn't really pay much attention to the performance, I must admit, which was fine for my purposes; for me it only really needed to crash less than the default shared folder stuff...
I did have to supply an extra "mount_options:['actimeo']=1" option to the synced_folder command to make file watcher-type systems work tolerably. With the defaults I'd often be waiting several seconds before any changes made by the host would be picked up by the guest.
I've been struggling with 1) for years. My current attempt is to exploit ZFS snapshot shipping, since modern ZFS ports are available for both mac and linux (and apparently one for windows is on the way).
Unlike rsync, ZFS always knows what's changed since the last snapshot, so there's no directory scanning.
Unlike inotify, there is (seemingly) no(?) risk of dropped events, which I've experienced with tools like watchman. My ZFS solution still needs prompting event to start the snapshot, but it no longer needs to be told _what_ changed.
Critically the sync flows in one direction. This is a deal breaker for applications that need to communicate file changes back out to the host, but that hasn't been an issue in my use cases.
Unlike NFS, it also means we must (and can) relax the strong consistency model, and so using the filesystem is not so latency bound. This becomes important if the virtual machine is not running locally..
2 has been a big issue with a recent client because Virtualbox is a fucking pig in terms of features and performance compared to the commercial alternatives.
My advice is to cough up an hour or two salary and pay for parallels or VMware (if on Mac)
I've been using Vagrant + Virtualbox and have been happy with it. Is Vagrant + VMWare really that much of a step up in performance? Could you explain which features have been helpful? Thanks.
Not the parent, but anecdotally, Vagrant + Parallels runs MUCH faster on my 15" MacBook Pro than Vagrant + VirtualBox. VM startup time is shorter, and CPU usage seems to be lighter. For instance, I worked on a project where I had a long-running process that would do a lot of file I/O and periodically collect the results and run some calculations. On VirtualBox, this basically pegged a CPU core while it was running, but when I switched to Parallels, CPU usage hovered around 5%. (I'm guessing this particular example is more to do with VirtualBox's dog-slow shared folders, but still relevant, I think.)
Well one of the killers I found very recently was if you open the console of the VM before it has booted it sets the caps lock on which instantly shafts entry of the grub commmand line so initial configuration of that is pissing in the dark because you can’t observe it without affecting it.
Then there’s the periodic refusal to talk on the network even if the virtual switches are configured properly.
Then the inexplicable “slow boot from hell” which happens randomly where it’ll just hang starting up the kernel at 0% CPU for up to 8 minutes.
At least for me, using Hyper-V is problematic because I can't also run VMware Workstation at the same time. I have to boot into different configurations to run them side-by-side. My macOS experience is that running both on your machine is straightforward.
I did that a few years ago, and I found that VMware support lagged VBox. There were some plugins that looked like they might help, but I could never get them to work.
at least in the US, anything work-related that you pay for out of pocket is tax-deductible if your employer doesn't reimburse you or you're self-employed.
Doesn't make a lick of difference if you lack enough itemized deductions, otherwise you take the standard deduction like everybody else.
Honestly though, VMWare is pretty cheap. VMWare Workstation Player and VMWare Fusion aren't cheap upfront, but the upgrade pricing to stay current is decent - I spend more on my JetBrains licenses every year. It's worth it to have a fast and mostly headache-free virtualization experience.
I find that Vagrant works relatively similarly between Mac and Linux to the point where "it just works", but Windows is a whole different ballgame with shared directories and symlinks. On the current software that I'm working on for example, the first step of the onboarding process for new developers using Windows, is to install Linux, which seems pretty insane for something that lives inside of a VM!
What I do on some projects (not using vagrant) is install samba on the dev VM so that it can be mounted from a Windows workstation. Obviously much simpler and easier to do if you're developing on Linux (nfs / sshfs).
Are you talking about symlinks on synced folders? I absolutely love synced folders, but the whole idea is fundamentally incompatible with symlinks. Think about the VM actually interpreting the link and resolving the path against its own filesystem. Just doesn't make sense. If you want that, share the directory on the host and mount it in your VM so it's your host resolving the link.
My experience with Vagrant is that it's amazing, but I've only used it for a web application with Github and Jenkins for CI. Dev team doesn't have to spend a second thinking outside the git repo.
1. How to properly do cross-platform, high-performance two-way synced folders between host and guest. Most providers only support one-way syncing. Virtualbox has shared folders, but their performance is pretty lousy and they have issues with relative symlinks. In fact, I still don't fully know what the correct setup is for a dev environment where the files are edited on the host and the guest immediately picks up on them...
2. This idea that with vagrant you'll never again say "It works on my machine" is a lie. So many inconsistencies between Vagrant on Windows, Linux and macOS. Internet connection sharing, line endings issues, symlinking issues, ...
If anyone wants to see how we're using Vagrant:
https://github.com/hearthsim/hearthsim-vagrant
I don't want to make it sound like Vagrant isn't solving a real problem though, it is. It's just not the unicorn it claims to be.