Sure, there may races due to limitations of underlying OS interfaces, but no such races really matter when you edit with mammal speed, so it's perfectly sufficient, and so it is perfectly possible to make good file system watcher, without any user-space power-hungry hacks.
inotifywait is a user-space program that borrows it's name from inotify (because it uses inotify) and when using the -r (recursive) flag it sets up multiple inotify watches to work around the very problem I just described.
In fact it's man page comes with a big fat warning about using -r:
> Warning: If you use this option while watching the root directory of a large tree, it may take quite a while until all inotify watches are established, and events will not be received in this time. Also, since one inotify watch will be established per subdirectory, it is possible that the maximum amount of inotify watches per user will be reached. The default maximum is 8192; it can be increased by writing to /proc/sys/fs/inotify/max_user_watches.
> inotifywait is a user-space program that borrows it's name from inotify (because it uses inotify) and when using the -r (recursive) flag it sets up multiple inotify watches to work around the very problem I just described.
What problem?
What you quoted is totally irrelevant for my workflow. On my crappy laptop and all projects in my work directory, the watches are established almost instantly, without any spike in CPU.
There are 4677 directories in Linux kernel and you can just put a knob change in /etc/sysctl.d/ if your code base is bigger than kernel.
You're just inventing OS limitations to justify limitations of lazy user-space programming.
EDIT: Setting up watches on kernel tree including directories in .git:
~/src/linux % inotifywait -r -m -e close_write --format %f . |& ts
Jul 01 16:03:10 Setting up watches. Beware: since -r was given, this may take a while!
Jul 01 16:03:10 Watches established.
Your workflow might be ok, others might not. But blaming developers for being "lazy user-space [programmers]" when Linux lacks a feature that Windows and macOS support, and which could cause complications when worked around, is a really unproductive way to hold a discussion on HN.
You still didn't specify which feature Linux lacks that Windows and macOS supports. I have shown you that there's no need for any workaround, that the OS feature works as intended, but it seems you're keen on believing FUD scraped from web pages of lazy user-space programmers instead of considering my arguments and evidence.
FYI: What OS interface do you think that `entr` uses on Linux? That's right, inotify.
Not the original poster, but Windows allows you to get all changes in an NTFS volume, effectively allowing you to monitor notifications to an unlimited amount of files and directories (didn't remember what the feature was called, a quick googling led me to https://docs.microsoft.com/en-us/windows/win32/fileio/change... ).
This allows software like Voidtool's Everything to index the entire filesystem and update that index in real time. Using inotify for that is not possible since you'd have to recursively register for hunderds of thousands of directories, which far exceeds the limit of allowed registerations (and either way, would be quite wasteful to hold 100,000 registration handles). As a result, the Linux equivalent to Everything is unable to find recently created files, erronously finds deleted files and has outdated metadata fo recently changed files. Alternatively, you can attempt to reindex right before searching (e.g. rerun updatedb if you are using locate), but then your searches are much slower and the benefit of indexing for a quick search is reduced significantly.
A nicer API would allow to use one registration handle to get notifications for an entire directory tree or at least have a seprate API for an entire filesystem (like the NTFS option mentioned above).
At least, that was the conclusion I came to when I last researched the topic. If you know of a supported way of doing this on Linux I'd really love to know! It will allow me to finally make the program I wanted to make!
> As a result, the Linux equivalent to Everything is unable to find recently created files, erronously finds deleted files and has outdated metadata fo recently changed files.
You're being quite aggressive about your disagreement here. I don't think it's "FUD" to rightfully point out that using inotify on a large directory tree takes extra userspace code that is explicitly called out in the inotifywatch docs as performing poorly enough to worry about race conditions. Not to mention that it requires a limits tweak. This isn't "lazy," it's wishing for the mechanism to be more capable. Would you have this same attitude if the default limit were 1024? 512?
Why wonder what my attitude would be with lower limit? Default limit is 8192 and it's OK for use case exemplified by entr and you can easily change it.
I agree with your frustrations around this limitation and think file watchers should do their best to support arbitrary file addition/removal in directories. That being said, the races do matter when you edit with "mammal speed" but use source control to merge/rebase commits that affect your watched directories. Especially in a team setting when multiple coworkers are working on the same directories, git can be blasting rapidly through a lot of file moves, deletions etc so it's not really an edge case.
Git is a good point. However, the only time I desire commands triggered by changes in source tree is during actual code writing to instantly test my changes. On the other hand stopping it during Git operations could harm my workflow. This could be easily solved by simple 1s debounce, with this hypothetical debounce filter
