Yes, we've thought a lot about using inotify, our first prototype was also using inotify.
- Our system needs to cope with millions of directories. Millions of directories for inotify mean a lot of structures in the kernel. For large numbers it can also mean gigabytes of ram. Add to that the mapping in userspace of file descriptors for rename operations.
- Using inotify would take a lot of time at startup to recurse into millions of subdirectories.
- inotify will not automatically watch new directories. You have to list all the files right after the creation of a directory and watch the subdirectories, etc. Not a problem at all, but way simpler with FUSE.
- If your system is not able to keep up with inotify events, you miss events because the kernel cannot buffer all the events. For us it's better to slow down the system but never miss an event.
- inotify is attached to the original filesystem. That means it's hard (not impossible) to handle loops in an active/active replication setup. Whereas with SFS, replication is done on straight to the original filesystem.
- If the inotify application crashes, you lose events because software keeps writing to the filesystem. If SFS crashes the mountpoint is unwritable by the application which reports an error and can switch to a different storage.
Some choices as you can read depends on the requirements, our requirements where not met by inotify.
- Our system needs to cope with millions of directories. Millions of directories for inotify mean a lot of structures in the kernel. For large numbers it can also mean gigabytes of ram. Add to that the mapping in userspace of file descriptors for rename operations.
- Using inotify would take a lot of time at startup to recurse into millions of subdirectories.
- inotify will not automatically watch new directories. You have to list all the files right after the creation of a directory and watch the subdirectories, etc. Not a problem at all, but way simpler with FUSE.
- If your system is not able to keep up with inotify events, you miss events because the kernel cannot buffer all the events. For us it's better to slow down the system but never miss an event.
- inotify is attached to the original filesystem. That means it's hard (not impossible) to handle loops in an active/active replication setup. Whereas with SFS, replication is done on straight to the original filesystem.
- If the inotify application crashes, you lose events because software keeps writing to the filesystem. If SFS crashes the mountpoint is unwritable by the application which reports an error and can switch to a different storage.
Some choices as you can read depends on the requirements, our requirements where not met by inotify.