Since ext4 became the default, my most common cause of bizarre behaviors has been running out of inodes.
df -i
will show this, but a bare df command will not.
Getting off topic now, but does anybody know if the ext4 utilities changed the calculation of inodes when formatting compared to ext3/ext2 utilities? Running out of inodes on those filesystems was fairly unusual, but I've seen it happen a dozen times in the past 5 years on ext4.
Across the approximately 2000 VMs we look after as an MSP, we see this at least once a week. It's almost always badly cleaned up session files from a php app or similar.
The default ratio of inodes is configured in /etc/mke2fs.conf. You should either change that, override it with the -i argument when you create the filesystem.
My desktop's / filesystem is around 238,000,000,000 bytes, and with the default ratio of one inode per 16384 bytes, I have around 14,500,000 inodes.
Note that with a default blocksize of 4096, you're limited to 4× as many inodes as you have at present, so if you're seeing this weekly I recommend monitoring the number of remaining inodes (df -i), or changing the app to store sessions in a database.
We're talking about preexisting systems. inodes are already tweaked up during build to highest, and we monitor similar to how you suggest via a checkmk script. If the apps running were ours we could make changes to make them clean up better or store through other means, alas this is just something you have to handle when doing break fix response on 100s of clients making their own decisions.
Getting off topic now, but does anybody know if the ext4 utilities changed the calculation of inodes when formatting compared to ext3/ext2 utilities? Running out of inodes on those filesystems was fairly unusual, but I've seen it happen a dozen times in the past 5 years on ext4.