Yeah, been there, done that, although my first step is to umount and dd onto another volume so the sectors don't get reallocated and obliterate the data - and that's generally a sane first step for ANY recovery of this ilk.
You haven't lived until you unwittingly run rm -rf on a nfs mount of / on a remote box. Which happens to be the fileserver for a trading shop. In the middle of trading hours.
Indeed. The whole "never run rm -rf" thing is so ingrained that someone I know took real joy in finally getting to legitimately use this to nuke an old server once the new one was signed off.
Pity he'd not considered that the old one had nfs mounts to the live data...
I think 'rm -rf /' shouldn't work and there should be special command instead of 'rm -rf /', something like rmrfhell, which would ask you three times if you sure and will tell you about pain and tears of other people who ran it.
Most versions have --preserve-root (fail if the target is /) on by default, so you have to use --no-preserve-root , although rm -rf /* will still work.
I was sysadmin for a small ISP about 15 years ago. I was on vacation about 1500 miles away when I called to check in, the owner gets on the phone and says, "I deleted the entire /bin directory on [the primary web server], is that bad?" I told him to not touch it til I got home, and whatever he did do NOT shut it off!
Thankfully we had two machines which were virtually identical OS-wise (RedHat 6 if memory serves). I was able to get everything put back from the twin machine and keep everybody happy.
Thankfully that server kept running with relatively little issue the entire time even with all those core OS files gone. I don't think any customers were at all aware.
Open processes will hold open their files. So long as it's an 'rm' that you've run (which merely removes directory entries) and not a destructive action on the disk contents themselves, it's often possible for things to continue in a startlingly unaffected manner. Though not always.
The extent to which new calls to deleted files are made will have a strong impact on this.
Is unmount really the best first action, surely that forces a sync and that could be when the data actually gets removed?
I seem to remember having a problem where I'd rm-ed a file I needed and using lsof to find the file handle and then being able to cat the data in to a new file using the handle instead of the filename? Details pretty fuzzy, sorry.
Edit:
Example of recovery this way, tested now, works for me - http://pastebin.com/c2djEcqr - the crucial part that I was forgetting is that there needs to be a file handle somewhere that's still open, which is probably not true in most cases. Worth a quick check before unmounting.
I've used ddrescue and photorec for these sort of "issues" before with much success.
You haven't lived until you unwittingly run rm -rf on a nfs mount of / on a remote box. Which happens to be the fileserver for a trading shop. In the middle of trading hours.