I can think of two examples. The first is somebody else's that I helped them with, and the second is mine. Context: Mid-2000's snack-sized hosting company with less than 10 employees.
A customer with a dedicated server called in one day. I knew him well, and he I. It was that kind of company and I loved it. Anyhow, he called in and said "I've been hacked!" I asked him to explain the problem and he said that his server wasn't working right and wouldn't reboot. I looked into it, and a ton of things were missing from the file system. /bin, /boot, /dev, etc. Gone. So I did a restore of the server and got it back online.
He called me up an hour or two later "I've been hacked again!" Same problem. Same solution.
An hour later he called me up, sheepish. "I think it might be ME. Every time I think it's happening every time I run this command." I managed to get him to email me the command he was running, verbatim (he was copy/pasting it into the shell) and it looked like this:
The other harm was my own. I was budding system administrator tasked with designing and deploying dedicated servers (this is before VPSs were a thing, and way before cloud computing). I challenged myself to learn Perl because the greybeard admin (who cut his teeth on PDP-11's) was a master at it and I could benefit from his tutelage. But I didn't. Instead I got the SAM's book on Perl and started learning. Having an interest in security, I decided that I'd write a simple Intrusion Detection System for our dedicated server products.
I don't have the code, but it's irrelevant. A the time one of the bigger problems was servers getting rooted due to various exploits. The attacker would rewrite /bin/ls with their own version that hid their executables. The solution (obviously right?) is to look at the md5sum of /bin/ls, and if it changes, the server must be rooted, right? So I wrote code that would shut down sshd, close the port, and display a message via `wall` that said something very much like:
System Modification Detected! Shutting down system and alerting security staff. YOU HAVE BEEN CAUGHT! BUH BYE!
I was very pleased with myself. I couldn't understand why the aforementioned greybeard didn't revel in my brilliance. sigh I deployed the script on a cron job and felt very satisfied with my good work. Until a call came in. "Yeah, I ran a yum update, got some message about being caught, and now I can't get into my system."
Yeah that's right. I'd hard coded the md5sum for /bin/ls and had no means of updating it. I've since forgotten Perl but I never forgot this lesson: Take everything into account. Spend less time coding and more time thinking about what you're building and how it'll affect others. Since then I've written several utilities that are/were in use at the hosting companies I work(ed) for and I saved people countless time, effort, and trouble. But it sure didn't start out that way!
A customer with a dedicated server called in one day. I knew him well, and he I. It was that kind of company and I loved it. Anyhow, he called in and said "I've been hacked!" I asked him to explain the problem and he said that his server wasn't working right and wouldn't reboot. I looked into it, and a ton of things were missing from the file system. /bin, /boot, /dev, etc. Gone. So I did a restore of the server and got it back online.
He called me up an hour or two later "I've been hacked again!" Same problem. Same solution.
An hour later he called me up, sheepish. "I think it might be ME. Every time I think it's happening every time I run this command." I managed to get him to email me the command he was running, verbatim (he was copy/pasting it into the shell) and it looked like this:
The other harm was my own. I was budding system administrator tasked with designing and deploying dedicated servers (this is before VPSs were a thing, and way before cloud computing). I challenged myself to learn Perl because the greybeard admin (who cut his teeth on PDP-11's) was a master at it and I could benefit from his tutelage. But I didn't. Instead I got the SAM's book on Perl and started learning. Having an interest in security, I decided that I'd write a simple Intrusion Detection System for our dedicated server products.I don't have the code, but it's irrelevant. A the time one of the bigger problems was servers getting rooted due to various exploits. The attacker would rewrite /bin/ls with their own version that hid their executables. The solution (obviously right?) is to look at the md5sum of /bin/ls, and if it changes, the server must be rooted, right? So I wrote code that would shut down sshd, close the port, and display a message via `wall` that said something very much like:
I was very pleased with myself. I couldn't understand why the aforementioned greybeard didn't revel in my brilliance. sigh I deployed the script on a cron job and felt very satisfied with my good work. Until a call came in. "Yeah, I ran a yum update, got some message about being caught, and now I can't get into my system."Yeah that's right. I'd hard coded the md5sum for /bin/ls and had no means of updating it. I've since forgotten Perl but I never forgot this lesson: Take everything into account. Spend less time coding and more time thinking about what you're building and how it'll affect others. Since then I've written several utilities that are/were in use at the hosting companies I work(ed) for and I saved people countless time, effort, and trouble. But it sure didn't start out that way!