One of my proudest moment in my career is when I lowered our app processing time from ~8hrs to 17 minutes. When I deployed my first update, it reduced it to 2 hours. The sysadmin immediately contacted me that there was something unusual. I confirmed the results but he was skeptical.
Then with my second update, he told me that the app must be broken or that the script must be dying. There is no way it could complete this fast.
What was the issue? We processed terabytes of data. Each and every single line processed created a new connection to the database and left it hanging. A try catch was added when the connections failed and restarted the process. Removing the connection from the for loop and properly handling it reduced the time drastically.
And... why would you loop through millions of records when you can use batches? Also this was a phperlbashton* script. I turned it into a single PHP script and called it a day.
As a consequence, backup time was reduced to 2 hours as opposed to 12 hours (no one was allowed on the website until the back up was done).
I have a similar story where I reduced the memory used by a script from 1TB (yes, TB) to a few megabytes. The runtime was massively reduced too, from something like 1 day to a few minutes.
This was for a genomics project and they ran it on a supercomputer. When I looked into it, they were reading the entire input into a giant array before doing one pass and dumping the result out to disk. I made a tiny change (it was a Perl script) to make it stream the I/O instead.
This is the most extreme example I've come across of people using computing power just because it's there. Nobody questioned why the script took so long to run because the data really was in the TBs and other stuff also took that long to run. Waiting a day for the results was considered normal. I see the same thing on desktop apps etc., on a much smaller scale, of course. When I run an electron app it takes several hundred milleseconds to do anything at all. But nobody questions whether it should because everything takes several hundred milliseconds.
Mine was when I rewrote some application code to be processed within postgres to send a few thousand sms's (the logic to decide who should get them was the slow part + amount of data involved). It went from about 45 minutes to less than 1 minute. Was an amazing feeling to see the sms table filled up with the correct data - I also thought something was broken since we just accepted that those runs were normally supposed to be slow.
I had a similar experience as a intern years ago! Part of my daily responsibility was to manually enable/disable API integrators who had increased levels of traffic as it would take the API down for all partners. Pretty bad. Thousands of requests at peak would bring the server to its knees.
Until I worked out during some minor maintenance task that every request was logged to a flat file. Appended. Every request. The file was probably 100gb by the time I found it and every request log would lock the logging file. The server had been running for a couple of years by that time.
Then with my second update, he told me that the app must be broken or that the script must be dying. There is no way it could complete this fast.
What was the issue? We processed terabytes of data. Each and every single line processed created a new connection to the database and left it hanging. A try catch was added when the connections failed and restarted the process. Removing the connection from the for loop and properly handling it reduced the time drastically.
And... why would you loop through millions of records when you can use batches? Also this was a phperlbashton* script. I turned it into a single PHP script and called it a day.
As a consequence, backup time was reduced to 2 hours as opposed to 12 hours (no one was allowed on the website until the back up was done).
Modern machines are incredibly fast.
* PHP/Perl/Bash/Python