I think it's less about pure suitability, and more about availability.
If I write a Bash script, I can be reasonably certain it will run on any unixoid system that came out in the last years. With Python, I am now in version hell (it's ridiculous how many Python2-first-servers I still find), I'll run the risk of includes suddenly becoming incompatible or buggy.
Can Bash do everything that Python can? Almost certainly not. But it is available, it is relatively simple (and almost minimalistic), and it forces you to learn more about standard unix tools.
Your specific use case ... well, I think that's pretty unusual - I'd wager if you had avoided the parallel processing and done it in a more linear way, you've had a better time.
That makes sense! I will admit it was a bit of fun clobbering together the bash version at first. It reminded me of starting out programming where everything felt a bit esoteric lol.
> if you had avoided the parallel processing and done it in a more linear way, you've had a better time.
I had to process ~500,000 files, and the aws api call was on average ~1 second, so it would have been a significantly longer time to process linearly. For example the bash version I whipped up processed ~30k files in 2.5 hours, while the python version did 30k files in ~20 minutes.
But yes I agree, if I didn't have to do such a large volume at once the bash version would have been just fine.
Other than bash not being portable to even the exact same machine, due to having so many footguns. Creating a new file can potentially fail a previously working bash, because it may have used .txt expecting a single file, or whatever, and there are a million others.
Don’t get me wrong, I accept that there are legitimate use cases for bash, but I really can’t help but feel that anything longer than 3 lines (including #!) is better off in anything*.
If I write a Bash script, I can be reasonably certain it will run on any unixoid system that came out in the last years. With Python, I am now in version hell (it's ridiculous how many Python2-first-servers I still find), I'll run the risk of includes suddenly becoming incompatible or buggy.
Can Bash do everything that Python can? Almost certainly not. But it is available, it is relatively simple (and almost minimalistic), and it forces you to learn more about standard unix tools.
Your specific use case ... well, I think that's pretty unusual - I'd wager if you had avoided the parallel processing and done it in a more linear way, you've had a better time.