Since he mentions this being for a work thing, how I've handled a similar situation (long running shell thingies) is:
Break up the script into steps, and place them in order in a dot-d style folder, like 1.foo.sh, 2.bar.sh, 3.baz.sh. Have a separate _context.sh which supplies the upfront variables or whatever are needed for each step. Ensure that each step can run on its own with just the context sourced. Then have a run.sh in the folder above that sources the context and the steps in order.
Now, that said, typically these kinds of things are dataflow or even buildsystem-adjacent, so I've also experimented with doing them in CMake (intermediate products managed with add_custom_target) or as Nix derivations (intermediate products are input-addressed in the Nix store), but I'd love to be aware of other tools that are better for expressing this kind of thing.
I'd write run.sh like this: Put the names of the files from that dot-d style folder into some ".cc" file (if it doesn't exist yet); then loop over the lines in the ".cc" file, executing each line if it doesn't start with '#' and then prepending '#' with sed to the just executed line. After the loop, delete the ".cc" file (maybe with a diagnostic messages if all lines in it were #-prefixed). Maybe throw in some traps for reliable SIGINT-handling as well.
1) Use bash PS4 to show the line number currently executing (perhaps append start of line with #).
PS4='# ${LINENO}: ' bash -x script
2) Log to a script transaction file.
unset PS4 before a function, reset after function call to restart at function call instead of restarting within a function.
trap signal error & dump variables to state file (via env or set command)
3) check for 'transaction file' on restart of script, if exists,
a) restore saved state file settings;
b) pick up where to start via sdiff of original script and transaction file (aka first line not starting with #
sdiff -v ^'# script transaction_file | sh -
I would just use Make, or write a checkpoint file for each step, and wrap them all in if statements so they don't run if their checkpoint is there.
Or, if I were doing it from scratch, I would not have a bash script that takes days to run, that sounds like some Real Programming, and I'd do it in Python, where it's easier to manage complexity, in case there was some more fine grained caching that could be done, maybe even between runs.
Maybe I need to make another coffee or something but I really don't understand at all why they wanted Goto for this.
I feel like there has got to be a better common denominator if you're willing to stick with just Linux.
Yesterday I was trying to track down why a bash script that worked on pretty much every platform I'd tried and across a bunch of different bash versions was not working on a new operating system. Turns out the script was relying on "non-standard" behavior in tr(1). Definitely not the first thing you think of when you see bash complaining about undefined variables.
A bit further back I had similar fun with FreeBSD's installer now that they've ripped perl5 out of the base system. The installer is, of course, an unholy mix of C and sh.
Well, there is Perl, which is what actually was used to be for "can we do this in something that is not shell but also portable?" kinds of scripts for quite some time. But then again, it's Perl.
Why is this a thing still? Who is out there saying "We want you to develop an app but we won't let you use Python"?
Granted, I really wish there was a python-lts that didn't break stdlib stuff every few years, but it seems like everything remotely modern breaks compatibility constantly...
Not many people are saying that, but there's a few cases it could make sense. Like if you want to setup something on an embedded machine with no/restricted internet, then bash scripts will be easier.
Some people also like not having to do extra installs to run something simple.
And some scripts are simple enough that it's just easier to use bash.
But I agree, these cases are pretty obscure and most of the time the benefits of using a real language is worth the investment.
Break up the script into steps, and place them in order in a dot-d style folder, like 1.foo.sh, 2.bar.sh, 3.baz.sh. Have a separate _context.sh which supplies the upfront variables or whatever are needed for each step. Ensure that each step can run on its own with just the context sourced. Then have a run.sh in the folder above that sources the context and the steps in order.
Now, that said, typically these kinds of things are dataflow or even buildsystem-adjacent, so I've also experimented with doing them in CMake (intermediate products managed with add_custom_target) or as Nix derivations (intermediate products are input-addressed in the Nix store), but I'd love to be aware of other tools that are better for expressing this kind of thing.