I came here expecting someone to recommend 'make' - and some others to link to interesting make resources. I'm kind of surprised no one has.
Make is a utility that's seemed super useful to put into use for non-programming tasks. In my case: take a LaTeX file, create a PDF from it, run tex2html on it and ftp the results up to some web server, run pandoc on it to create a dated epub, etc. But my initial forays into learning make were frustrating and I gave up. Is anyone using make to automate workflow processes, or is that wishful thinking?
The thing about make is that it's a tool for reducing how much computation has to take place when an incremental rebuild takes place.
This is done at the cost of great obfuscation of the build process.
If you can suffer the cost of a full rebuild each time you change something, it is far clearer to have a linear script which executes certain steps one by one.
Such a script can be sped up for incremental builds with some judicious checks like "run this command which makes A out of B, only if A doesn't exist or is older than B".
(P.S. I don't want to overlook make's ability to parallelize builds on multiple cores. That can just be scripted too, with utilities like GNU Parallel.)
Make doesn't really force an incremental build. It's also quite trivial to have a semi-incremental build, where the build unit is libraries not object files.
I'm yet to see a tool that doesn't become just as obfuscated for complex builds though, you're script for instance (bash? python?), is probably just as illegible to someone new to it.
Make doesn't force an incremental rebuild, but that is what it is for. It lets us write a set of rules, which are evaluated as one big whole to determine the minimal set of actions that has to be taken to bring every implicated target up to date. That is pretty much the definition of "incremental rebuild".
If a program is made of libraries, such that when we change a single source file, the entire library which contains the .o has to be re-archived, and then the entire program made up of all those libraries has to be re-linked, that is still an "incremental build".
You might possibly be mixing up "incremental link" (optimized way to just update a function or object file in a previously linked program image) with "incremental build" (compiling only parts of a program in response to a small change).
It is also a bunch of scripts in the one known file, and these can be dependent, programmed (if/for) or included from other sources. If you don't care about what it is mainly for, you can see the value of its own.
Honestly, I'll never go into some .sh file to see what happens there, but often open Makefiles to find and tune details of anything. It... adds something manageable(?), I can't find a word for it.
Of course I use makefiles for tasks not connected to building projects. Not "a lot", but if I need reproducible actions in workdir, then it is Makefile.
> It is also a bunch of scripts in the one known file, and these can be dependent, programmed (if/for) or included from other sources
So it's not just one known file. Big trees can have lots of little makefiles.
> I'll never go into some .sh file to see what happens there, but often open Makefiles to find and tune details of anything.
Thanks for sharing. My own personal preference is such that I will never go into a Visual Basic file, but I will go into a Lisp.
You do know that the recipes in Makefiles are shell scripts? (With a subtle difference: each line implicitly executes in its own subshell, so setting variables or doing "cd" doesn't propagate). In a Makefile you still have to read shell code. It's mixed with make code, and with additional quoting rules and where true multi-line shell scripts that run in a single shell have to be written with backslash escapes; ugh.
(Note by the way that I didn't say anything about specifically using shell scripts for builds; I deliberately used the word "script" without qualification.)
> Not "a lot", but if I need reproducible actions in workdir, then it is Makefile.
If you want irreproducible parallel build problems, that's when you want make.
Sometimes Makefiles depend on execution orders which are not actually encoded as dependencies and these non-asserted orders turn into race conditions in a parallel build. You get a situation where in one out of twenty builds, that "y.tab.h" file out of yacc didn't get generated yet, while the lexical analyzer that wants to #include-s it is already being built. In single threaded mode, it just so happens that the rule which makes "y.tab.h" always runs first.
I once created an embedded Linux distro from scratch (build and packaging system and all). Over the base of supported packages we were pulling in, I saw quite a number of build issues of this type when the distro build would randomly fail to build a package due to race conditions like this.
GNU Make doesn't have any diagnostic ability that I know of to help you uncover where rules have unexpressed dependencies (a recipe refers to an input file which is the output of another rule, and that input file is not named as a prerequisite).
I've been too lazy to learn make, but I pull a python library named doit when I need to do automation that is more complex than a linear shell script (dependencies, etc).
I use make for scientific data analysis, and it has saved me a tremendous amount of time. It's definitely worth the investment to learn the language, in my opinion. I know many folks have tried writing their own version to "fix" the arcane parts, and maybe they're better in some use cases, but make is thoroughly documented and tested and quite general purpose.
I tried this once to automate a several part procedure. After spending a couple evenings, I decided it is hopeless. make is coupled tightly to building projects, shoehorning non-build workflows into a makefile is both awkward and incomprehensible to colleagues.
Well, personally I would just use a shell/python script to do the same. I dont see the point of learning an arcane language for something any language can do easily.
Why have Python when you can use Shell to do the same? Why have Shell when you have C? Why have C when you have ASM?
All of those increments make particular tasks easier. Makefiles eliminate the need to write 'if this doesn't exist or it exists but is older than its dependencies' over and over again. The built in targets also make building C programs quite trivial.
Makefiles can get nasty, but for simple chains of "if this file has been modified take this action" they're a great tool. The syntax isn't even that hard for those types of actions.
Make is a utility that's seemed super useful to put into use for non-programming tasks. In my case: take a LaTeX file, create a PDF from it, run tex2html on it and ftp the results up to some web server, run pandoc on it to create a dated epub, etc. But my initial forays into learning make were frustrating and I gave up. Is anyone using make to automate workflow processes, or is that wishful thinking?