fork() and exec() are fine. As long as you aren't running arbitrary binaries, it's pretty hard to mess up (as far as RCE goes). system(), however, is a RCE foot-cannon just waiting to happen. Don't ever use system() unless every argument is static. And even then think long and hard.
Thanks for linking to libpipeline; I didn't know about it. The API looks good. However, since it is licensed under the GNU GPL (the initial version was based on code in GNU troff) non-copyleft libraries like ImageMagick are unlikely to adopt it.
If you are looking for a small, standalone, non-copyleft alternative, the closest I can think of is the [exec] command in the Jim Tcl [1] embedded scripting language.
Here is an example (without the error checks you'd have in production code):
#include "jim.h"
int main(int argc, char const *argv[])
{
Jim_Interp *interp;
int error;
Jim_Obj *cmd;
interp = Jim_CreateInterp();
Jim_RegisterCoreCommands(interp);
Jim_InitStaticExtensions(interp);
// The input redirect below does *not* invoke the POSIX shell. It is handled by Jim Tcl itself.
cmd = Jim_NewListObj(interp, NULL, 0);
Jim_ListAppendElement(interp, cmd, Jim_NewStringObj(interp, "exec", -1));
Jim_ListAppendElement(interp, cmd, Jim_NewStringObj(interp, "awk", -1));
Jim_ListAppendElement(interp, cmd, Jim_NewStringObj(interp, "1", -1));
Jim_ListAppendElement(interp, cmd, Jim_NewStringObj(interp, "<", -1));
Jim_ListAppendElement(interp, cmd, Jim_NewStringObj(interp, "/etc/passwd", -1));
error = Jim_EvalObj(interp, cmd);
if (error != JIM_ERR) {
printf("%s\n", Jim_String(Jim_GetResult(interp)));
}
Jim_FreeInterp(interp);
return error;
}
While I am a fan of the language and of the "hard and soft layers" approach in general [2], it is a commitment: it requires you to embed the language runtime in your program and learn the basics of the scripting language itself and its C API. The upside is that Jim Tcl's [exec] works on Windows, too (and you get other goodies with Jim like a fast, high quality implementation of strings and hash maps).
Alternatively, you can use the "big" Tcl [3] as a C library. It's larger but more mature and is available in every major Linux distribution.
To elaborate on the above a bit, for me the choice between Jim Tcl and the "big" Tcl 8.x comes down to choosing between vendoring the dependencies and embedding the runtime (Jim Tcl) vs. using the distribution libraries and extending the runtime (Tcl 8).
Either way you get a very fine C library that prevents you from succumbing to Greenspun's tenth rule, so I can only recommend it for most C programs and libraries of sufficient size and complexity.
Oh yes, it is pretty widely available. What I meant was that it was smaller and easier to vendor: for example, you can produce an SQLite-style amalgamation file for it.
In my opinion Tcl really straddles the line between a language and a library. I've heard it favorably compared to GLib. When used from C it feels a lot more like a library than other scripting languages do (Tcl 8 even more so than Jim due to a greater C API surface).
I agree with you on writing in another language but presumably you wouldn't be considering libpipeline anyway unless you had to write C. My default approach when I need C to access some API is to write an extension to an interpreter from which to access it. I leave embedding an interpreter in C for when you can't do that.
In the meantime, maybe a DONT_RUN_COMMANDS ifdef around every call to fork/system/exec is merited...