I write scheme not for a living. My build script is a scheme file or a makefile, depending on the project.
The build complexity is one of the reasons I stay away from Java: If these people think they need XML for builds, what other needlessly complex horrors have they perpetrated? And that sort of thing.
The Maven POM syntax might look bloaty to modern post-XML eyes, but it's not actually complicated: a typical POM only specifies the name of the project and its dependencies. The rest is typically all done by convention. It's also purely declarative so ordering doesn't matter.
M4 and sh are very concise languages. Nonetheless autotools is orders of magnitude more complex than Maven. You really can't compare at all.
At any rate, if you want a more concise syntax there is gradle (but it's a bit slower as it's actually executing a real scripting language) and, perhaps a nice middle ground, a thing called Polyglot Maven which is the same build engine but with a variety of non-XML syntaxes. The YAML one is quite nice:
To this effect, I use leiningen (which is mostly a frontend atop Maven's dependency management) even when I'm building a Jython or a Java project. If there is something funky I need to do as a build/deploy step, I'd rather be writing it in Clojure than Java, and mostly I just want to paste [com.example/foo "1.5.0"] into my :dependencies.
I'm beginning to think that build scripts should be written in the language they are building, or a more expressive language. So many seem to go in the other direction.
I disagree. The worst problems I see with build scripts are that they are trying to do too much, create branches/tags of themselves, be environmentally aware, etc.
I really do agree that complexity is the enemy, but people keep shoving complexity into build scripts against better judgment. I've given up and would rather deal with a complex ruby script than a makefile of equal intrinsic complexity.
That looks nice. It combines strong, static analysis with simple DSL and underlying power in language if necessary. Looks like The Right Thing approach in action.
You actually NEED arbitrary code execution during builds. Allow me to explain...
Let's say we have a make format called dmake. It invokes $CC with the specified arguments for each file, and links them together into a binary/so/whatever, putting it into the build directory and cleaning artifacts. Okay.
Now say that you start a new project in rust. Well, crap, dmake doesn't work. You have to use rdmake, which is built by different people, and uses a more elegant syntax - which you don't know.
Then you write Haskell, and have to use hdmake - which of course is written as a haskell program, using a fancy monad you don't know, and python has to use pydmake, and ruby has to use rbdmake, and scheme has to use sdmake, and lisp has to use ldmake, and asm has to use 60 different dmakes, depending on which asm you're using.
Instead, we all use make. Make allows for arbitrary code to be executed, so no matter what programming environment you use, you can use a familiar build tool that everybody knows. Sure, java has Ant, Jelly, Gradle and god knows what else, and node has $NODE_BUILD_SYSTEM_OF_THE_WEEK, but even there, you can still use make.
You haven't countered parent's point at all. You could've just as easily said the common, subset of SQL could be implemented extremely different in SQL Server, Oracle, Postgres, etc. Therefore, declarative SQL has no advantages over imperative, C API's for database engines. Funny stuff.
Let's try it then. The declarative, build system has a formal spec with types, files, modules, ways of describing their connections, platform-specific definitions, and so on. Enough to cover whatever systems while also being decidable during analysis. There's also a defined ordering of operations on these things kind of like how Prolog has unification or old expert systems had RETE. This spec could even be implemented in a reference implementation in a high-level language & test suite. Then, each implementation you mention, from rdmake to hdmake, is coded and tested against that specification for functional equivalence. We now have a simple DSL for builds that checks them for many errors and automagically handles them on any platform. Might even include versioning with rollback in case anything breaks due to inevitable problems. A higher-assurance version of something like this:
Instead, we all use make. Make allows for arbitrary code and configurations to be executed, so no matter what configuration problems you have, we can all use a familiar build tool that everybody knows. That's the power of generic, unsafe tools following Worse is Better approach. Gives us great threads like this. :)
From the perspective of security, make is not great, but there's always a more complicated build, requiring either generic tooling, or very complex specific tooling. This is why the JS ecosystem is always re-inventing the wheel. If you design your build tool around one abstraction, there will always be something that doesn't fit. What will happen if we build a tool akin to the one I described is that it will grow feature upon feature, until it's a nightmarish mess that nobody completely understands.
>You could've just as easily said the common, subset of SQL could be implemented extremely different in SQL Server, Oracle, Postgres, etc. Therefore, declarative SQL has no advantages over imperative, C API's for database engines. Funny stuff.
No, that's not my point, my point is that a build tool that meets parent's requirements would necessarily be non-generic, and that such a tool would suffer as a result.
>Instead, we all use make. Make allows for arbitrary code and configurations to be executed, so no matter what configuration problems you have, we can all use a familiar build tool that everybody knows. That's the power of generic, unsafe tools following Worse is Better approach. Gives us great threads like this. :)
Worse is Better has nothing to do with this. Really. Make is very Worse is Better in its implementation, but the idea of generic vs. non-generic build systems, which is what we're discussing, is entirely orthogonal to Worse is Better. If you disagree, I'd reccomend rereading Gabriel's paper (that being Lisp, The Good News, The Bad News, And How to Win Big, for the uninitiated). I'll never say that I'm 100% sure that I'm right, but I just reread it, and I'm pretty sure.
"No, that's not my point, my point is that a build tool that meets parent's requirements would necessarily be non-generic, and that such a tool would suffer as a result."
A build system is essentially supposed to take a list of things, check dependencies, do any platform-specific substitutions, build them in a certain order with specific tools, and output the result. Declarative languages handle more complicated things than that. Here's some examples:
I also already listed one (Nix) that handles a Linux distro. So, it's not theory so much as how much more remains to be solved/improved and if methods like in the link can cover it. What specific problems building applications do you think an imperative approach can handle that something like Nix or stuff in PDF can't?
Didn't know that. Interesting. It looks like an execution detail. Something you could do with any imperative function but why not use what's there for this simple action. Nix also manages the executions of those to integrate it with their overall approach. Makes practical sense.
"It's fully generic."
It might help if you define what you mean by "generic." You keep using that word. I believe declarative models handle... generic... builds given you can describe about any of them with suitable language. I think imperative models also handle them. To me, it's irrelevant: issue being declarative has benefits & can work to replace existing build systems.
So, what's your definition of generic here? Why do declarative models not have it in this domain? And what else do declarative models w/ imperative plugins/IO-functions not have for building apps that full, imperative model (incl make) does better? Get to specific objections so I can decide whether to drop declarative model for build systems or find answers/improvements to stated deficiencies.
That wasn't what the original post by ashitlerferad was calling for. I have not problem with generic declararive-model build systems that can be used for anything. However, the original call was for build systems which don't require arbitrary code execution. A generic build system must deal with many different tools and compilers, and thus REQUIRES arbitrary code execution: Somewhere, there's got to be a piece of code telling the system how to build each file. And if you don't build that into the build system proper, you wind up either integrating everything into core, or adding an unweildly plugin architecture and winding up like grunt/gulp and all the other node build systems. Or you could just allow for arbitrary code execution, and dodge the problem all together. This is possible in a declaritive system, but it's a lot harder to do, and means at least part of your system must be declarative.
It seems some kind of arbitrary execution is necessary. I decided to come back to the problem out of curiosity to see if I could push that toward declarative or logic to gain its benefits. This isn't another argument so to speak so much as a brainstorm pushing envelope here. Could speculate all day but came up with a cheat: it would be true if anyone had replaced make or other imperative/arbitrary pieces with Prolog/HOL equivalents. Vast majority of effort outside I/O calls & runtime itself would be declarative. Found these:
Add to that Myreen et al's work extracting provers, machine code and hardware from HOL specs + FLINT team doing formal verification of OS-stuff (incl interrupts & I/O) + seL4/Verisoft doing kernels/OS's to find declarative, logic part could go from Nix-style tool down to logic-style make down to reactive kernel, drivers, machine code, and CPU itself. Only thing doing arbitrary execution, as opposed to arbitrary specs/logic, in such a model is what runs first tool extracting the CPU handed off to fab (ignoring non-digital components or PCB). Everything else done in logic with checks done automatically, configs/actions/code generated deterministically from declarative input, and final values extracted to checked data/code/transistors.
Hows that? Am I getting closer to replacing arbitrary make's? ;)
...I'm not sure I totally understand. Here's how I'd solve the problem:
Each filetype is accepted by a program. That program is what we'll want to use to compile or otherwise munge that file. So, in a file somewhere in the build, we put:
*.c:$CC %f %a:-Wall
*.o:$CC %f %a:-Wall
And so on. The first field is a glob to match on filetype,%f is filename, %a is args, and the third field is default args, added to every call.
Target all is run if no target is specified. The first field is the target name. The second field is list of files/targets of the same type, to be provided to compiler on run. It is assumed the target and its resultant file have the same name. The last field is a list of additional args to pass to the compiler.
This is something I came up with on the spot, there are certainly holes in it, but something like that could declaritivise the build process. However, this doesn't cover things like cleaning the build environment. Although this could be achieved by removing the resultant files of all targets, which could be determined automatically...
There you go! Nice thought experiment. Looks straight-forward. Also, individual pieces far as parsing could be auto-generated.
Far as what I was doing, I was just showing they'd done logical, correct-by-construction, generated code for everything in the stack up to OS plus someone had a Prolog make. That meant about the whole thing could be done declaratively and/or c-by-c with result extracted with basically no handwritten or arbitrary code. That's the theory based on worked examples. A clean, integration obviously doesn't exist. The Prolog make looked relatively easy, though. Mercury language make it even easier/safer.
Yep. If you're running it for a build, you're running unprivilaged - chrooted, jailed, or zoned if you want to be really safe - and if you're running it for install, than you trust the software in any case. And because makefiles are fairly transparent, you can check what the install is doing beforehand.
XML files cannot be easily processed with standard UNIX tools like grep, sed, and AWK. XML requires specialized libraries and tools to process correctly, making it an extremely poor choice for... well, just about anything. It's a markup format for text, not a programming language.
Building software is a programmatic process. No XML please! We're decidedly not on Windows, and since I have the misfortune of fitting such square pegs into round holes, please don't use XML for applications which must run on UNIX. It's a nightmare. It's horrible. No!!!
There is no particular relationship between Windows and XML. And just to play devil's advocate, is the lack of XML support in grep, sed, and awk a problem with the data format or with the tools? Why can't we have new standard tools that operate on hierarchical formats such as XML / JSON / YAML? Current standard Unix tools have plenty of flaws and as forward thinking developers we shouldn't be afraid to replace them with something better.
I have noticed a particular relationship between Windows, Java, and XML: all Java programmers nowadays seem to come from Windows (and then I end up with ^M CR characters in all the text files, even shell scripts!), use Java, and write configuration in XML.
YAML doesn't need any special tools - it's ASCII and can easily be processed with AWK, for example.
I don't know about you, but the last thing I want is to have to have a whole new set of specialized tools, just so somebody could masturbate in XML and JSON.
XML is a markup language. That means it's for documents, possibly for documents with pictures, perhaps even with audio. It's not and never was meant for storing configuration or data inside of it. XML is designed to be used in tandem with XSLT, and XSLT's purpose is to transform the source XML document into (multiple) target(s): ASCII, ISO 9660, audio, image, PDF, HTML, whatever one writes as the transformation rules in the XSLT file. XML was never meant to be used standalone.
If you really want to put the configuration into an XML file, fine, but then write an XSLT stylesheet which generates a plain ASCII .cf or .conf file, so its processing and parsing can be simple afterwards. XML goes against the core UNIX tenet: keep it simple.
Do you like complex things? I do not, and life is too short.
If you must have structured data, use a lisp program. Congratulations on using a format that was designed to be executable. and if it's a build tool, you better believe it's executable. I suspect that Annatar is a Murray Hill purist (I don't know for sure), so he may disagree with me.
Of course, like any real programming language, it's hard to process with regex, but then again, I don't want to process makefiles with regex. And you might have some luck coaxing AWK or the SNOBOL family to parse it, and it would be far easier than doing the same with XML.
>please don't use XML for applications which must run on UNIX. It's a nightmare. It's horrible. No!!!
I'd disagree with you there. DocBook, HTML, and friends, are all good applications of XML (or near XML), doing what XML was designed for: Document Markup.
Seriously people, when you're writing a program in a language that has "Markup Language" in the name, does that not ring any alarm bells?
No, I wrote that XML for use in applications is bad, as it cannot be easily processed with standard UNIX tools. And it's most definitely bad for building software, as it is limited by what the programmer of the build software thought should be supported. A really good example of that is ANT/NANT. make, on the other hand, doesn't limit one to what the author(s) thought should be supported. Need to run programs in order to get from A to B? No problem, put whatever you need in, and have it build you the desired target.
First off, I simply used some of PCRE for the syntax, as it's what I'm familiar with. \w could be easily replaced, and non-greedy matching is a relatively common extension.
As for when your record spans multiple lines, with recursive structures, the previous regex is for extracting simple atomic data from a json file, which is usally what you want in these cases anyway. If not, the json(1) utility can, I believe, extract arbitrary fields, and composes well with awk, grep, etc.
Yes, the json utility can process a JSON file into key:value pairs. Now ask yourself: if you end up with key:value pairs on stdout, why couldn't that have been the format in the first place? Why artificially impose not one, but two layers of complications (one as JSON, the other as the specialized json tool to process JSON)? Why not just keep it simple and go directly to a flat file ASCII format to begin with?
Well, it means not rolling your own parser. But that's not hard. The real advantage is when you actually ARE dealing with structured data, with nested objects. Most standard UNIX formats are bad at this, and sometimes you find it necessary.
Also, because JSON is so common, you get really good tooling for handling structured data by defult, instead of kinda-okay tooling for 50 different slightly-incompatable formats. 10 operations on 10 datastructures vs 100 operations on 1, and all that.
But for unstructured data, or for one-level key/value data, JSON is overkill. You can use DSV, like this:
