For those who don't know, Hendershott wrote Cakewalk, the most popular music-creation software on the PC. He knows his stuff. It's also an engaging article, esp for us old C/C++ hacks who get nervous around Lispy things.
I saw Greg give a nice talk at Racket Con this year. He was really humble and talked about the challenges of writing a Markdown parser. Didn't know he wrote Cakewalk! Cool!
Don't forget that Cakewalk includes CAL - the Cakewalk Application Language, which is a bit Lisp'y, I have to say, as well as being a bit C'ish, to boot. ;)
Confession: I used s-expressions only because even I could figure out how to parse that. I didn't appreciate Lisp in any meaningful sense.
Fast forward to a few years ago. I wanted to learn a lisp, finally. I gravitated toward Scheme. PLT Scheme seemed like the best choice. Today it's known as Racket.
I remember putting Cakewalk through its paces in the late 80's/early 90's on my old DOS machine in the studio .. it was really a great way to get a lot of music written, and I really appreciate your aesthetics for how you managed that application over time - the arrival of CAL was a really interesting move, and as things moved so fast in the pro audio world in the 90's, it always sort of bothered me that more and more tools were not written with scripting - for the user - in mind.
My friends and I wrote all sorts of CAL scripts to 'funkify' our Drum/Rythmn tracks - I wish I still had a copy of funkify.cal around, it'd be nice to see it again.
I moved on from Cakewalk around the DOS/Windows migration, as I've never been a heavy fan of Windows, nor a real user, so I got set up with all kinds of other tools after we left the DOS machines alone .. mostly hardware sequencers.
What do you think about one day doing a hardware sequencer, Greg? With the way things are going, it seems like you could return to the world of music-composition, albeit with a hardware bent, and make some serious progress in that department .. as we know the music industry/market is heavily cyclic, soon enough there will be a return to hardware as a reliable, dependent way of composing music. Any plans in that regard? Imagine: a hardware MIDI/Audio sequencer .. with onboard SCRIPTING! :)
i don't fear macros, but i am wary of them: when there are only a few macros, it seems as though replacing them with functions makes more sense, even if the syntax is slightly messier. when there are so many macros that you've essentially made an entirely new language, writing a REPL feels like a cleaner separation, even if it is more work.
i would be interested to see links to libraries, particularly clojure libraries, that make heavy but judicious use of macros. it totally makes sense that macros are useful because they capture part of this "code as data" idea that's so central to, esp., lisps. but having more examples of actual problems that they solve would be helpful.
They look the same as normal logging function calls, but entirely avoid the issue of "logging guards."
I'll point you at a good link, but basically if you have a lot of calls to logger.debug, logger.trace, etc, and they involve a lot of string conversions, you pay the penalty for those string conversions all the time even when you're logging at warn or higher. This leads many java developers to do the following:
if (LOGGER.isDebugEnabled())
{
LOGGER.debug("last node was: " + node);
}
... which makes logging significantly heavier visually.
With a macro you are essentially adding syntax, so you can bake the guards into your log statements.
Although, for what it's worth, Java programmers solved this quite some time ago by allowing crude lazy evaluation of the string. Logging calls actually look like:
LOGGER.debug("last node was: {}", node);
Where the parameters are a template string and a varargs of objects. The objects are stringified and substituted into the template only once the logging library determines that the message will actually be used.
In Scala, you just make the parameter to the call a lazy or call-by-name value, and then there's no need to write an explicit template, the language captures the expression which produces the string and defers its execution until it's needed.
Of course, there are many things macros can do which mechanisms like these can't. The choice in language design is between introducing macros, or introducing enough other mechanisms to make macros unnecessary. Introducing macros seems more parsimonious, but given the utter confusion that macros can be used to wreak, it's not a slam dunk.
And, of course, the Scala designers have introduced all these mechanisms and then added macros anyway, because they're crazy.
My favorite example would have to be CLOS the Common Lisp object system. Although it's a bit hairy in certain places, it essentially transforms a procedural/functional language into on of the most powerful object oriented languages around. Macros play a small, but crucial part of it, subverting the rules of lisp in order to add another dimension to the language.
Another great example might be Postmodern and CLSQL, both of which include SQL DSLs, much better than concatinating strings or muddling around with ORMs. CL's loop macro basically ads an algol like language to lisp(if you dislike loop, there exist lispier alternatives like iterate).
Macros take care of the case between writing a function and writing an interpreter, and with judicious use, they can help build much cleaner abstractions that give a qualitative improvement to a code base.
core.async [1] is basically the canonical example of a Clojure library that relies on macros to do its job, although surprisingly few macros actually need to be exposed in the public API. Most notable is the go macro, which performs deep-walking code transformation on its body, permitting clients to write async code in a straightforward, boilerplate-free way.
There's usually a heavy emphasis in the Clojure community on keeping macros as small as possible – the thinking is that any behavior that can be implemented with ordinary functions should be, and most macros should be lightweight wrappers over these functions. This seems to fit together well with what I've observed: functional programming idioms often circumvent some of the more common legacy uses of macros in practice, relegating macros to a still-essential but less prevalent role.
All of Racket's `#lang` family of languages are implemented with macros on top of core Racket. For example, Scribble, mentioned in another comment, is the documentation language used to author the article, and the Racket docs[1]). Or Slideshow, Racket's programmatic Powerpoint alternative.
Try to add a static type system to Racket or Clojure. Oh but you don't have the ability to change the runtime system. With macros you can do it. See Typed Racket[2] or Typed Clojure[3].
oh, yes, indeed, the ->> macro is slick. reminds me of the bind operator in haskell. i don't know why i haven't seen something like it in other lisps. but as you said, that's part of the language.
anyway, should clarify: i'm interested in libraries that define and then use lots of macros in order to solve a specific problem. i'd like to know what problem domains are particularly well suited to the sort of language extension that macros provide.
I don't know Clojure, but here's a couple of good macro examples in Scheme.
Scheme's SRFI-41[0] implements Streams, which are lazily evaluated lists. A few macros are used in order to introduce lazy evaluation to the language. The Streams API wouldn't be very nice at all if no macros were used.
How about something more basic like the `let` macro? It's something you can't live without.
Whenever you need to control the order of evaluation, macros are what you want. Don't be wary of them. They are wonderful.
I developed a fear of macros after realizing that they're not first-class, and you can't pass macros to functions as arguments for instance (instead you need to wrap them up in function, which doesn't work well with variadic forms).
An even more useful concept exists in J.Shutt's Kernel programming language[1] - that of first class operatives. Every entity becomes a first-class object, which can be an operative (like some special forms) or an applicative (a scheme function). operatives do not implicitly evaluate the operands, so you can easily create macro-like behavior by defining your own evaluation model. It's also trivial to convert between applicatives and operatives with wrap and unwrap (which are not special forms, they're defined in terms of vau and eval).
It turns out that fexprs, which is what Kernel provides, are widely considered a bad idea in the Lisp community, and were abandoned in favor of macros for a reason. The basic problem is that you can never tell whether your function call is really a function call or a macro application, until you run the program, and therefore you can't reason about your code at all. Incidentally, this also means that your compiler can't reason about your code, and thus optimization is basically impossible.
The fexprs provided by Kernel are not the same as the ones in traditional lisps which were abandoned - Shutt has developed a theory for reasoning about them in some ways (the vau-calculus, which is capable of defining the lambda calculus too). He describes pure and impure versions of the vau calculus in his dissertation.
Firstly, operatives are improved over the traditional ones by having lexical scoping - where the original lisps were dynamically scoped. The other main feature of operatives is that they gain access to the dynamic scope from where they were called - as this environment is passed implicitly to the operative when it is evaluated.
You're right that they do not allow us to reason inductively about code in the same way as the lambda calculus due to this, but it's a trade off for incredible flexibility. I think we can build alternative models to reason about them for purposes of optimisation and such. (Or alternatively, one can have operatives behave as compilers - in which you feed in type-tagged expressions, and your compiler-operatives can perform full type checking - then output an applicative which strips this type information). I see the possibilities as endless, because the model is extremely abstract.
Kernel choses not to enforce the static separation of operatives and applicatives for simplcity - instead, a convention based approach is used whereby operatives are prefixed with "$", (such as $vau, $lambda, $define!, $provide!). One can check them during runtime with the operative? and applicative? predicates too.
However, it would be perfectly possible to create a compiler which can know this difference, since the forms `($define! ($vau ...))` and `($define! ($lambda ...))` which distinguish them could be given special forms or special syntax in an alternative representation of this model, although you would not be able to tell whether an applicative uses some operatives internally - that's the idea of encapsulation - implementation information is meant to be hidden. (Even at the cost of performance, which often doesn't matter.)
Even if Kernel is not as widely practical as lambda-calculus based languages - Shutt's dissertation is definitely worth a read, and shouldn't be discarded because of a perception of bad fexprs from 40 years ago.
before this article i didn't want to touch scheme macros.
he does a great job by introducing macros using basic syntax transformers, every other tutorial failed for me by introducing define-syntax-rule before explaining how it worked.
I have gone from Unix shell script to Perl to PHP to C to C++ to Java to Python. Most of them are a lot of the same Algol-descendant syntax: if/else, for loops, while loops, variable assignment etc.
Racket Lisp is a little more of a leap. Even something like the behavior of a for loop is conceptually different. When most of the languages you have been bouncing around are more Algol-like, Lisp is a bigger leap into a different programming paradigm.
At least conceptually, for loops work very much like that in the other languages. You might be referring to Lisp in the general sense. If that's the case, remove the "Racket" parts in your comment, refer specifically to the Lisp family, and then the comment should be ok.
Racket has a documentation system called Scribble.
Which is actually one of the most interesting features of Racket -- that it lets you implement a `#lang scribble` which does not even use s-expression syntax, but you still have the full power of Racket if you need it.
The default `#lang racket` is a wonderful modern lisp, and you can use Racket as "only" that. But Racket is also a system for making languages.
