The largest reason for use of S exprs is making Meta programming easier. While certain things like arithmetic are more difficult with S-exprs, Macros and DSLs are far easier than with C like syntax. Since Lua is actually only a few steps from something like Guile, it takes very little code to translate a scheme like language to Lua without adding too much runtime overhead. The difficult parts of lisp to implement, namely closures, come for free.
With LuaJIT as good as it is, I would not be surprised if something like this would outperform guile.
As a guile fanboy, I doubt outperforming guile is very hard (even though it is fast enough for most uses).
LuaJIT is most certainly faster than guile, but it isn't a very good comparison. Guile is far from the fastest lisp (or even scheme, for that matter), whereas LuaJIT is probably the best JIT on the planet.
That's true. I don't want to criticize Guile or Lisp implementations in general, which have a number of features that Lua can't easily emulate (a numeric tower, true continuations, probably a whole lot more I'm not aware of), but I think Guile and Lua have very similar uses as embedded scripting languages (small, and with fast incremental compilation). Chances are if you using guile you may have considered Lua.
As for use-cases for Fnl, I would say it's more applicable for places where one would use Lua or Guile, rather than something like SBCL or any really fast, compiled Lisp.
Lua makes a pretty good dynamic language target, though, because of the many great implementations and small language spec.
I have often asked myself how LuaJIT can be so fast. It runs laps around every other JITed dynamic language, even fast ones like JS-v8. First of all: Mike Pall, but secondly: lua's semantics must lend itself very well to modern optimization techniques.
LuaJIT is pretty friggin great and I have embedded it a couple of times with great success :)
I do however preferred guile model for multithreading with one instance that an spawn multiple threads, and that can be used safely from multiple threads,but that is just amatter of taste. If you want to spin up multiple instances, guile isn't for you (although you could use multiple threads in a single instance).
So whatever floats your boat :) There was some work implementing lua on the guile VM, which would have been cool.
Mike Pall has written about it in some.places online (googling is a good idea), an he has also written quite a lot about LuaJIT and JITs in general on Reddit:
LtU has an old, but interesting thread. Can’t tell how much details changed from that time, but since 1Gb limit was removed from LJ recently, it is my The Only Dynamic Language to Consider.
Mike Pall indulges in some (well deserved) gloating far down in the thread:
LuaJIT also does: constant folding, constant propagation, copy propagation, algebraic simplifications, reassociation, common-subexpression elimination, alias analysis, load-forwarding, store-forwarding, dead-store elimination, store sinking, scalar replacement of aggregates, scalar-evolution analysis, narrowing, specialization, loop inversion, dead-code elimination, reverse-linear-scan register allocation with a blended cost-model, register hinting, register renaming, memory operand fusion.
Due to the nature of a trace compiler, it implicitly performs partial and interprocedural variants of all of them. And many traditional optimizations, like straightening or unreachable code elimination are unnecessary.
All of that in 120KB for the VM and 80KB for the JIT compiler. And I didn't need 15 years and a billion dollar budget for that, either.
I'm planning to add value-range propagation, array-bounds-check elimination, escape analysis, allocation sinking, if conversion, hyperblock scheduling and auto-vectorization. Anything I forgot? I'll see what I can do. :-)
Arithmetic isn't more difficult with S-expressions, and in fact complicated arithmetic expressions are easier to format across multiple lines to make their organizations recognizeable. It also naturally supports N-ary operations. For instance (* a b c d) could be a matrix multiplication which chooses the optimal order for the decimation, taking into account all arguments simultaneously.
I like lisp and prefix notation in general, but I definitely still prefer reading large mathematical expressions in infix. Its really just a matter of preference.
Most maths applications written in Lisp use infix/mixfix syntax for the languages they implement, too: Macsyma, Derive, Reduce, Axiom, ...
Implementation of those slightly diverges. Macsyma internally is largely written in Lisp syntax, where for example Reduce is written in RLisp (which is a Lisp written on top of Portable Standard Lisp), which does not use s-expression syntax.
I don’t have much cause to write Lua, but I can very much support lispy syntax.
When we all started coding, we probably used a text editor and thought in terms of editing characters: backspacing, typing, maybe mousing around for bigger changes.
Then at some point we graduated to a programmers editor where we got comfortable cutting, pulling, yanking, killing larger units of text: words, lines, paragraphs.
With tools like paredit, you can take another evolutionary step to edit the structures of your code. I find this very productive. It feels a little like those code block programming languages for kids.
If for no other reason than to enable slurping and barfing code structures in my text editor, this is a win. May all languages someday have a lisp syntax.
I'm with you almost all the way.... however, I think what we should learn here is that every language deserves tools as powerful as paredit--that is, ast-aware editing--not that all languages should convert their ast to s-expressions.
Imagine not being able to save syntactically incorrect code, or being able to have semantic diffs between domain objects (e.g. functions, classes, strings, types, apis) rather than thinking in terms of characters and lines, or being able to query a code base, like "find me all places where I assign to a variable named 'status' with type under 'error' and list me the names of the functions whose frame the variable belongs to". Paredit is simply where text and ast manipulation converge to the closest point.
S-expressions are not directly an AST: they are a form easily matched to an AST (which could be the Lisp at hands, or some language for which there is some translator/parser/interpreter around, or some language for there is no code in the system to (fully) handle it). This makes the use of s-expressions simple, they can be handled by the development tools without them always having to know what language they represent.
If your development tool knows how to handle a particular language, then it can handle the AST for that language, but not any other. You could instruct the tool about the other languages, but that requires ways to let the tool know; in the end this is similar to the static versus dynamic typing discussion: static typing doesn't allow any potentially unsafe data, and a tool working with an AST directly wouldn't allow any potentially invalid programs, which in principle (or theory) is nice but difficult to achieve practically, it presents overheads for the tool implementer and for the user probably alike. And it probably takes away a lot of interactivity: just like reloading modules in a statically typed language (like ML or Haskell) is not as direct as being able to modify values and functions in a running system and having them be called directly.
So I think S-expressions are a very pragmatic concept that is going to be difficult to replace without the replacement costing more than it benefits.
S-expressions are a syntax for data. It's more related to what a tokenizer does, but with nested lists of tokens - not a flat token stream. It's only loosely related to an AST, since the s-expression represents some form of nesting - but not explicitly what kinds.
S-expressions know nothing about the syntax of the program language Lisp (what is the syntax for the LET operator?). If we have a Lisp expression, an AST for it would need to represent specific language constructs - like knowing that LET is a built-in operator and has parts like a list of bindings (where each binding is a symbol or a list of symbol + value form, optionally declarations (with their specific syntax) and a list of body forms. The s-expression does not represent that: in
(let ((let 'let))
(list let 'let))
the s-expression does not represent that LET is a built-in operator, a variable name, a symbol, a variable name and a symbol. All we see is five LET symbols and no additional information.
Writing Lisp code using s-expressions allows us to use data manipulation functions both on text and on internal Lisp data: reverse, transpose, move, move in, move out, ...
We can then also enrich the editor (or other tools) to understand the syntax of Lisp. Example: know what the structure of a LET form is and then allow for example to manipulate the binding list, knowing where it is and what a variable and what a value form is.
One thing to consider is that s-expressions make implementing paredit easy. It takes an enormous amount of effort (e.g., libclang in C++) to get close to that without s-expressions. It's as if you think of programming languages as "fixed", and we only get to choose which kinds of tools we develop, but another way of doing it would be to design programming languages in ways that specifically make developing such tools easier. Then s-expressions could well be the go-to solution for that.
Red hints annoy me honestly, when I meet these in real editors. I was thinking of something like grammar walker that completes input to shortest finished constructs. While not semantically correct, but syntactically like yes.
pr
pr;
printf(
printf(<?expr?>);
Or to not edit code as text at all, but as a tree. Like apple plist editor or windows registry editor, but with convenient shortcuts to add constructs. Xcode can sort of simulate this with templates and placeholders:
As a fallback. I like LISPs as much as the next guy/gal, but editing the AST by default is too low-level. The importance of paredit is a language smell in my opinion, not something to aspire to.
Can you explain in which direction something better should go? The AST is very much at a higher level than how the vast majority of programming is done, which is by manipulating the text that later gets parsed into an AST. I'm having trouble envisioning what editing a higher level than the AST would look like, without jumping straight to visual programming, which I find very inefficient in every incarnation I've seen. (And we could debate how far above an AST such systems actually are.)
It depends on the AST, but most languages' ASTs are quite unwieldy to work with directly. They get marked up with all kinds of semantic information useful for a compiler or interpreter quite early in the parsing process, and are often more explicit and regularized than the syntax is. For example, you can dump a Python AST with the 'ast' module, but it's not pretty. The statement "x=5" balloons into:
I think a comment up this thread (https://news.ycombinator.com/item?id=16386702) is right that Lisp manages to make this work because the s-expression structure of Lisp source code isn't really an AST, though it has some relationship to one.
That Python doesn't look like abstract syntax; it's an intermediate form full of additional semantic objects geared toward further translation or interpretation.
A Lisp compiler might build up something like that, from a fully macro-expanded body of code. That's not usually accessible to programs.
Like the fact that n is converted to a Name() node, where it has a ctx property indicating Store() has nothing to do with syntax; it is the result of an analysis arising from how x is being used semantically. The user didn't specify any such attributes on the x just that x is assigned and that is already recorded by something like Assign(x, 5).
That seems backwards to me - AST-aware editing is basically embedding more of the compiler into the editor. It seems more valuable to me to be writing in a language that can express higher level concepts with lesser boilerplate, which would eliminate the need (and make far more difficult) for AST aware editing.
You might be surprised how little a Lisp dialect brings to the table specifically for developing the text editor features geared toward editing specifically Lisp.
The data structure representing Lisp data (i.e. code) is stripped of relevant text editing attributes, like how it is divided into lines and using what indentation, and, oh those semicolon-delimited comments.
Text editors also have to let the user deal with bad syntax.
So it's not just a matter of, "oh, we're written in Lisp, so just hand the buffer to the Lisp reader, do something and spit it out again".
I'm not sure what that is even supposed to mean. Having high quality tooling is completely independent from having a programming language with less boilerplate code. You can have both at the same time. So why are you advocating against it?
Take Java for instance.
In Eclipse the a lot of AST-aware editing features are hidden in the "quickfixes" which you can trigger with CTRL+1.
For example you want to use an ArrayList but you haven't imported it yet. Use quickfixes: Bang the import was created with just two keypresses.
In a raw text editor you have to remember the full package string or google it but when you want to look up what a "XYZ" it's rather easy to find it.
The java import system is verbose but it makes it obvious what you're importing.
one class = one file
Want to know what X is? Just open X.java.
Compare it to something like C/C++ where you include a header that contains multiple definitions. I honestly cannot use C++ without an IDE because of this. At the bare minimum my editor needs to be able to jump from a function call to the function definition or variable declaration to the struct definition.
In this case a feature that reduces boilerplate actually made the language harder to use without AST aware navigation.
Then there are quality of life things like extracting parts of a big function into seperate smaller functions. Just select the code you want and the IDE is going to create a function definition based on it and replace your selection with a call to that function.
I don't see how you could avoid this by making a more expressive language.
Does your fancy programming language automatically organise your code into functions or data types and all you need to do is crank out code without a care? No, you still to need think about that yourself, but the IDE can automatically move the code around for you.
Also let's talk about my favourite feature that I would never use without the assistance of an IDE: typeinference.
value blah = foo.bar([1,2,3])
What the hell is a blah? With an IDE I could just hover over blah and instantly know what the function returns. Without that feature I have to go to the definition of bar which is still easier with an IDE. After all if foo is also defined via type inference I have to lookup several function definitions and that's going to take some time. On top of that a tool like grep is imprecise and will give me hundreds of call sites but only one definition.
You can't fix everything by making a programming language more expressive and in my experience the dumber a programming language is, the easier it is to use with a dumb text editor.
This could be interesting to integrate support for racket to allow it to interact with Torch. You would just create a small language to compile to lua and then have that run in Torch.
This is really interesting! I've been looking for a Lua-like Lisp implementation that would be as easy and clean to integrate, something that would compile with clang/gcc/msvc easily and allow spinning up multiple interpreter instances without having any global state. Only TinyScheme comes quite close I think.
The largest reason for use of S exprs is making Meta programming easier. While certain things like arithmetic are more difficult with S-exprs, Macros and DSLs are far easier than with C like syntax. Since Lua is actually only a few steps from something like Guile, it takes very little code to translate a scheme like language to Lua without adding too much runtime overhead. The difficult parts of lisp to implement, namely closures, come for free.
With LuaJIT as good as it is, I would not be surprised if something like this would outperform guile.