Lua defines functions with the `function` keywords, and as most other C-style languages uses parenthesis to wrap the function parameters, not brackets.
It does use `{}` for table literals but requires the `=` and `,` separators, where fennel (and clojure) uses no internal separator at all: a map literal is a pair of braces around a plist (https://www.gnu.org/software/emacs/manual/html_node/elisp/Pr....
Historically, lisp have used something around "define" to define functions (e.g. define, defun), parenthesis as essentially the only grouping, and would use macros or special-forms rather than complex literals.
Clojure is the first lisp I know of (so I may well have missed a predecessor which inspired it) which largely diverged from that.
[0] clojure allows commas but ignores them entirely, I don't know whether fennel allows them as well
If we think Clojure is a 'diverging Lisp', then there are other diverging Lisps with different syntax like MDL, Logo, Dylan, even JavaScript. Some Scheme code uses angular brackets.
Historically core Lisp has left other grouping character or syntax to sublanguages or new data types. There are many examples for that. This is usually done via reader macros, which open up the s-expression reader to the programmer.
This is a Joshua rule:
(defrule grandfather-determiner (:backward)
;; there are 3 ways to find your grandfather
IF [or [is-genetic-grandfather-of ?gramps ?kid]
[is-grandfather-thru-step-parent ?gramps ?kid]
[is-step-grandfather-of ?gramps ?kid]]
THEN [is-grandfather-of ?gramps ?kid])
Many internal or external domain specific languages for Lisp use special syntax and defining forms, since their authors either believed that it's easier to use in general or that it was necessary for 'end users' of their systems. Early examples were maths systems like Macsyma, which were programmable, but addressed mathematicians as audience. Logo, which was Lisp for kids. One of the first theorem provers had the first ML as the programming language interface for its users.
Historically Lisp 2 was supposed to get rid of simple s-expressions on the surface. There was a different syntax defined for it.
I was mostly talking about diverging in the way Fennel had.
> Historically core Lisp has left other grouping character or syntax to sublanguages or new data types.
Sure, but Clojure uses other grouping characters for the main dialect itself, and Fennel uses very similar deviations from the "Lisp baseline", hence my wondering about the inspiration.
The difference for Clojure would largely be that other delimiters are sugar for the various data types, so you make maps with {}, vectors with [], sets with #{}, etc.
it then mixes syntax and data type issues. For example arguments are then vectors, probably to have [] as syntax. Though in Lisp one would keep argument lists: there is no technical need to make them vectors.
Scott Bell wrote a lot of that. I was fascinated by the self-hosting technique. The whole repo is so small that you can almost dismiss it as some kind of academic toy, if you don't realize it's a full-featured production-grade lisp.
(Not to take anything away from Fennel! The more lisp, the better.)
Thanks for posting this. I took a look at Fennel a few months ago and although they list 2 web frameworks on the front page, there's little documentation (that may have changed since then) about how to interface with other Lua projects. I kept running into things I couldn't figure out when working with the existing Lua ecosystem and projects.
I’ve been using it quite successfully in a large OpenResty project.
While you’re right that there aren’t any docs on working with any of the 3rd party projects listed in the homepage - I found that it was pretty straight forward to use as a drop in Lua replacement by examining the compiled Lua from the online REPL. I was surprised how clean the Lua output is. There is really no magic here. I believe one of the project goals is to stay as close to Lua semantics as possible. Basically just lisp flavored Lua.
In my project I just use the compiled .fnl files as I would any of other .lua file.
Is something specific you’re getting stuck with? I’d be happy to try to point you in the right direction.
It’s really nice! I encourage you to give it another try.
Lua is way more common for game scripting than Python. One of the few major games which used Python I can think of was Civ IV, and Civ V switched to Lua.
Lua is smaller, easier to embed interface / interact with (e.g. it's way less ceremony to expose a native function to Lua than to Python), and tends to be faster, especially when jumping back and forth between the engine and the scripting. Embedding / scripting (within a larger program) is the original use-case of Lua, not so Python.
I believe it's also much easier to secure / sandbox (remove bits you don't want script writers to have access to) as well, the stdlib is smaller and I think it has less interactions between modules.
Python embedding / scripting tends to be more common for software where the securing / restriction aspect is smaller but flexibility & larger embeds are necessary e.g. 3D, CAD and other "production pipeline" software tends to be scripted with Python.
> I believe [Lua is] also much easier to secure / sandbox [than Python]
Yes. C code that embeds the Lua VM has total control over which functions are exposed to the Lua code. It's possible to create a VM that, for example, has no access to the filesystem.
AFAIK it's not possible to do that if you embed Python. Dangerous functions like `open` are always available and sandboxing facilities have to try and prevent untrusted code from getting a reference to them. Unfortunately, there are numerous ways to work around these restrictions and obtain access to dangerous functions, e.g. via `__builtins__`.
Another reason running untrusted Lua code is considered fairly safe is that the VM is well-engineered and has a history of very few bugs [0]. However, the latest 5.4.0 release seems to have more bugs than older releases.
It might still possible for untrusted Lua code to use 100% CPU and hang a program, or to read data from the embedding process using a Spectre-style attack (although even that's unlikely because the Lua VM is an interpreter rather than a JIT compiler). However, it's quite possible to secure an embedded Lua VM to prevent it from doing things like accessing arbitrary files.
Sandboxing Python used to be and still is hairy and very hard (it requires intricate knowledge of CPython, because you are going to inspect the AST for unwanted things and do some other relatively low-level stuff) and Python is also a pretty big dependency (~5-10 MB). I think many recent-ish additions to Python may make sandboxing easier (e.g. audit hooks, subinterpreters), but it's still hard and messy.
I've made a suite of digital musical instruments for Android phones in Lua. Now I'm using same language to build some VR prototypes.
Here are some benefits of Lua over Python. It's really powerful language (anonymous functions, closures, tail-call optimization, full lexical scoping, coroutines...) that supports many paradigms, and yet it is tiny and elegant. There are few surprises and everything is explicit.
Runtime is easily embeddable, which means it will be more readily available on exotic platforms. The interpreter is much faster than Python. C code is effortless to call into with FFI and resulting code looks just like normal Lua.
Not everything is roses. I prefer readability and less verbosity of Python. The 1-indexing needs some getting used to. Batteries are not included (to be more portable), so there are dozens of implementations of basic things like serialization, OO classes and table copying. Most online material (wiki) was written very long ago and it's full of language proposals that never succeeded. Some valuable resources can be found in documentation of hosting frameworks (Defold, Solar2D, Roblox, ComputerCraft, WoW).
I wrote a console-based email-client, using Lua for all configuration and scripting needs.
Using Lua is generally done because it is easy to embed inside a host-application. While it is possible to embed Python, Perl, or other languages, they're relatively heavyweight and not so commonly used in that case.
In my own application I always felt annoyed that mutt didn't have "real" scripting. Just an ad-hoc configuration that made lots of things possible, but neglected some basics (such as loops and similar).
Configuring a mail-client in Lua was a nice exercise, but eventually I moved on to pay for gsuite rather than self-hosting a mailserver of my own so it became a "done" project. Definitely a useful learning experience though, experimenting with user-interface, embedded scripting, and going through lots of learning relating to MIME-handling & etc.
Your project sounds cool - is it possible to process email with Lua scripts, so that I could use a mailbox as a business process queue that is also user-friendly, while having a suite of Lua apps processing the mailboxes? If so, care to share details of your project if its open/available?
It wouldn't be impossible to iterate over a series of mails, and get the message-bodies, but it would be a bit fiddly and annoying as the program was more designed as an email-client.
Lua is a tiny, relatively simple language with a simple data model built on a small runtime that's very fast for an interpreter and easy to embed into C/C++ applications (once you dealt with the stack, mentally or otherwise). Startup time is ~nil.
Python is a large, complex language with an exceedingly complicated data model built on a runtime where the core interpreter is about ~4 MB, which is also a pretty slow interpreter (mostly due to the exceedingly complicated data model). Python is certainly not hard to embed on an API level, but somewhat annoying to package up for a build. Python is very hard to sandbox, whereas Lua is basically sandboxed by default. Startup time is on the order of ~100-200 ms.
If I use Python instead of Lua for monte carlo tree search, my machine will page out, and even before it pages out, my training pipeline will be running dozens of times slower than it was with Lua.
If I use Python instead of Lua to make a game, I will not be able to update my game state and render all my images at 60hz.
If I use Python instead of Lua to write programs that do many tasks at the same time, I will have to write "await" a thousand times instead of 0 times.
the easiest way to put it is that you embed Lua into your app (say scripts that are loaded at startup or even as a command line) while it’s more natural to embed your external functionality into Python (e.e. numpy).
It’s simpler to connect lua to your c/c++ datastructures than it is to connect Python, and it’s more complicated to write little things in Python.
Sometimes it’s easier to grab a pencil and scribble a little note on a piece of paper. Sometimes it’s a lot more useful to type a note into your phone or computer than to deal with paper.
I have embedded lua in Spring Boot webapps where I required an external rules files which could be easy to read for business folks.
Example: Specialised handling for different customer types and you don't want to manage all the rules in the RDBMS but would rather run a separate script for each customer type and have this script in a plain text format and version managed using Git.
Sounds very interesting! Any chance this is open source? If not, can you give a bit of additional information on how you did this (including project structure, some insight in the lua code, etc)?
I use it at work to process SIP messages on behalf of a cellular carrier. The actual parser is written using LPEG, a parser expression grammar for Lua. Because Lua is meant to be embedded, it only has the functionality required to do the job and no more.
This and a recent feature addition to neovim (passing lua closures to vimscript functions [1]) should make a good substitute for elisp to configure neovim.
I don't see it mentioned but there is a neovim plugin called conjure [1] that is written in fennel. This is a general purpose REPL client that works for Clojure (I use it daily for Clojure and ClojureScript) but also for fennel itself. So you can have a really nice neovim plugin dev experience using this tool. Conjure works with fennel out of the box, but if you want to go deeper you may be interested in another plugin called aniseed [2] which provides interesting facilities for the fennel language and that is used internally by conjure.
Disclaimer: I am not the author of those tools, just a random Clojure dev.
I haven't spent any time with Fennel, but it seems more promising to me.
I couldn't get comfortable with Hy. It drops so many lispy things, such as let statements and persistent data structures, in order to stay close to the Python substrate. (You can get both from libraries, but at the cost of easy interop with Python packages, which kind of defeats the whole point of being on the Python platform in the first place.) It ultimately came out feeling less like I was working with a lisp, and more like I was working with an s-expression syntax for Python.
OTOH, if you think that Python, while otherwise great, is really suffering for lack of defmacro, Hy absolutely succeeds in getting you Python with defmacro.
There is another small lisp that targets both Lua and Javascript called "lumen"[1]. Small enough that you can easily tinker with it. I added clojure style {} literals to the reader and paired it with hyperapp to write a toy web application.
I really wish Lua were popular for small-scale utility scripting instead of Python and JS. Because Lua is way snappier: even with language translation, plenty of Fennel scripts will run under Python's startup time. Likewise, Lua seems like a great fit for small mobile apps and automation (dunno about threading though, but I seem to have read that Lua can do threads with a library).
My primary woe with Lua is the absence of ‘null’. You can't preserve structure of deserialized tables if there are nil values in them, so you can't really build API middleware and such, without extra effort for that (so you'd also not be able to use functional libraries like Penlight in your middleware).
Lua will never be more popular without a proper standard library.
Leaving everything up to the user so that you have to manually implement (or hunt the web for) even the most basic features is mega annoying. How many millions of people need to rewrite string.startswith?
The whole "sequence iteration stops at the first internal hole and therefore the # operator doesn't do what you think it should" gotcha is extremely user hostile.
Metatables are cool and all, but holy crap are they confusing for Lua newbies. Having an integrated class system would go a long way.
Agreed. One of the reasons that Python grew popular so fast was its "batteries included" approach. When a developer needs to do some common thing, it's really helpful to have one obvious way to do it that is highly reliable and well integrated with the rest of the system.
Counterpoint: Python format strings. (I think there’s four or five different built-in ways to do those now.) Python’s “batteries included” may have helped it to win vital audience early on, but it’s also locked in a ton of deadweight that’s seriously slowed down its own evolution since, while bloating its feature complexity to such a level I could not recommend Python as a good language for beginners now.
Which suggests there’s a more general meta-problem waiting to be solved here: how to [automatically] mass-migrate code written against interface A when a superior successor B comes along. I have many gripes about Apple’s #SwiftLang, but seriously appreciate that this is one requirement they’ve given some thought to. Having seen the decade-long logistical horror of Python’s relatively minor 2-to-3 upgrade, Xcode’s ability to auto-update projects written for one Swift version to another has let them evolve the language with minimal pain.
And while I’m not a fan of Lisp itself (a decent first attempt that should’ve been vastly bettered by now), I think its metaprogramming model an obvious foundation for such an automated migration system. Thus, rather than bake in all the libraries, keep the libraries external and build in a standard mechanism for migrating codebases written for one onto another. In essence, automated refactoring by “recipes”, where these recipes are as common and easy-to-use as the libraries themselves.
..
Of course, this still leaves the other (larger) problem of how to sort out current trash houses like PyPI, NPM, LuaRocks, et al, where contributors’ enthusiasm is not necessarily matched by their knowledge, competence, and ability to savagely critique their own work before inflicting it on the world (you know, the “Science” bit in “Computer Science”). Preferably without destroying that enthusiasm in the process, since there’s no point having a wonderfully polished platform without bums on seats as well—a perennial Lisp failing, it should be noted.
This is not merely idle speculation on my part. Many years ago I cut my teeth on AppleScript, which makes the stdlib-starved Lua look like fat Python 2, and tried to bootstrap a library ecosystem for it more than once. Good practice. Zero success†.
And now I’m trying to devise a “stealth Lisp” successor to them all (https://github.com/hhas/iris-script), I realise cracking the greater library problem is what will make the next generation of languages. A platform is only as robust and dependable as its weakest component, and for as long as a buggy or withdrawn left-pad function can tip the whole stack over then it’s not robust at all.
But this is as much a sociological challenge as a technical one, and not one I’ve got answers to yet.
--
† After which I realized it’d be far easier just to port the useful bits of AppleScript to Python instead. Which almost worked too… but that’s another story.
It would still be lightweight and embeddable with common sense pure Lua functionality like classes and major utility methods like string.startswith/string.endswith/string.trim/string.split/table.deepcopy. You don't have to integrate a multiprocessing web server for different operating systems.
Instead you're left with bullshit community essays like http://lua-users.org/wiki/SplitJoin which shows pages and pages of different user attempts for one extremely common function, many of which explicitly do not work!
Lua has its strengths, but faffing about refusing to just standardize the right way to do things is super user-hostile.
Lua, by design, favors "mechanisms not policy"[1]. It's not a failing it meant to empower users. There are plenty of mature utility libraries that serve for batteries if you like that.
Freedom to implement bespoke solutions is great if what you want to spend your time on is implementing bespoke solutions. Freedom from needing to implement bespoke solutions is better for everyone.
Even non-programmers can understand what common simple operations are for and when they might be useful. Saying that one needs to know how to implement or find them as well, especially when a bunch of extremely learned participants in lua-users can repeatedly fail to implement working versions, is literally the opposite of empowering.
The examples you're listing, I don't use those very often at all. If I did it's much easier to just grab a lib, then to carry all my tools with me for every script. Even the libraries that are included can be cast aside if you want a slim build. Not every language does or should strive to cater to those looking for quick hacking or rapid prototyping. I like that Lua is slim and loads fast. I understand your perspective though.
1. Luarocks.
2. I regularly carry these little enumerables around with me and use it so frequently I found it quaint to be reminded of the 'hole problem':
--- get all packed entries, until gap
-- func on the provided list
-- @param list table to be inspected
-- @param func do func what you func want
table.each = function(list, func)
for i,v in ipairs(list) do
func(v, i)
end
end
--- get all entries, packed or not, until completion
-- func on the provided list, completely
-- @param list table to be inspected
-- @param func do func what you func want
table.all = function(list, func)
if (list ~= nil) then
for i,v in pairs(list) do
func(v, i)
end
end
end
Disclaimer: I love Lua and would us it for everything if I could.
Do you also carry a table deepcopy with you? What about string.trim?
And LuaRocks is a trash fire that doesn't integrate well with Lua's primary role as an embedded plugin engine and leaves you to waste your time in endless discovery of a slew of unmaintained packages that all do the same thing in different ways and with different bugs. Want to collaborate? I hope everyone is familiar with "classy" and not "oops" or "objectlua" or "Luaoop" or "lobject" or "classyng" or "klesi" or "halo" or "lua-c3class" or "pool" or "middleclass" or "Sunclass" or any other module that implements one of the half dozen official recommendations for faking classes with metatables or closures. It also becomes a huge "left-pad" risk.
Table deepcopy: table.unpack() does pretty much everything I've ever needed it to do, but for those tricker tables, there's always https://gist.github.com/tylerneylon/81333721109155b2d244 - and I freely admit that yes, the link to that gist is in my "Lua dev notes" bookmarks file, which I refer to often.
Hmmm ... string.trim: umm, not really something I find I need, a lot of times.. but as usual, Lua-users provides some incite into why, exactly, its not always as simple as one might think. (http://lua-users.org/wiki/StringTrim)
As for your brilliant discourse on Luarocks, ok. Not gonna get any resistance from me on that. I know the pain you describe. But I have had success with using it to copy environments on other machines 99% of the time .. which is not what I could say about the hell that python has put me through, at times. But I guess that's par for the course.
> as usual, Lua-users provides some incite into...
(insight)
It makes me sad that Lua's refusal to standardize good ways to do things led to the language's primary resource being a community message board filled with back-and-forth postcard essays saying "I tried this and it didn't work. Then I tried this and it didn't work." followed by someone else responding "Have you tried this?" and then yet another person saying "No, that doesn't work either." It's absolutely awful, and I hate myself every time I land on the Lua-users site.
I linked to it in another comment, but http://lua-users.org/wiki/SplitJoin is a perfect example. The document starts with "split and join are two common string operators". Note the word "common"! Then it promptly throws up its hands at declaring a reasonable interface and proceeds to plow ahead for literally pages of failed attempts that don't work right!
Even your link to StringTrim, which at least doesn't dick around with a ream of implementations that don't even work, provides _15_ different obscure as hell implementations, many of which only differ meaningfully in performance on really stupid cases, and the clearly superior one is in C, not Lua, so you can't even use it! But Lua itself could have used it and saved everyone the headache!
People shouldn't have to deal with this mess for "common operations".
As the one who wrote the C version (which I haven't had a need to use once I wrote it), why can't it be used? Not allowed to use modules written in C? Or lack a C compiler?
Also, please define "common operations" for me. I suspect that this is similar to the problem Microsoft has with Word---the features you find "useful" and the features that someone else find "useful" may be quite small, thus the large number of features. Not once have I reached for a "split" function, nor a join function (besides, that's the builtin operator "..", unless I'm missing some subtle functionality that "join" has).
And isn't the saying in the Python community "the standard library is where modules go to die"?
It's not Lua. This is not a thread about things missing from C. It's a thread about things missing from Lua. Suggesting to just use an entirely different language is bizarrely mishandling the issue.
I feel like the question is humorously akin to asking why a plain text reader can't portray a jpeg. It can, of course, if you write a description of the jpeg in a text file. Naturally, I should have been more clear.
> Not allowed to use modules written in C? Or lack a C compiler?
Often. The moment you start writing things like plugins for lay people, you stop being able to say "you should just grab a compiler".
> And isn't the saying in the Python community "the standard library is where modules go to die"?
People say dumb things all the time. We shouldn't begrudge them.
.
[edit] HN doesn't want to let me reply to your reply. I guess it doesn't like conversations. :(
> But doesn't Python also support modules written in C?
Whether a language _allows_ you to implement libraries in other languages is orthogonal to whether it should have standard implementations of basic things that are both extremely common and also often nontrivial to implement correctly and performantly for the person who needs the functionality right now.
> Or is it because the standard Python library is so large that it's not needed as often?
The Python standard library being so complete is probably the biggest reason why people love Python. Also, Python libraries get written in C exclusively for performance in cases where performance is paramount because Python is rather slow and frivolous with RAM. It's only done for things that are computationally intense (numpy, pytorch, etc). If Python were as fast as LuaJIT, nobody would write any Python libraries in C (excluding thin interfaces).
I think that's a big part of the problem - the perception of Lua as a language, and not as an execution/runtime environment that happens to have a scripting language at its core.
I get your point thought - it is frustrating to have to find these things to glom into a Lua project - but on the other hand, this is a language you can embed in anything, so tacking a standard library - which may not be useful in the "anything" application - is counterproductive to the goals of Lua, which is to be a simple and easy language to embed into anything.
But doesn't Python also support modules written in C? Does Python get a pass for that? Or is it because the standard Python library is so large that it's not needed as often?
Lua doesn't distinguish between, "the key is not in the table," and, "there is a value for that key, and the value is nil." Setting the value for the key to nil simply erases it from the table.
You can just create a sentinel value and use that. eg. `local mySentinel = {}`. Then insert that value when that's the behavior you want. `nil` on its own rarely makes for a good semantic understanding of what value is actually in the table anyways.
Also, in Lua 5.4, the behavior is different -- you need to do `t[key] = undef` (which is the only context you can use `undef` in). And `nil` can actually be a value in a table. Not saying what's better or worse, but that the behavior in 5.4 is different.
Do libraries like Penlight know about your sentinels or ‘undef’? Maybe in ten years they will use undef, or maybe you whipped up your own functional library and keep it optimized and full-featured. But now we have what we have.
Besides, please tell: if you receive an empty object as a value in JSON, how do you preserve it in your ‘sentinel’ approach? Gotta need another special value for that.
For the JSON case, it doesn't make sense to use an empty object as a sentinel, for sure. JSON parsers usually have their own sentinel for `null`. Lua has a lot of issues, but `nil` being not-present hasn't been an issue that has come up a lot in practice. If you have any specific cases where that has been an issue, would be interesting to hear about them.
As I wrote in the first comment, the use-case is e.g. API integration, which is what ‘scripting’ often is, these days. More widely, any transformations or filtering or helper code where you accept structures and pass them on after mild fiddling—again, quite typical use for Python or JS if you ever pipe data between scripts in the terminal or over the network. You don't put your data schema in each of those scripts and maintain them on every change at the source. Imagine implementing `jq` in Lua with all those ‘sentinels’ throughout the whole function library.
The issue is that interoperability is crippled. The fact that you haven't so far run into these use-cases doesn't help with the problem that an entire, very common, class of functionality can't be done without tracking those nulls at each and every step with some crutches that bring their own troubles.
Yeah I actually don't think Lua is great for those things, tbh. Lua is not a good tool for manipulating JSON--JavaScript is probably sensible for that (it is JavaScript's object format, after all). And yeah I don't think it's good to use a bunch of external libraries in Lua, since as you say there's not a common ground for how data is organized. The libraries I rely on are are more native integration things like lpeg, luasocket, luasec, love2d, ... They've all been quite good. And yeah you keep saying 'sentinels' in quotes but honestly it's worked fine when using lua-cjson. There's always an impedance mismatch between JavaScript's object format and Lua's though. Lua also allows non-string keys which are great in Lua but not a thing in JavaScript, and in practice I've found the array/object-table mismatch more problematic than `nil`/`null` and so on.
I've mostly applied Lua for embedding in game engines, and to that end there's nothing that comes close, short of rolling your own VM that's exactly built the way you want. The main way to use it I think is to embed it somewhere and build up the set of utilities / libraries that you need to script over. But yeah it's not great for API glue code bc. APIs tend to bias toward JavaScript / JSON and other ways of doing things.
Well, my lament in the first place is that Lua could be a good and fast general-purpose scripting language. Instead, I can have ‘good’ but then I'll have to sit around twiddling thumbs while Python or JS start up. Especially Fennel runs circles around ClojureScript in terms of toolchain and build-cycle lightness.
JSON is pretty non-specific to JavaScript, which is probably a part of its popularity. And it's not the only serialization format. Null works about the same way in the majority of popular languages—so you're bound to need it for interoperability if you go outside C.
By libraries, I mean Lua libs like Penlight. Once you touch functional programming and function composition, you want structures to be preserved as they are, unless explicitly mangled.
> And yeah you keep saying 'sentinels' in quotes
It's because I haven't seen that notion before, at least under this name.
Have you tried lua-cjson, out of curiosity? They use a cjson.null (or something like that) value for nulls in JSON. But yeah agree with what you've said in general, if I had to do some API JSON manipulation Lua isn't what I'd jump to, even though I'm pretty familiar with it. Do Python or JS actually take that much longer to start up for a script? Node starts up pretty quickly on my system.
Now that I think of it, cjson.null might be workable even with data-transformation libraries, since it's some Lua meta-voodoo-value that can't come in the input from other systems. But then, of course, there's the problem that other data-exchange libs invent their own voodoo-values for null, like ngx.null. Precisely because they need it for actual integration use-cases.
As for the performance, I happen to use an underpowered machine, which is how I discovered the drastic difference with Lua. However, performance still matters beyond just ‘pretty quickly’ if you consider desktop software, especially productivity software—which has to be snappy. Scripting your productivity software is a textbook case for dynamic languages, and almost the exact niche for Lua. Even on my machine, there was a moment when I had to make sure that Alfred actually called my Lua script—because the results popped up immediately (despite going through the command-line interpreter, not Luajit). Meanwhile, people are making productivity software with Electron or Python, which to me verges on ridiculous.
Yeah, I definitely care about performance more in that tradeoff. In my experience when performance matters, these details about using such a null representation and what different representations lend themselves also tends to matter. vs. gluing libraries to libraries willy-nilly. At that point this null issue is far from the main thing that comes up bc. you care about data layout enough that you have control over that representation.
One example I'm running into in practice these days is--I'm using chipmunk2d game physics engine in a library. It does heap allocations, while all the component data for other stuff in my game tends go be well managed into pools that I can walk through contiguously. Even if I use a custom allocator for it, it uses pointers to refer between objects internally so I can't "move" instances of its eg. `cpBody` object around in memory. Would've been so nice to just own its data and let the library be layout-agnostic but alas.
But yeah those are the data-data glues I've been interested in--libraries that kind of own representation but not location I guess.
For your particular issue though, I think it really is bc. Lua is a language that out of the box doesn't tend to own representation, to the point of not eg. having its own class system etc, and it mostly shines when used in ways that play into that strength (eg. you want to wrap your own object ontology for ppl to script). The best libraries for Lua have tended to the invent their own representation and expect you to care about and own the mappings between libraries. esp. for JSON, the impedance mismatch with Lua tables puts it in this weird uncanny valley.
Tbh my fav method of using JSON is to not have an in memory document format at all and just treat it as reader / writer objects directly into domain data so that you don't keep paying this intermediatw representation conversion cost. eg. https://github.com/beached/daw_json_link is interesting here In my own game stuff I just have game components accept reader/writer objects to read from / write to and each component tends to have domain logic about what it means to be missing something, what some defaults are, etc.
> when performance matters, these details about using such a null representation and what different representations lend themselves also tends to matter. vs. gluing libraries to libraries willy-nilly
Well, Lua juuuust about hits the sweet spot of a good ‘layman’ dynamic language with great performance, and my hope is that perhaps it could be moved a bit to the generic-scripting-language side without losing performance. Basically, since libraries invent magic values to use instead of null, presumably Lua could provide such a value for them all to agree on.
> for JSON, the impedance mismatch with Lua tables puts it in this weird uncanny valley
Not seeing the mismatch here: to my knowledge, tables can be employed as (untyped) arrays or dictionaries, exactly the structures in JSON—and what I've long been using in PHP, JS and Python. If I'm not forgetting something, null is the only thing missing from making this triumvirate an integrated quartet.
> Tbh my fav method of using JSON is to not have an in memory document format at all and just treat it as reader / writer objects directly into domain data so that you don't keep paying this intermediate representation conversion cost
As we're discussing this on a post about a Lispy thing, I personally can't back your approach, for I'm lately buying into functional transformations big-time. Here I'm more on the convenience side: it seems to work fast enough for interactive desktop cases, though I've heard that the in-place method works wonders for busy web apps. In fact, afaik some high-level languages/environments kinda do the in-place thing implicitly by not copying strings between structures or when extracting substrings, and possibly by doing COW. Dunno if Lua does any of that (afaik it copies pointers between structures, not the data—this behavior is pretty much expected of runtimes these days).
However, I occasionally do wish that Clojure-style lazy structures were more widely employed, so I could use transformations without worrying that they might crunch some stuff needlessly.
https://luafun.github.io/intro.html is the best I've found for performant stream-y transforms. But I tend to just believe in always writing for-loops because they are how computers actually work and IMO it's easier to reason about what they are doing (of course, encapsulating logic per element and so on into their own functions). C++'s STL stuff I think is actually also good / ok here and one exception I make because you can see the code generated and things get inlined, and things like `std::remove_if` or `std::lower_bound` are actually well implemented. The rest of the functional shenanigans are cool I think for their theory (eg. as far as applicative functors for parsing and whatnot); but I think if you care about performance, the data layout in memory for cache utilization seems to be what matters now. Clojure is ok yeah but IMO you should know that it's impl'd with structural sharing + GC and decide that that's the behavior you want. Here's eg. a library that does persistent data structures of that sort in C++: https://sinusoid.es/immer/ (no transients for maps though)
Re: the Lua / JS mismatch -- I think the main thing is that you kind of have to decide if a table is an array or not, and in a lot of cases there's no sensible thing here necessarily, especially if there are a lot of `nil`s in the array -- so is it a sparse JSON array or an object with those keys (stringified -- which is gnarly) or what? etc. You can see settings for this in lua-cjson [1], and the fact that the settings exist is the issue. It's not as bad for pure JSON as much as when you start wanting to do things like generate a diff from changes in the Lua side (that's when you start getting sparse arrays) and apply those to the same document in JS, or the other way round and so on. I wrote a real-time automatic diff sync'ing thing once that was something like this; and that was the main place where there the impedance mismatch came up. nil-null basically didn't matter -- I decided on one conversion and stuck with it (I basically decided we would not use `null` for anything -- just use something that represents the actual semantic thing you want instead).
Also FWIW, I don't find CL super 'functional'. Scheme and Clojure a little bit, sure. The most functional stuff I feel like is Haskell / ML. If it's just about closures--I feel like every language except C, Zig and Ada has closures these days.
Lua strings are pretty simple -- just an array and immutable (there's no COW because there is no W) and always interned. You have to build a new string with the modification you want. For a 'builder' approach usually the pattern is to accumulate a table then call `table.join`. Usually I manage fine with `str = str .. 'new stuff'` though unless it's in some hot code path.
Yeah, I can see that. But I have to say: I think the mistake was on JSONs part. That's what `nil` should mean, imho: the value of anything which has no other value.
An object { "foo" : 12, "bar" : null } could just be { "foo" : 12 }, and if I was expecting a "bar" field, well, I can set it to null/nil myself.
Other people have pointed out ways to work around it, so I won't go down that road myself.
In Lua, nil is not a value. It's an all-encompassing void in which variables are suspended when not having some other value—simultaneously existing and non-existing: both manifest variables or fields of this reality and the infinite multitude of potential variables of other timelines. And you can't tell which reality you're in.
So it's useless for interfacing with ‘null’s of other languages.
In Lua, nil is assuredly a value. `return type(nil) == 'nil'` is a valid program which returns `true`.
It doesn't have the semantics you're used to from other languages. I happen to prefer Lua's approach, I think the distinction between undefined and null is clunky and I'm glad that `undef` didn't make the cut in 5.4.
You presumably disagree, which is fine.
There is one rough edge I've encountered, which is down to a serious wart in the standard library: `table.insert(tab, value)` inserts the value at the end of the array portion, which in Lua we express as `#tab + 1`.
But `table.insert(tab, index, value)` is what's used to insert into another slot in the array portion. This is a serious mistake which only prevails for historical reasons.
I beg of you: never shift the meaning of parameters around when you add more of them. Nothing good will ever come of this!
As a result, `table.insert(a, b, nil)` doesn't do the same thing as `table.insert(a, b)`. In the three parameter version, `b` is an index, in the two parameter version, it's a value.
It's not possible to write this sort of monstrosity in pure Lua, where nil is well behaved; but `table.insert` is a builtin, implemented in C, or whatever the host language of your particular Lua implementation happens to be.
In C, there's a difference between a `nil` on the parameter stack, and a shorter parameter stack. A Lua programmer will never see this: passing a third `nil` or just calling a function with two parameters will have the same effect.
You can actually implement that sort of logic in pure Lua too using `...` in your parameter list and then using `select('#', ...)` in the function body. For example:
> function f(...) print(select('#', ...)) end
> f(1, 2)
2
> f(1, 2, nil)
3
I didn't mean "it is impossible to build logic incorporating the number of parameters passed to a function", but rather "there will be no difference between calling an ordinary Lua function with (a, b, nil) vs (a, b), unlike table.insert, without extensive effort"
Sooner or later, a budding Lua programmer makes an index an optional, last parameter to a function, and then passes it through to `insert`.
If what you're inserting happens to be a number, this silently does the wrong thing. Suffice to say I won't ever forget this particular wart on what's become my favorite language.
You seem to be frustrated that Lua doesn't work like one other programming language: Javascript.
That's fine. Probably shouldn't use it. The nil handling is one of my favorite things about Lua, so let me offer you an alternative: instead of being grumpy about the language not working the way you'd like, instead embrace what it is, and work with it.
On the contrary, apparently you're unaware that a ton of languages and serialization formats have ‘null’. And there's nothing special about nil in Lua, it's just called ‘undefined’ in other dynamic languages (possibly with minor differences). Lua doesn't have a proper ‘null’.
But yeah, this discussion is pointless since you're obviously not concerned either about the use-cases of exchanging data with languages that are not like C, or about keeping structures ‘schemaless’ in Lua code.
The reason I used Lua was its "accessibility" -- so end users can script -- and not necessarily for the beauty and elegance of the language. It was also very easy to integrate into a C++ project.
I'd like to see him visit this thread.
https://news.ycombinator.com/user?id=technomancy