I feel the need to ramble slightly OT on how great Lua is: I was looking for an architecture/tech stack recently that would allow me to develop a cross-platform desktop GUI.
I went on a several-weeks-long fact finding mission (the longest of its kind I've ever done in my 10 years as a professional software developer).
The option that won was to write all business logic (a few thousand lines of code) in Lua, then write the GUI in each platform's native language+ui-library combination and re-use the same business logic by embedding Lua.
Another option that made the shortlist was using Haxe instead of Lua, but after several weeks of active Haxe development, it became clear that that was a bad idea, and with Lua, the developer experience is now so much better.
I definitely plan on continuing to use Lua as my main programming language.
This comes after 20 years of having python as my main programming language because I'm displeased with feature creep and bloat on python. With lua, I find that I barely miss any features/abilities from the vastly more complex python while the simplicity of lua means my code gets to "go places" where python can't go.
With lua, you find casual implementers making fully compatible alternative implementations (e.g. NeoLua for C#, Luna for Java, fengari for JavaScript, ...) With Python, alternative implementations seemingly just can't keep up with the pace at which CPython is introducing unnecessary new features and CPython-compatbility is de-facto the only meaningful python standard there is. Jython and IronPython would make the platform so much more appealing, but they appear dead in the water. Python implementations for the browser pop up every couple of years only to see little adoption and quietly disappear again.
What's more: Once you've settled on Lua as an embedding language, developers of Lua logic are free to use not just Lua, but they can pick from a host of cool transpile-to-Lua languages [1].
I guess you have already tried it but if not: there is the iup lua library which use native toolkit on windows and gtk on Linux. https://www.tecgraf.puc-rio.br/iup/
Also I would be happy if you briefly describe the problems you had with haxe.
I looked into JS as an alternative, given that ES6 cleaned up a lot of the things that had previously been bugging me about JS. I thought JS must be well-suited for embedding usecases, and the vastly larger language community, compared to Lua, must be an advantage of some sort.
But I ended up concluding that I still don't want to go anywhere near JS.
It's like browser-based use of JS and node-based use of JS are sucking the air away from JS's potential as a language for embedding.
e.g. Nashorn was part of the Java standard API in Java 8 but then got dropped. There's project Detroit to try to replace it, but it doesn't seem to be going anywhere. Most people interested in embedding JS in Java will presumably want node compatibility, even though node is not a great architecture for embedding, so that's what GraalJS is doing, but it comes at the cost of a loss of portability to non-Graal Java environments. A widely-used way of embedding JS in C# is by actually invoking the system browser through WebView in some weird and convoluted way. ...it's things like that which make me recoil in horror. Meanwhile Lua for Java and Lua for C# is just quietly and humbly in existence and using a sane architecture.
...and then JS libraries are full of this async-stuff with promises and whatnot, and I'm reluctant to go anywhere near that if I'm positively certain that it serves absolutely no purpose in the applications I actually intend to build and comes at the price of making code so much more complicated and unreadable.
I think your reasoning makes total sense and agree with the criticisms of the JS ecosystem. Just to throw out an alternative embedded-JS option for you or anyone else looking to do something similar though, there’s QuickJS from Fabrice Bellard (author of ffmpeg and more) which seems to have a very Lua-like vibe to it in terms of intended use.
Only used it for throwaway/toy code myself, but it’s been a generally pleasant and frictionless experience.
Can you elaborate on what it is you find unmanageable about Lua at scale? Is it just typing that you're missing?
The biggest difference between Lua and TypeScript in my mind is that Lua has been going on a minimally-invasive and continuous upgrade path for decades, so I assume that it will likely keep doing that for several more decades. With TypeScript I can't tell whether it's here to stay or just this week's iteration of CoffeeScript on the JS fashion treadmill.
That may be less of a consideration for more ephemeral types of applications like websites and games, but it becomes a huge consideration for applications that require large spans of time to ammortize the cost of building them like individual software in the corporate space or applications that are somehow highly specialized. (This is the area that I'm in).
I want to like Teal, but the lack of nil checks[1] is a pretty big sacrifice in the name of "staying true to Lua idioms".
>>> Every type in Teal accepts nil as a valid value, even if, like in Lua, attempting to use it with some operations would cause a runtime error, so be aware!
I know you can use (or implement your own in about 8 lines) strict.lua[2], but that just seems like such a shame to omit nil checks from a gradual type system. I still follow the project, but I put a little more hope in Pallene's[3] development, although the aims are slightly different.
Haxe is great for when you want to write an entire application in Haxe, then transpile to multiple targets, and then care about the target platform only to the extent that you want your application to just work.
The various haxe game engines are a good example of this kind of use case, where they transpile your haxe code to C++ or JavaScript and then your transpiled code combines with JavaScript code that the game engine provides or with C++ code that the game engine provides and the result is that everything "just works", both natively and in the browser.
I also have high hopes for this approach in relation to GUI programming (HaxeUI, FeathersUI, etc. are very interesting).
But my usecase was different: I wanted to make an API for my business logic that would transpile to Python, JavaScript and Java, and give a developer on each target platform a native look and feel, as if they were interacting with a native Python API or Java API, etc.
Basically, when you do that, you'll discover that Haxe is too much of a leaky abstraction.
For example when your API requires passing an iterator over Strings or returns an iterator over Strings, you'll discover that this doesn't automagically turn into java.util.Iterator<String> when you transpile to Java. Rather Haxe has its own implementation (haxe.Iterator) that transpiles to some weird Java object that a Java developer will have a difficult time interacting with. ...this is true not just for iterators but for a lot of very basic things like collections, exceptions, etc.
Haxe does expose target-specific things on the Haxe-side, so you can write Haxe code that will accept java.util.Iterator or return java.util.Iterator, but, obviously, java.util.Iterator will not be available when you try to transpile to something other than Java. So whenever you want to just use an iterator, your code is going to be full of "if this platform then behave like x, if that platform then behave like y, ..."
Outside of development around the main Haxe-based game engines, you'll also find that the community is just too thin to properly maintain libraries. For example, several libraries that implement very basic things haven't been touched in years and I discovered that the HashMap implementation in the Haxe standard library is just broken and no one seems to care or notice. It just doesn't handle collisions [1]. ...the only way I can explain that to myself is by assuming that there just aren't enough people using HashMaps on Haxe directly, because the game engines or target platforms provide something else that displaces that functionality or that games are just the kinds of applications where the errors that result from hash collisions aren't noticeable. The advice I got from a seasoned Haxe veteran regarding libraries was to not use haxelib and dependencies at all but instead copy & paste the library code I really needed, fighting hard to keep that to a minimum, and maintain it myself.
It's a pity, because the Haxe language is great, and the ideas behind the architecture are great, but the current implementation IMO just hasn't yet made enough progress on the journey it embarked upon for Haxe to be an option yet for major investments in any non-game software development.
Lua is a great language, simple enough to start with and do something with. Lightweight and fast. I love using Lua even though I hate having to reinvent every library worth of functions I constantly want to use elsewhere. Lua rocks is okay but I don't really like using it. Lua has a Simple syntax, easy to grasp but hard to master and achieve complexity with.
I suppose the more lua-esque thing to do is to not use lua in isolation and not rely on lua rocks, but instead embed it in some host language, and then rely on the host language to do the heavylifting where libraries and ecosystem are concerned.
For example, if you're scripting and need to deal with yaml files, then instead of using a yaml parser from lua rocks, you might embed lua in python through lupa and then expose PyYAML through lupa to your lua code.
This sounds limiting, but it's actually liberating. Depending on what it is you're actually doing, you can just pick the host language and the libraries you want to use in a way that best integrates with the things you need to integrate with.
> then instead of using a yaml parser from lua rocks, you might embed lua in python through lupa and then expose PyYAML through lupa to your lua code.
Well this might take the cake for the most bizarre suggestion I read online this weekend. If someone on my team genuinely suggested embedding an interpreted language inside another interpreted language that we have embedded in a static language, just to load config files, I would genuinely think they're totally out of touch.
"Please respond to the strongest plausible interpretation of what someone says, not a weaker one that's easier to criticize." [1]
Embedding python in something else was not part of my suggestion, rather I suggested using python in cases like scripting, where python would already be a good choice of an ecosystem. Also YAML-parsing was merely an example, and I wasn't necessarily envisioning that parsing a config file was the only actual purpose it would serve.
The fact is that no piece of code ever exists in a vacuum. It's always part of some application ecosystem, e.g. Windows GUI programming in C#, apache spark based ETL development in Java, scientific computation in Python, etc. ...most applications have an ecosystem (language+libraries) that's a clear "winner" in terms of adoption and richness of functionality for the given application.
My suggestion is merely this: Just go with the flow, pick the winning ecosystem, then embed lua and write your own stuff in lua while exposing whatever bits of ecosystem you need to integrate with to lua through some minimal logic that exists on the host language side.
If you context-switch at any point in the future, that gives you a good chance of being able to reuse your lua code. For example say you've developed some highly specialized piece of math for a scientific data crunching project in Python. Now you want to use that math in a GUI. Just embed lua in a host environment that's good for GUI development. ...chances are you'll be able to reuse that Lua code. If you had written the math library in Python to begin with, you're now in the difficult situation of being forced to use Python for GUI development, embedding Python in something that's good for GUI development, rewriting your code, all of which are pretty bad options and Lua might be a lesser evil.
I've found a lot of useful Lua code out there, but it's a very different ecosystem.
I need some function(s) in python, I'm browsing pip and bringing in whatever looks like it has stars in recent activity.
Lua? I'm looking for a Lua file, usually in a git repo (or a gist, oddly enough), and vendoring it with a link to the original and license.
The 'funny' part, if you will, is they're often 8 years old with like three commits total and less than a dozen stars. You would think that would make it buggy, but that hasn't been my experience.
It's just that Lua shows up in so many different contexts, with all the idiosyncrasies. The majority of Lua users don't care about luarocks, so when they want to share some code, they don't necessarily bother with a rockspec.
Just a very different culture. What JavaScript is to the browser, Lua is to anything you want: and this is a large difference, JS's relationship to the browser is much more intimate.
The Lua interpreter is best viewed as almost a demonstration program, sure, you can write software with it, but this is the degenerate case of embedding the interpreter in just enough host program to run it.
Bogdan, not sure if this interests you since you've already commented on your approach in the article, but at Planimeter we use some utility scripts based on some comments at StackOverflow by Jonathan Leffler that help autogen LuaJIT FFI bindings. We started the luajit-ffi-bindings[1] tag at GitHub.
If you're ever interested in their practical usage, Google me and let me know.
Implementing Lua is a great way to test out your language-generating program, since Lua is an extremely simple yet popular language. CLISP or Scheme is also a great test language...though Racket already has that covered.
It would be interesting to see someone implement a language like Python, C (interpreted or compiled to ASM), or Java in Racket. But it would also be an insane amount of work, probably to the point where you're almost just re-implementing the language, and I also wonder if Racket would get really slow or crash due to all of the complexity.
I believe implementing a subset of Python was also an assignment for some course.
There is `#lang c` for writing C programs in DrRacket but it uses the installed C compiler, rather than re-implementing C in Racket. https://docs.racket-lang.org/c/
> a couple things are still missing lexer support (long brackets, scientific notation for numbers)
How much of the standard Lua test suit can your implementation pass? Do you have some benchmarks to compare the speed?
What is the problem with scientific notation? Racket can parse floating point numbers, including some weird cases [1], but I guess Lua has a slightly different set of weird cases.
> How much of the standard Lua test suit can your implementation pass?
I haven't tried it yet, but I don't believe it would get very far since I haven't yet exposed ways to load Lua code from Lua modules (`dofile`, `load`, `require`, etc.). I've been building up my own test suite[1] while working through the reference manual[2]. I had never really used Lua before this, so it's been pretty interesting learning the language while implementing it.
> Do you have some benchmarks to compare the speed?
I've been using programs from the benchmarks game[3] and comparing them against regular Lua on my machine. On these programs, the overhead is usually somewhere around 2-3x. I believe that's mostly because these programs tend to be very mutation heavy. It's also possible that the way I'm compiling things down to Racket is defeating optimizations, but I haven't yet dug in.
If you're curious, you can get it to print the code it generates for any given module by setting the `RACKET_LUA_DEBUG` env var to `x` before running some code. Eg:
> What is the problem with scientific notation? Racket can parse floating point numbers, including some weird cases [1], but I guess Lua has a slightly different set of weird cases.
Yup. I just haven't gotten around to it. My goal this week has been to get it to a usable spot for embedding in my app and then iterate from there.
> On these programs, the overhead is usually somewhere around 2-3x.
Nice!
> It's also possible that the way I'm compiling things down to Racket is defeating optimizations, but I haven't yet dug in.
Part of the secret of the reimplementation Racket on top of Chez Scheme is that there are many macros pretending to be functions. The optimization pass inlines some of the functions, but sometimes the heuristic is not good enough, and you must cheat using a macro. (And sometimes the "obvious" improvements are wrong, and using a macro to force the expansion of the code has a worse result.)
For example in https://github.com/Bogdanp/racket-lua/blob/master/lua-lib/pr... I'd try using a macro to inline the fast case where both numbers are exact integers, and then call a real function for the slow cases. (Is it common that someone wants to use 7.0 to do bit operations? Perhaps it's better to inline more. That's the tricky part, understanding which is the best split. I'm not sure. Probably only Matthew knows.)
I’m currently working on a macOS app that’s built with Racket and allows the user to write small scripts to process and filter some data. While Racket is definitely my preferred language and I could easily use it for these scripts, my target audience for this app would probably appreciate a more familiar language.
I'd love to script something in racket, over lua.
Is basic racket scripting even more challenging? I guess some programmers are allergic to s-expressions. But if you gave them a DSL, you could get something wonderfully expressive.
Dunno. Hard for me to imagine someone grokking Lua metatables but being terrified of parens.
I don’t expect that these scripts will be complex enough to warrant knowing about metatables. They will mostly be about filtering data and maybe doing some math, in which case the infix notation will come in handy. That said, the app will let folks use Racket if they want to. I just don’t want to scare them off with the parens, which many people are unfortunately allergic to, as you say.
Honestly, I've relaxed about that part quite a bit.
Array tables are usually iterated with `ipairs`, appended with `insert`, popped with `remove` and so on. The difference between putting new stuff at `tab[#tab]` (the usual) and `tab[#tab + 1]` (the Lua style) just doesn't come up much.
What is a genuine mistake, which I'll never get over, is the one-based semi-open interval string indices. Representing the spot before the first character as `1,0` is just awful, and it has real consequences for interval arithmetic, which is frequent with strings. The bugs are usually shallow, but I get really creative with cursing when I see something moved by two bytes, because I got the sign wrong on some fencepost arithmetic. pretty simple: is it (a - b + 1), (b - a + 1), (a - b - 1), or (b - a - 1)? There should only be two options.
That said, it's pretty fantastic to use a language with a flaw, rather than a flawed language. I would definitely change how strings are treated, if I could, and yeah, probably zero-index tables as well, but that's about it.
The lack of multiple return values in Python, requiring changes to every call site if you decide a tuple is better than a scalar, is dramatically worse than this, and I don't see people complaining much.
Having used Lua and other zero indexed languages for a while, it's not that different from switching between a mac keyboard and windows keyboard. It can be annoying in the beginning, but your brain figures it out eventually.
The option that won was to write all business logic (a few thousand lines of code) in Lua, then write the GUI in each platform's native language+ui-library combination and re-use the same business logic by embedding Lua.
Another option that made the shortlist was using Haxe instead of Lua, but after several weeks of active Haxe development, it became clear that that was a bad idea, and with Lua, the developer experience is now so much better.
I definitely plan on continuing to use Lua as my main programming language.
This comes after 20 years of having python as my main programming language because I'm displeased with feature creep and bloat on python. With lua, I find that I barely miss any features/abilities from the vastly more complex python while the simplicity of lua means my code gets to "go places" where python can't go.
With lua, you find casual implementers making fully compatible alternative implementations (e.g. NeoLua for C#, Luna for Java, fengari for JavaScript, ...) With Python, alternative implementations seemingly just can't keep up with the pace at which CPython is introducing unnecessary new features and CPython-compatbility is de-facto the only meaningful python standard there is. Jython and IronPython would make the platform so much more appealing, but they appear dead in the water. Python implementations for the browser pop up every couple of years only to see little adoption and quietly disappear again.
What's more: Once you've settled on Lua as an embedding language, developers of Lua logic are free to use not just Lua, but they can pick from a host of cool transpile-to-Lua languages [1].
[1] https://github.com/hengestone/lua-languages