In case anyone is interested: Roblox (who is making Luau) is hiring.
We've been growing fast, and are looking for strong C++, systems, and application engineers. As a highly vertically integrate gaming platform, we've got a lot of hard problems to solve - from programming language design (see OP!) to debugger tooling.
Hi JHelms, I'm the author of the largest pure-Lua game engine on GitHub, Planimeter's Grid Engine 9 (https://www.planimeter.org/grid-sdk/). I'd love to chat, and sent you an email.
I wonder how the interpreter works. In their Performance page they write:
> It’s similar to Lua interpreter in that it’s written in C, but it’s highly tuned to yield efficient assembly when compiled with Clang and latest versions of MSVC. On some workloads it can match the performance of LuaJIT interpreter which is written in highly specialized assembly.
One thing I've been experimenting for my own project, is using tailcalls. All main compilers (gcc, Clang, MSVC) are able to compile that into actual tail calls (i.e. `jmp`, not `ret`, so that the stack doesn't overflow) (with optimizations enabled), even for indirect calls (i.e. dispatching depending on the next instruction). Function parameters (VM state) are assigned to registers, alleviating Mike Pall's main complaint about writing interpreters in a high-level language, that the compiler cannot optimize the huge loop as well as a human can. I need to test this on a full interpreter with 50 or so instructions, to see if the optimizations still apply.
I am currently exploring the similar approach. The idea is to compile to llvm IR and then augment it. There's a musttail marker in the language to enforce tail calls no matter what the optimisation level is.
Passing VM state via arguments is suboptimal since argument registers are often clobbered and one would normally use callee-saved registers. I have developed a patch for llvm allowing to specify registers used for passing arguments and making more registers available for allocation without spilling.
I intend to reimplement LuaJIT VM in C with IR augmentation to verify the performance.
My understanding is, for "fast path" instructions (e.g. integer addition) there's no register spilling necessary, so passing VM state in registers makes sense. For more complex instructions, you rely on LLVM to optimize registers as it would normally, with callee-saved registers. Of course, one must still pass as little state as possible - e.g. just sp, cp, maybe top of stack / heap, and then a pointer to *vm_state for all the rest.
GHC allocates registers for arguments from the fixed list where most but not all registers are callee-save in other common CC-s. This is a big improvement; still there are valid reasons to want it customisable.
E.g. in LuaJIT next instruction is decoded as a part of an instruction handler before dispatching to the next handler. Instruction arguments are passed to a handler; they shouldn't use callee-saved registers.
Another complication is the stack frame. There's a requirement to keep the stack pointer aligned (16 byte on x86_64). Call instruction pushes the return address and the pointer becomes misaligned. If the called function wishes to call further functions it has to adjust the stack pointer to make it aligned, even if it doesn't use the stack. The stack pointer is adjusted in a function prologue, hence there's a slowdown even if a nested function call is on a cold path.
Whether these micro-optimisations are worthwhile remains to be seen.
I'm not sure I understand your complaint about callee-saved registers. The point is that arguments are passed in registers, as opposed to on the stack, and most calls are tail calls anyways (so `jmp`, not `call`/`ret`). It doesn't matter if the registers are caller-saved or callee-saved, as the context is destroyed anyways, the function never returns.
In case of GHC, the functions could return, but the fast-path is that most functions access VM state, and many functions have very little need for registers; therefore, forcing registers be caller-saved would just cause most functions to push and pop the stack without any reason. If non-standard calling convention (e.g. C calling convention) functions are called only rarely, then using caller- or callee-saved registers doesn't really matter.
We are on the same page irt tail calls. I assume that instruction handlers (non-standard calling convention, tail-calls next handler) need to invoke helper functions quite often. These helpers adhere to standard C calling convention. VM state is in registers. Unless they are callee-saved, registers have to be saved to stack and restored after each helper call. (LuaJIT VM uses helper functions heavily.)
The Luau FAQ[1] when they first announced the project mentions that Mike Pall post. I assume they do the computed goto trick for fast interpreter dispatch (or just have some inline assembly now).
They explain it in some subsequent page. Also computed goto is not really performant enough for fast scripting languages, it's essentially a switch (indirect func table) without range check.
The performant variant is to squeeze it to have room for other code, keep important registers asis, and jit the bytecode array into sequences of static calls.
They next best variant is to call the next op statically. Indirect func tables are hard to predict by the CPU.
They explicitly removed tailcall for better user experience, as it messes up debugging and stack traces. Maybe it could be added back in perf mode.
It may not be exactly the same but Terra http://terralang.org/ uses Lua a meta programming language and the Terra language itself is effectively a typed Lua compiled to machine code with LLVM.
Terra looks really interesting (Halide is another language along this meta-approach - https://halide-lang.org/). It's too bad it never made it further than being used for a few PhD theses.
I like Terra, and I have played with it, but I have not found a use for it to push me to really explore it. I'd love to hear from anyone using Terra for cool projects. I had watched a talk about MAD-X [0] that I found very interesting, but it is specifically for particle accelerator design in physics [1].
Not necessarily, since it's in production they might be hoping for a bit of securty-by-obscurity if they haven't had time to get an audit the code (runtimes are always hard to get 100% correct since they're so generic, esp for a language such as Lua).
Seems like this runtime was announced to Roblox developers almost a year ago. But i don't think they've put much thought or effort into open sourcing it and it just happend to pop up in peoples feeds now since Arseny published an article(#1) about his 8 years working for Roblox 2 days ago and somebody probably took the link from that article (Quite an interesting read if you're into gamedev).
Very happy to see this, Lua has a bunch of great properties (small footprint, speedy JIT, simple datastructures). It's great for quickly hacking together some script functionality, and it (already) handles enormous scale in enterprise applications (e.g. Cloudflare).
I maintain the Haxe Lua target, and have a good understanding of how Lua relates to other languages (at least through a "Haxe" lens).
I think the main things I get hung up on is the 1-indexing scheme, and the lack of true "null" fields on tables (Lua treats these as missing). This makes Lua tricky to integrate with other languages, especially if they serialize/de-serialize data. You essentially have to pick a work-around, and unless you know what you're doing (and what you're doing 5 years from now), you're going to feel a lot of pain.
I realize changing these things would make Lua less "Lua", but virtually every other language Haxe supports has straightforward support for these behaviors. It's really the only two nits that have marred an otherwise pleasant experience working with Lua.
Possible to make this work on LuaJIT as well? If this is transpiled to Lua 5.1 it should be possible to make this work with LuaJIT runtime. Combined with Openresty and gradual typing this could be a quite a good language for writing Backend services.
Good idea. But then why not use one of the already available statically/strictly typed languages which compile to Lua or LuaJIT bytecode? See https://github.com/hengestone/lua-languages
It's also made by the same person who created LuaRocks and htop.
I'm personally hoping tl or something will take off as the de-facto TypeScript for Lua. I'm already convinced that types are better. I'd use types all the time if I could. It's just that not a whole lot of people seem to care about Lua, even with how fast LuaJIT is, so you have to compromise with projects adding types to Lua that are only in their infancy, assuming they'll still be around after a couple of years. TypeScript has proven that gradual typing can be done right and had a bunch of innovations I never would have thought of (the keyof operator for creating a type of "the set possible key names of T" for instance). Having that kind of power in Lua on top of one of the fastest JIT compilers on Earth is my dream.
Together with Teal, LuaJIT can become a very safe platform. Vela (fork with many improvements to LuaJIT) has optional, native recursive immutability built into the JIT for Lua-land objects and also compiles next() to dig pairs out of the nyi hole.
Lua runs the same risks as Scheme which is another small language - of multiple incompatible implementations cropping up because it is so easy to create an implementation. [Dont mean to downplay the difficulty of writing an implementation just to highlight that the engineering cost of doing so is not prohibitive enough]
I was almost semi-proud of having done my implementation, why did you ruin the party.
Jokes aside, and not to boast about what I did, honestly, but Lua is (in hindsight - I wish I realized before) full of gotchas which are definitely not easy to implement (just one word which still makes my stomach hurt: COROUTINES).
I honestly think there are simpler (and better) languages to target. With hindsight, experience and pain, of course.
Starting with scheme for most people I talk to seems to follow a similar path: start with one implementation. Write some code. Try it in another scheme. Discover r6rs. Either like it and switch to chez or guile or keep using whatever it was you are using.
I have a pretty small data set (5 people) and three of them are guile users.
Scheme is however willfully underspecified, whereas most other languages lack specifications of even a proper base language (except for code). Implementing scheme is pretty easy apart from continuations and macro hygiene, but those parts are very well specified somat least you know what you are aiming for.
On the contrary - multiple implementations happen if there is a specification. These languages tend to fragment if the spec is too big or if there is no good compatibility testsuite. On the other hand, languages like Perl make it impossible to create another implementation because the original implementation is the spec. That requires any other implementation to follow the original's behavior in every weird corner case that has been observed or might just be spotted in two years from now. Reimplementers will never be done and be tightly tied to the original project, which would make the whole endeavour pointless for most intents and purposes.
The front page sample does not explicitly use the Point type in the declaration of p. Typo or does the compiler actually match table variable definitions to available type definitions?
I gave this game a try last night, it was amusing all the different types of games you could potentially play, I was wondering who developed each one. It seems you can build your own games (game modes? unsure of the term for Roblox) and earn some income on it. I had no idea it was done in Lua, might give it a shot to see what is possible.
After having read this HN post this morning I went ahead and installed the studio, it's pretty damn nice, there's a bit to unload for me mentally but I love the whole thing so far, you can test within the IDE and switch between user / server which seems interesting. I'm currently teaching myself Godot so I can make some games on my time off but this seems like an even easier approach given that a lot of graphics are already available plus the user base. I guess now I have to learn how this all works, and what kind of games most users like so I can try to make something interesting and different.
Thanks for the links, so far from what I can see Roblox Studio is pretty good!
Roblox is essentially a full on game engine and store now, comparable to Unity + Steam with free, automatic cloud hosting and matchmaking. There's a lot of daily users, and the top earners on the platform make multiple millions of dollars from it.
I first learned programming due to Roblox. It's really easy to pick up, and you can make almost whatever type of game you want. I honestly believe that highschool programming classes should universally use it for introducing how to code rather than Javascript or Unity.
The way I see it, Unreal is eating Unity from above, extending down, and Roblox is eating Unity from below, developing awesome new technologies. Gotta be hard being Unity :-)
Roblox has tons of innovation. It has a volumetric/voxel based lighting system. It has networked rigid body dynamics built right in. It lets you take your avatar between each game, keeping your look. And you can late-join any game -- you don't need to be there "at the start."
Some games have tens of thousands of moving pieces, yet load reasonably quickly. Check out the tornado in Natural Disaster Survival for a good example!
I've played this a lot back when Roblox was still brand spanking new. It's absolutely a lucrative area if you can manage to make multiple game-modes with people buying your in-game items for RBux (RBux is the premium currency roblox, which can be used to extend your developer membership or be redeemed to cash). Those big game modes out there with a lot of traffic are actually making real decent money. The game is played by millions of kids willing to spend money on cosmetics/advantage on game-modes. If I recall, I think it was 2017, Roblox rewarded $200k to a young developer for his work on a DayZ Roblox clone.
Roblox is like a game engine that you can script in Lua, plus the servers to run the engine for you. The barrier to entry to make a game is relatively low -- making an obstacle-course game requires almost no coding, for example.
Ah man! I'm just starting to learn, so if I get far enough along maybe I'll write up a blog post and submit it on here documenting my concept, still unsure what I want to do currently though.
I've been playing with Minetest instead, because it is easier to work in isolation without a Roblox server. Can you run a Roblox server in single player mode?
Thanks! I'll give it a try this weekend. I tend to isolate, so social gaming is not my comfort zone, which is why I like to play Minetest on my local machine by myself. I do play with my kids in the same game on one computer though, so there is hope ;)
No, literally TS. It compiles to plain Lua, doesn't add any language features apart from types, has an equivalent strict mode and uses .d.tl files just like .d.ts
I like the mention of an internal tool called "SECRET TOOL". It makes me wonder why the name would be redacted, although I'm amused by the idea of that actually being its name.
As a Roblox user, I believe it might be their tool that allows Luau code to be stored on the filesystem and automatically synced to games. There are references to "FileSyncService" in the binaries, but the engineers refuse to open it up for some reason.
Hello! I am the lead developer of the Luau type checker.
I actually hadn't heard of Teal! I don't think it existed when we started.
Judging from the Teal documentation, I'd venture that it looks pretty similar to Luau in everyday use.
If I were to guess, I'd venture that the biggest differences are probably the kind of type inference we do (Luau's inference engine draws inspiration in equal parts from OCaml and TypeScript), a bunch of Roblox-specific features, and that our inference engine is all C++ for performance.
I've been meaning to try out roblox for a while and lately I've been getting pretty hyped about teal (for context, the guy behind it is the creator of luarocks and htop, among others) so I might see the differences up close sooner rather than later :D
EDIT: (unrelated) Already not looking forward to people asking "Lua" questions about typing and constantly having to explain that Lua is not just a roblox thing xD
It actually only gained proper type support relatively recently (I don't remember exactly when as I haven't touched it in a while). It also has some other features other than types (again, haven't used it in a long time, so I can't really remember, but have a look around).
The real difference tho is age, Luau has been around since 2006 or so (used to be known as Rbx.Lua).
I'm curious what they do to the garbage collector; over the last few years we've made quite a few optimizations to our Lua 5.1 codebase to reduce the impact it has and yet just last week I was in there again squeezing blood from that stone.
What problems have you had with the garbage collector? Excessive pauses? Insufficient throughput?
Many other game-focused scripting languages use reference counting instead of tracing GC to eliminate pauses at the expense of throughput. Would that help in your situation?
We use a hybrid of ref-counting for many things but Lua still needs to do sweep phases in its GC that cannot be amortized across multiple frames. While we do most of Lua's GC asynchronously some of the phases take so long that they can stall the main thread that wants to execute some bytecode.
Here's an older tweet with some examples (I'm not the one tweeting but I made the optimizations):
> Note: our garbage collector optimizations are still in progress, so this section doesn’t document them.
It will be interesting to see whether they make any major changes to Lua's garbage collector, which is an incremental tracing GC.
Several gaming-focused scripting languages use reference counting instead of tracing GC, even though throughput is lower. This is because RC (almost?) eliminates the need for stop-the-world pauses, making it easier to maintain a consistent frame rate.
I don't know anything about it, but just looking at it, it seems like a giant mess. 1-based indexes. Semicolons required here, but forbidden there. Like... why?
- you basically always want a local but you have to ask for one, it's a bit noisy. 'strict mode' is easily applied to the global environment, and catches most outright errors here.
- there is an adequate amount of library and extension code, but less than you'd expect coming from Ruby or Python. The lack of a blessed object system sometimes means taking more effort to understand what other people's code does.
pros:
- variadic functions and multiple return values make for a truly dynamic language
- first class functions which are fast and work well: everything is a closure, a whole file is just an anonymous function called a chunk
- the table as a single compound data structure works very well, and metatables are a unique and excellent solution to extending behavior. Array portions are required to be compact (non-nil from 1 to #tab), which can be worked around on the few occasions when it's inappropriate.
- asymmetric coroutines.
- first class environments, which are just, yep, tables. this allows for a few really nice things.
- fast. LuaJIT is very fast. Both Lua runtimes fit in a typical L1 cache, and it uses a register VM, which is a good fit for real processors vs. stacks. No global interpreter lock, all C level Lua functions receive a Lua_state as the first argument and are fully reentrant.
- in general the emphasis on minimalism means the whole language "fits in head", and the tools provided are more than adequate to get jobs done.
- the LuaJIT FFI, which parses C declarations directly, is simply a pleasure to use, and is the only one of which I'm aware where you pay an absolutely minimal penalty to use it. There's just nothing like it. More languages need to have this.
miscellaneous:
Semicolon insertion is simply a non-issue, unlike JS it was implemented properly. I have well under one semicolon per file, less than one per project, even. It's very, very occasionally nice to write more than one statement per line; that's what the semicolon does.
> [...] and multiple return values make for a truly dynamic language
If you are using a word "dynamic" as "once-good-but-now-increasingly-considered-worse-and-worse", I agree. Lua is definitely too dynamic to be safely used.
Weeks ago I had to debug a Lua code looked like this:
If you can catch a problem, great! But most wouldn't be able to do so. And I used Lua for a prolonged time, to the extent that I have made a type checker which absolutely flags this case [1] and developed a very good list of intrinsic problems of Lua which obviously includes this one and still failed to notice the problem. Lua is a small language, but packed with lots of similar problems.
I tend to use lpeg for any complex string manipulation. gsub has bitten me once or twice by not thinking through whether something is a pattern nor not, sure.
In some sort of deploy scripts, converting a command "template" into the actual command by replacing a fixed string "<arguments>" into the escaped arguments (I know, this is not sufficient but also not the exact code, I've simplified it for the easier inspection). Now guess what's the problem without looking at the manual.
Also, don't say "use lpeg". Unless Lua supports a proper package system for every circumstance it's not a solution. Note that this deploy script runs in an embedded context, so no to luarocks and kins.
But, no: I meant that after the fiftieth time I had to modify every call site of a Python function because I realized I needed to return a tuple rather than a scalar sometimes, I came to really appreciate multiple return values.
Thanks! Quite a few things I didn't know in here, and on balance it does seem compelling.
Still can't quite understand why they would choose the 1-based indexing, it's just such an unforced error, unless the intention was literally just to fuck with people.
I suppose I could get over being fucked with, though.
It's small, simple, and really easy to integrate into a bigger application. It thus makes for a great language to use for glue code when you're doing things like writing game logic / UI.
Also, semicolons forbidden? I don't think there's anywhere you're disallowed from using them to separate statements, though they're largely optional / redundant with newlines.
We've been growing fast, and are looking for strong C++, systems, and application engineers. As a highly vertically integrate gaming platform, we've got a lot of hard problems to solve - from programming language design (see OP!) to debugger tooling.
You can email me at my username plus roblox.com.