I've lamented for a while that, while JS engines are flourishing from a speed perspective, there has never been one as small or embeddable as Lua.
Now we have JS as embeddable as Lua, and nearly as small (it probably can't be quite as small because Lua is a simpler language).
Now the only thing Lua has on JS, implementation-wise, is an implementation that is very small/embeddable and very fast (LuaJIT). LuaJIT is also about as small as a JIT-compiled interpreter can be.
> Now the only thing Lua has on JS, implementation-wise, is an implementation that is very small/embeddable and very fast (LuaJIT).
Well, and a simpler, more regular syntax. Simpler semantics. Real coroutines. Tail call elimination. Multiple returns. Better iteration syntax and protocol. Better lexical scope. Operator overloading. A well-established FFI. No weird variable hoisting, nasty implicit conversions, semicolon insertion, etc.
But, sure, aside from those things, Lua doesn't have anything on JS as a language. (Of course, as an ecosystem, JS is miles ahead.)
Your parent explicitly stated that he's talking about JS implementations and not about the language. All you've said may be true, but it's completely off-topic and has no relation to the parent post at all.
Still interesting to people not knowing the other advantages Lua has. Might get some interested in the language.
Not everything has to be strictly on-topic to have merit, especially since we've already derived from the topic (from a specific embeddable JS engine to the comparative merits, implementation wise, of JS vs Lua).
Those are all very important comparisons (and I would add that both Lua and JS got the global by default thing wrong0. I would note that ES6 requires proper tail calls and that trampolining can accomplish tail call elimination (tail call elimination is different than proper tail calls).
The major issue to me is time. I know quite a few languages, but have only mastered a couple. JS is everywhere and the browser makes sure this will be the case for a long time.
The issue is that these advantages do not outweigh the ubiquitous nature of JavaScript. If I already need to learn all of those JS quirks, why learn Lua too?
I still think that https://love2d.org/ is one of the easiest, most fun and rewarding ways of stepping into game programming (especially when compared with the currently existing JS Game Dev Frameworks).
The issue is that these advantages do not outweigh the ubiquitous nature of JavaScript. If I already need to learn all of those JS quirks, why learn Lua too?
In my case, because Awesome is a nice window manager.
Although on this theory, I should really have learned some Haskell by now. And yet. Every trip to the xmonadrc is a fresh glimpse at the depths of my own ignorance.
I wanted to reply exactly that but then noticed that op is talking about implementations, not the language itself.
I'd still prefer programming in lua to programming in javascript where possible. Honestly, the only advantage of javascript outside the browser are the hordes of people who already sort-of know it (and often don't want to learn anything else).
I love LuaJIT but I am starting to feel seriously uncomfortable about its memory limitations.
64-bit LuaJIT can only use memory allocated within the first 2 GB. Mainstream desktop systems consume more and more memory so by the time you fire up your LuaJIT app there might be (next to) no memory left within that range.
32-bit LuaJIT can use the first 4 GB but again sooner or later desktop systems will leave little there too + 32-bit code is already deprecated technology on the desktop.
Makes me damn sad really. Unfortunately making LuaJIT capable of using the full address space requires a major rewrite.. one which will likely never happen. LuaJIT's author has moved on to other projects and LuaJIT is now in maintenance mode, getting nothing but small fixes.
Seriously if you are a Mike Pall-grade elite low-level hacker (I am talking to all four of you ;) ) would you please continue LuaJIT development?
IIRC, he's said that even if the pointer model gets extended, the garbage collector's not up to scanning more than 2GB efficiently so your programs will run like a slug.
Also, it's worth remembering that the 2GB limit only applies to Lua objects. Objects allocated through the FFI API aren't limited by this. So any pure data structures which don't contain Lua object pointers can be allocated with simple malloc(). It'll be loads faster that way, too.
If I could be sure that LuaJIT has access to all the memory the GC can efficiently handle I would be happy. Again the problem is that LuaJIT may only be able to allocate a few MB or even no memory at all on a desktop system where the user starts your LuaJIT(embedding) app after the first 2GB of address space are eaten up. And that scenario is becoming more and more likely every year because the RAM requirements of mainstream desktop systems certainly are not going down.
Right now I do not feel comfortable offering a LuaJIT app for download because it may crash immediately with an "out of memory" error when the user has some memory heavy app running in the background.
How does system memory usage stop Lua from addressing memory in its own address space? That doesn't at all fit with my understanding of how virtual memory works in modern operating systems.
It appears LuaJIT employs optimizations that assume that its pointers fit in 31 bits. This wouldn't be particularly surprising, fast interpreters often try and steal bits from pointers for lots of reasons, e.g. inlining values (so the indirection is eliminated), encoding the class type in the pointer (some JVMs have done this), etc.
LuaJIT stores non-number objects as NaN floating point values. (A NaN value has 48 bits of payload which can be set to anything.) LuaJIT uses 16 bits of that value for the type and the remaining 32 bits as payload, so pointers are limited to 32 bits.
If you control the application that is embedding LuaJIT, you should be able to reserve (but not commit) that address space early on in the application startup.
Other applications won't affect your own process's address space.
Thanks for the interesting link. If I am reading it correctly, they are planning to do the thing that Mike Pall tells people not to do: target LuaJIT bytecode [1]. I'm very curious to see how that will work out.
Pall is thinking about how to create the most efficient implementation here and of course compiling to LuaJIT bytecode is not optimal there. However compiling to JVM bytecode is not optimal for any language except Java either, yet everyone and their dog does it.
Accepting sub-optimal performance in exchange for getting a JIT, garbage collector etc. for free seems like a good deal in many cases. Note that Pall estimates a development time of multiple years (!) for the optimal solution.
I wonder if they'll beat castl [0], which compiles JS to goto-enabled Lua (Lua 5.2+ and LuaJIT).
Targeting the bytecode directly will not buy much, it is very close to Lua itself, semantically. The LuaJIT parser and bytecode compiler are also wicked fast.
Furthermore, IIRC, the JIT works best with idiomatic Lua code, compiled to "kosher" bytecode by LuaJIT itself.
Could someone clarify why you'd want to have a JavaScript engine in your C/C++ Project?
Is the idea if you have a desktop app executable wrapper and you could include this engine to start building desktop apps in JavaScript?
I don't think I have a clear understanding of this type of use case or how this will be used, please share.
The only thing I can think of is you could use this JS engine allow folks to code in JS, instead of C/C++, that actually interface with C/C++ APIs? Examples anyone?
There are lots of reasons to want some form of scripting language embedded inside a C++ project. This is the primary use case for Lua, especially in game development. For example
- In a game, write quest logic in the scriptable language so that game devs can tinker with quest design without recompiling the game binary.
- "plugins" (usually UI plugins of some kind). Games like Elder Scrolls Online and World of Warcraft allow for 3rd parties to write UI extensions in Lua.
- Any other logic that you want to run and frequently modify without modifying the binary.
There are other great uses-cases that I can't think of right now, hopefully others can help.
Scripting in game development is also very helpful for programmers when bug fixing. I've had the situation where a game designer told me about some bug in an elaborate mission, and instead of playing through the entire mission to recreate the one thing that breaks, I could quickly with a few lines script the small scenario necessary to create the bug. (eg. spawn a car, attach car to road, make the car drive through the intersection, hit crash)
Any non-trivial JS code is likely to be closely tied to the original platform/runtime for which it was meant, and thus hard to port. (Vs. purely computational)
I have experience of porting JS code from one environment to another. First I was using Delphi Application with MS Active Scripting and MS JScript Engine then moved to nodejs on linux, then back to windows on nodejs, and still have JS code originally written for MS Active Scripting. With nodejs vm module you may create needed context. I also was successfully using socket.io and code on top of it with MS Active Scripting based application by implementing simple WebSocket wrapper to Delphi component.
A big one for me is networking (using other high level languages as well). It frees you from having to worry about buffer overflows and other exploits, as long as you deal with all untrusted data in the sandbox and only pass your resulting data between the high and low level languages as something like UTF-8.
It also greatly simplifies logic because you can do pattern matching and filtering (similar to Erlang) more concisely than in C.
I think that coroutines are on the verge of replacing callback hell so we'll start to see patterns from Go begin to show up in other languages, perhaps embedded in this way. The browsers dragged their feet for years by not implementing generators and yield (in ECMAScript 6) and set javascript back immeasurably, so being able to go around them with embedded libraries will be huge.
In a more generic sense, since 10% of a program causes 90% of the slowdown, there is little speedup to be gained by using C in the real world, especially if it's mostly using system calls that are already optimized. Some examples of this are how a high level language like php has succeeded in the server world even though it's not particular fast or elegant, and how Objective-C is often still slow even though it's low level (because programmers sometimes do too much with their "bare hands" rather than building upon heavily optimized libraries like C++'s Boost). I think this explains the rapid adoption of Swift, which is basically another high level language in place of a more painful low level one but providing similar performance.
A embedded script engine is good for fast development, fast modification, fast experiment, and adding some app-specific DSL. E.g. in a C++ game, changing some rules or some config would take a re-compile of the project, deployment, and restart. Moving the game logic, rule, map layout, asset, config, etc, to the embedded script engine would allow immediate changes via reloading of the script while the game is running.
For embedding, you need some small language with small runtime. Something like Scheme/Lisp/Lua would work. Something like Python is not suitable. Lua is popular in the gaming scene because of its small size and its well designed architecture for embedding. Looks like Javascript is getting there.
This is a wild overgeneralization. Civ4 used Python as its embedded scripting layer to great success.
(Me, I just use C# as a scripting layer inside of a C# application. On desktops, the content C# is compiled or read from cache at runtime, but on other platforms I can include the scripting libraries as project references and have them incorporated into stuff like Mono's AOT linker. http://github.com/eropple/exor )
Low-level (like, just about assembly, maybe slightly higher) workloads that require high performance or are annoying to do any other language.
What is C++ good at?
All of the stuff C can do, but also bolting on a weirder type system, generics, metaprogramming, and a whole bunch of other stuff. It's fast, but rapidly becomes a festering tumor of functionality.
It's a katamari of design features, and even though it exposes a lot of awesome high-level abstractions the fact that it still allows working at the C level prevents it from being compelling.
What about JS?
Javascript is dynamically typed, which is terrible in many ways but bloody useful for quickly getting shit done. It is a functional language with prototypical inheritance, and so naturally lends itself to high-level programming...with regexes and functions as first-class objects, and a really pretty simple type system.
It sucks at high-performance stuff, but it allows for a lot of clever and very brief code.
So, if you want high-performance and the power of high-level abstractions, it makes sense to use C and JS--each a fairly small language well-suited to its problem domain. C++ would require too much annoying work on both sides of the problem.
A good workload for this would be: in-game scripting, network message transforms, simple command-line utilities, game engine writing, numerical library writing, asynchronous IO processing..basically, any application where you can easily separate out the low-level primitives from a flexible high-level solution description.
~
Then again, for all of what I'd said, I think you might as well just be using Erlang or Elxiir. :)
Did you coin this phrase? It's a fascinating, evocative, and quite possibly accurate description of many of the systems I find most frustrating. In any case, thank you for bringing a smile to my face today :)
It's handy for creating DSLs. For example, I'll point to SpringRTS, an open-source RTS gaming engine.
There are three concerns that make an embedded language good:
1) Lockstep multiplayer. Each player has a full copy of the gamestate on their client. Every action performed against the gamestate is done in sync on every machine in a deterministic fashion and order. This means that any custom code must be written with preserving sync in mind - something as simple as rand call on the clients would break sync. So Lua allows the engine developers to properly restrict the users to only use libraries and language constructs that will respect the sync (it fails at this, but it's a goal and Lua helps with that).
2) Code from peers. Users develop code for other users - widgets that help on the user-interface and games for each other to play. A sandboxed environment is a must.
3) Graceful failure. If a widget fails it doesn't bring down the whole engine with a segfault.
The main one for me is portability to mobile devices. I've recently built a web project with python/flask where the meat of the logic was done in serverside JS in V8. This needed to be offline capable on mobile, so when the time for that came we only had to port the Flask shell and DB logic to Obj-C while the renderer / update mechanism etc. would just work in JavascriptCore.
I guess it would allow you to ship the core of your software compiled as C/C++, and then system-integrators/sysadmins/hackers would script against your APIs in a higher-level language. No need to recompile or for users to see your source code.
Embeddable, small-footprint JS allowing the masses to script things with sounds pretty useful!
I love this. There are a ton of web developers out there who know JavaScript, and there are a ton of scripting systems written in Lua. It makes a lot of sense to add support for putting (what I'm assuming is) the most widely distributed programming language on earth in your C/C++ app.
Duktape seems to be based on the same principle of sqlite3. Distributed as one big c file. Really cool.
Another note: Try to do fib(33) in duk, it takes 9s, while lua interpreted mode takes under 2s. So no doubt it's still got a long way to go in terms of performance.
Very little performance work has been done so far. Most of the effort has been in E5 compliance and more recently to support low memory environments (96-128 kB system RAM) better. But there's going to be performance work early next year.
Out of sheer curiosity, I wonder if duktape can run node? If so, I would think it to be possible to repackage node libraries as ruby gems. Not saying this would be a good idea, I am just wondering if it would be possible.
A bit disappointed at its heavy use of macros, makes it a fair bit harder to link to from other languages (just sat down to write bindings for this, and realized it'll involve a lot of rewriting).
Disappointed mainly because the fact of no platform dependencies and two simple files would have made it a very very convenient library even from non-C.
The macro usage might look daunting but most of it is pretty typical for libraries without platform dependancies (lots of compiler dependant config and shims for missing libraries).
Look for "BEGIN PUBLIC API" for the stuff you'd have to wrap. Stop at "END PUBLIC API". Nothing there should be that weird looking.
I checked out duktape a while ago and found the documentation quite impressive. Although it didn't seem to have too many examples regarding how to embed into a typical app (with class hierarchy, etc).
Also like a lot of these other "lua alternatives", it's interesting to note development seems to be driven mostly by one guy... which if you want to use it in a project brings up the question: "What if they get hit by a bus?".
A lot of these one-man projects aren't because nobody else in the world can work on them. It's usually because everyone else is busy working on something else.
If I ended up having to maintain duktape I'm confident I could understand and improve the code with a little rampup time. This isn't rocket science. The hardest problem is finding time and funding. If a company depends on the project they have incentive to provide those things.
The documentation was indeed very very well-written.
I suspect if you're familiar with Lua then you'll be almost immediately at home, although people new to both might struggle with the idea of stack-offsets/indexes.
I started using it for a small side-project but had to put it on hold because of issues elsewhere in the stack.
From a cursory glance at things, Duktape looks way more advanced, and being written in pure C should make it much easier to embed. I'll definitely give it a try when I have time to look at that project again.
That said, the TinyJS code is extremely simple, so it makes a potentially good starting point for extending, if you want something JS-like rather than a pure JavaScript engine.
I'm not a huge fan of C++. And that's primarily out of ignorance, I stick with C and leave all the strange syntax wizardy, templates, metatypes to other people.
Also the portability of Duktape impressed me. I was able to get a test up and compiled in cygwin within a few minutes.
Maybe I'm asking a dumb question, but does this mean I can stick JS onto a microcontroller of my choice, just by wrapping it in a C program? I wonder how big it is when compiled.
Around 200kB of code (flash), and around 25kB of RAM after startup, with low memory optimizations in the master branch (for the upcoming Duktape 1.1 release). This doesn't include User Javascript, just what Duktape needs to start up.
I've been using duktape for some while now and it's totally amazing, for the wonder minds about why using duktape instead of V8 or SM here's some of my list
First of all duktape written in C not C++ I hope this is a good point for you as it was for me :)
Not everyone seeks speed, for me, simplicity and ease of use beats speed any given day, after all I'm not after complex computations and fib that loops for ever :) one file to include and pretty sure it well compile in many platforms to get a full working Ecmascript E5/E5.1 how awesome!
Compilation time, don't know about current SM compilation time but I tried once to compile V8 and I still feel sorry for doing that to my poor old machine :)
Memory management, I'm not sure how, but duktape's GC is amazing.
Last and the most appealing point for me "unfortunately no one mentioned that yet" is documentations, I don't mean just api, please visit duktape.org and see how beautiful well written everything there, Sami "the author" put a great effort not just to write duktape but it's documentation too, api, usage, comments, specifications, examples ... and best of all it was clear and very easy to follow.
The only thing I hope is more people join to help Sami maintain the project, currently the man is very active and responsive, but I think the project will expand in the next few weeks with more and more users, so I guess he will need some help then :)
It serves a different purpose, but speaking of tiny JavaScript interpreters there is also the one for the Espruino boards. Last time I checked it was said to be 95% compatible, but I don't now against which specification. Probably it's OK for most tiny applications anyway.
You've got to love it when a library is at that point where ease-of-use and features perfectly intercept.
This library is great:
On the simplicity side:
Only include two files, no linking, no shared objects, no cruft.
On the feature side:
The functions are simple, the usage-pattern is familiar (alloc, use, free) and the code is fast (written in C, which _almost_ makes it fast by default).
Great library, will probably use it when I need to write some more scrapers in the future.
> the code is fast (written in C, which _almost_ makes it fast by default).
I hate to be that guy, but [Citation Needed].
Most language implementations are written in C and most language implementations are quite slow, once you include the giant slew of hobby, toy, and less successful languages. Remember, Ruby, Python, and early versions of JS interpreters were all written in C too and are or were dog slow!
Duktape does have a section on performance[1], but it's basically just a guide to things you have to do to not make it extra slow. I didn't see any reported benchmarks.
This doesn't mean Duktape is bad. Simple+slow is a perfectly good trade-off for many real-world uses. Embedded scripting languages tend to spend most of their time in the FFI, not executing code, so perf doesn't matter as much as maintainability.
But it's really dubious to just say, "well, it's written in C so it's probably great." It's using ref-counting and linear scanning or hash table lookup for property look-up. Perf is likely not great.
Just as a point of comparison, my little hobby language[2] is also written in very simple C, but it's carefully designed to be fast too. Property look-up and method calls are just pointer + known offset. GC is tracing. Because of this (and other stuff), perf actually is good[3], typically better than Python and Lua and on par with Ruby. (With the usual caveat that all benchmarks are bad. :) )
I think even then, you'd find C isn't particularly fast if you consider the mean speed of all languages implemented in it.
Languages that have decent implementation speed are a rare outlier compared to the hordes of slow implementations out. That horde of slow languages is wide enough to also encompass slow implementations in other languages too.
Yes, very much so. If you've embedded lua before it's very easy to do the same with duktape. A few conventions are different -- for instance stack indicies are 0-based instead of 1-based (mirroring the difference in JS vs Lua) and some of the strategies for interfacing with native-code objects is different (JS prototypes aren't exact analogues to Lua metatables, etc)
Lua has full support for creating multiple interpreter instances (ie. there are no global variables, and it's totally re-entrant). But interpreter instances are not thread-safe. So the way to support multiple threads is to have completely independent interpreters running in different threads.
You can make a Lua interpreter thread safe by adding a GIL. There is implicit support for this in the code: the interpreter invokes lua_lock/lua_unlock at the proper places, the default Lua implementation just has them NOP'd out. Adding the thread primitives is up to you, however.
when you have multiple threads, isn't the idea to share data between threads? I haven't really worked with embedded lua before - does lua help with that?
Actually dukluv is the other way. It's a re-implementation of of a node-like environment using duktape instead of V8. I think the original question here was to write a node addon that allowed you to run arbitrary code in a duktape sandbox.
Very happy to see this effort. Until about 2004 my little company licensed an embeddable javascript engine written in C. For the past decade I've continued to provide free support and advice for the old customers, but haven't been able to point them to any new engines with a good API, customizability, small footprint, and licensing to replace our old Nombas engine. From the guide and API duktape looks like it might be the one! Thanks, Sami et al.
This is a useful little piece of code, but it has risks if users are allowed to insert Javascript in something running elsewhere. That might happen if you're writing a multiplayer game where users can add content. Or using it in a browser-like environment. How much power does that Javascript have? How well is it sandboxed? What about buffer overflows?
Are there any tools to create a single binary from JavaScript (using Duktape or some other embeddable JS engine)? I am thinking about the deployment scenario here. In its current form Duktape shipping a binary and a JS file separately. Is there a way to “compile” the JS file into the C file and produce a single binary?
it's pretty trivial to do that - just define a macro and use it to insert the contents of a .js file into the .c file, and use it as a string like in the sample program.
It's pretty cool indeed what you guys are doing with it, although currently there are not as many MCUs where Duktape fits... The STM32F4 Discovery board that AllJoyn+Duktape targets, is the same one Java Embedded also runs on, IIRC. From my knowledge the STM32F4's are some of the fatter micros currently on the market.
Just when you were asking yourself "how can I avoid any future responsibility for maintaining this code and meeting my customers' usecase, while having industry support for my lack of responsibility?"
How does duktape compare to v8? I've been happy with the ruby gem 'therubyracer' to evaluate javascript inside a ruby app. Method calls work both ways Ruby<->JS, etc.
I thought about this, but at least the atmega328p has nowhere near enough RAM for it. The Arduino Mega's chip is still a bit small (ref this comment [1])
These amount of flash/RAM are probably not realistic for a compliant engine. For instance, the Duktape regexp engine alone takes around 10kB flash compiled.
Luckily Javascript has a great ecosystem, so there are non-compliant subset engines targeting tiny devices. What other (higher level) language scales this way? :)
Why not use a node script for this one part? Seems to me pretty wise to use a javascript runtime when evaluating javascript. You can still use Go or whatever else for all your other tasks.
I've lamented for a while that, while JS engines are flourishing from a speed perspective, there has never been one as small or embeddable as Lua.
Now we have JS as embeddable as Lua, and nearly as small (it probably can't be quite as small because Lua is a simpler language).
Now the only thing Lua has on JS, implementation-wise, is an implementation that is very small/embeddable and very fast (LuaJIT). LuaJIT is also about as small as a JIT-compiled interpreter can be.
But it looks like Tessel is working on filling that gap as we speak -- they are working on targeting LuaJIT for their JavaScript implementation http://blog.technical.io/post/102381339917/a-new-engine-for-...