LuaLaTeX allocates memory as-needed, see section 3.4.1 in the manual [0] (and comments/answers in this thread [1]).
Base TeX has an arbitrary, by modern standards low memory limit, leading to a whole class of errors plaguing unsuspecting users [2], and spawning entire extensions to deal with these limitations [3].
This is simply an artefact of times past and has no technical relevance nowadays.
LuaTeX allows dynamic allocation, with the available system RAM as the upper limit (so effectively, no limitations in everyday usage).
Now, I could not find a mention of memory handling in the XeTeX reference manual [4]. People are using tricks like `tikzexternalize` with xelatex [5, 6]. Especially the first point makes me think XeLaTeX inherits base TeX memory handling/limits, but I cannot confirm this.
I just know that all my problems disappeared when switching from XeLaTeX to LuaLaTeX.
Lastly, see here [7] for a comprehensive (albeit somewhat anecdotal) list of advantages of LuaTeX over XeTeX. Of that list, `microtype` is another significant functionality I rely on.
I’ve been a luatex advocate in the past¹, but I use xetex instead, unless I need the Lua integration. The memory handling is the reason. I find that for documents with a lot of fonts, luatex eats all the memory available and then crashes, taking a huge amount of time to do so, whereas xetex just breezes through the same document.
What's troublesome for me is that I have been using
* xetex when I needed a font that was not easily achievable in pdftex over the past decade
* pdftex for everything else because microtype(TM) just works(TM) (even though kerning can be done using fontspec and font features in xetex).
I've tried luatex multiple times over the past decade, it was mostly just too slow. Now luatex is fast. But I have no idea if I now "should" use luatex over pdftex for best out-of-the-box results or not.
Unfortunately, switching to luatex is not a zero-effort (moving to polyglossia, using fontspec, maybe removing some magic in many-lines private templates, and so on).
For all I know, because I'm always curious and peek at PDF file properties as a hobby (if only to check which cool font that is), basically every scientific paper I read is set using pdftex. luatex usage in the wild is, as far as I perceive it, nil, outside of enthusiast luatex user spheres. I don't think this will change unless texlive drops pdftex (as it still ships ptex and even uptex, it probably won't for a very long time).
> basically every scientific paper I read is set using pdftex
That is only because their templates are years behind the curve and they are slow to update. It is not an argument for the advantages of pdftex, aside from its stability, gained over many decades.
LuaTeX has been nothing but stable for me, so from a technical standpoint, there is no reason not to switch.
As far as scientific papers go, the publishers and editors probably value stability and backward-compatibility (I would).
Officially, luatex is the future. ConTeXt is based on it. I’ve heard that the kinds of problems I’m having are caused by its font-loading routines, and not the core parts of luatex, but without further research that doesn’t really help me.
Are there some more details of the memory limitations you can share with us?