Since all upvalues can be determined at “compile” time (cause they all are locals up the lexical scope), I think they don’t pin stack frames^ for their lifetime, instead they create separate upvalue blocks in advance when corresponding activation records get created. One can also create C closures with upvalues from the stack, e.g. push a string and “close” it for few C functions (lua_pushcclosure, lua_upvaluejoin, luaL_setfuncs). But this string gets popped from the stack, and these functions live on their own and have a shared upvalue, which is not in any stack frame. It would be reasonable to assume from the C API that lua_CFunctions and Lua functions have the same upvalue mechanism, as they are completely interchangeable.
^ I mean internal stack structures, not a public stack, which is fully dynamic and may be cleared without affecting anything, e.g. in lua_CFunction. There’s nothing you can refer to on the stack, the same way you don’t want to refer to a temporary technical `push` in Assembly. Consider this:
function f()
local arr = []
for i = 0, 100000000 do
table.insert(arr, function ()
return i
end)
end
end
If `i` was created on the stack on every iteration, it would quickly overflow it, because you can’t just erase previous i, since it was closed.
(For those unaware: `for i …` “creates” a new variable/upvalue at each iteration. Every arr[?]() returns a different value in this example.)
For first, I was hoping to omit obvious optimization like splitting unclosed locals into a stack for the simplicity of the conversation, to arrive at a conceptual model for upvalues. That's going to be clearly impossible. That's most of your first paragraph. The Lua C API … almost serves to muddle the issue. (The stack there is … an artifact of the API, really? from a point of view trying to model the language.)
(A corrected version of your counter-example:
function f()
local arr = {}
for i = 0, 5 do
table.insert(arr, function ()
return i
end)
end
return arr
end
I've also changed the constant to make it reasonable to run, without creating an enormous table.)
That's good counter-example, and does run counter to my intuition. (The behavior I describe, e.g., in Python, is different; the closures in Python all close over the same variable.) One might be able to boil this down to instead using "scope" where I was saying "stack frame". (The difference being that for loops in Lua create scopes, vs. Python, where they do not. IMO Lua's is the better of the two options.)
To combine the optimization in the sibling thread (that I was attempting to avoid for simplicity's sake), then the implementation could be a stack of scope-ishes, consisting of the unclosed variables only, and then optionally some structure under GC, containing the closed-over variables (none, if nothing is closed over). The scopes on the stack would need to know of the (potential) GC object holding the closed over variables; a (normal) use of `i` here would have to look up the same `i` as what the closure hold, and as you note, cannot live on the stack.
… this is where, again, I think a diagram would help. I don't know that terms like "upvalue" and "activation record" help build understanding of Lua to new practitioners, and IMO, do more to obscure it. (This isn't your fault, ofc.; these are the terms Lua itself uses for these concepts.)
^ I mean internal stack structures, not a public stack, which is fully dynamic and may be cleared without affecting anything, e.g. in lua_CFunction. There’s nothing you can refer to on the stack, the same way you don’t want to refer to a temporary technical `push` in Assembly. Consider this:
If `i` was created on the stack on every iteration, it would quickly overflow it, because you can’t just erase previous i, since it was closed.(For those unaware: `for i …` “creates” a new variable/upvalue at each iteration. Every arr[?]() returns a different value in this example.)