Not a ruby developer here: that sounds terrifying! Does it make it harder to have a proper mental model of the language (note: not the libraries) or is this mainly because of flexibility (too many ways to skin one cat)?
I don't write Ruby regularly either, but I wouldn't say that syntactic complexity, is necessarily equivalent to semantic complexity. And the syntax is the only part that's relevant to Tree-sitter: it's not an interpreter/compiler.
Note also that (as I alluded to above) the parsing technique that Tree-sitter uses, "LR parsing", makes some things more difficult to parse than they'd be with another kind of parser. This is a deliberate trade-off, because LR parsing makes certain features of Tree-sitter, like fast re-parsing in response to input changes, much much easier.
So, a syntactic tree is a list of elements, grouped by their ordering, which are to be parsed from their arguments, as they appeared in the input. Or a grammar tree, which is a set of elements.
There's many things we can do to make Tree-sitter simpler to read and write. Perhaps, like in Perl, there are syntactic categories of types that make it much easier to find things like nodes in a tree, since they're the ones that come in the input. Or I'd be willing to say that maybe, like in Haskell, certain aspects of the language, are syntactic categories, like the parser. So some things that might not be obvious in code, like what the syntax for a class of names is, might be obvious in theory, too. Or, at least they might be obvious in a particular way. Or some aspects of the compiler are really special, and we can infer those in terms of what the compiler does.
Or, of course, we can do all these other things, too. We can rewrite the parser, or the compiler, to try to do more or less anything that the parser does. Or maybe we can make Tree-sitter a lot simpler in general. Which I think is probably what you've been thinking about.
Flexibility. “Too many” is debatable: most organizations wind up settling on a subset of the idioms that Ruby provides, and some of the more esoteric constructs see infrequent use anywhere.
There has been, however, discussion about the need to clean up some of the lesser-used language feature, but obviously doing so carries risks.
> Not a ruby developer here: that sounds terrifying! Does it make it harder to have a proper mental model of the language
It is a little terrifying in the sense that I'd not want to write language level tools (eg: syntax highlighter).
But if you have scheme on one end and natural language on the other, ruby leans à bit towards natural language - but in a good way. In some ways ruby isn't that different from Smalltalk - but it has a lot (sometimes I think too many, sometimes not) conveniences.
Parantheses and brackets are largely optional "where it makes sense". Conditionals support postfix, eg these are equivalent:
if should_send?()
send_mail({to: 'u@x.com'})
end
send_mail to: 'u@x.com' if should_send?
My mental model of Ruby is one the simplest of any of the languages I've worked with, but it's also the hardest to put into any words. JS actually does beat it out, and then Scala and Python come after.
Everything is kind-of-but-not-really an object, a reference, and a function, all at the same time - which sounds complicated but in my head... turns out to be pretty simple. Everything's just kind of different flavors of the same thing. `attr_accessor` is a good place to see this in action.
The flexibility comes more from the variety of available core language options (procs, blocks, and lambdas) and core libraries (map/each/collect, for example), not from a variety of underlying concepts.
It's mostly to work less surprising to the programmer, AFAIR. Probably the most complexity is from having to differentiate local variables and methods depending if the symbol had an assignment before in the scope.
Precisely because the language is complicated and less amenable to LR parsing.