Hey! This is a post by matklad [1], one of the members of the Ferrous Systems [2] team. The Rust Analyzer project [3] is an experimental compiler front-end, designed to power things like code completion, syntax error reporting, and other things that are necessary when building IDE support. Let us know if you have any questions!
Sure. Great work! Would rust-analyzer provide enough of a foundation to build something like this?: https://gtoolkit.com
Mind you, in the IDE above reflection builtin the language allows for the incredible amount of power. In Rust's case, the only way I can I see it replicated is at a layer before compile time, hence my interest in your analyzer.
Unlikely: rust-analyzer focuses on static analysis. So it aims to power something akin to IntelliJ IDEA, but probably would be a bad fit for something like lightable/elisp/smalltalk ide.
To better understand why this project is trying to re-implement the compiler frontend, I really recommend this video from Anders Hejlsberg (creator of C# and TypeScript) explaining how modern compiler construction oriented towards IDE works:
There is another approach to bootstrap an incremental compiler frontend on top of a batch compiler, which I used to create a Java language server [1]. Batch compilers usually support incremental compilation at the file level, so you are already close to a solution: re-compile the active file every time the user edits. This works but is too slow. You can dramatically improve the speed by simply erasing all the code that isn’t relevant to the current edits, which usually means erasing all the method bodies. This simple trick allows you to build a fast incremental compiler on an existing batch compiler.
We (Sourcegraph) have also found the “dump all the data from the compiler” approach limiting because it doesn’t work for the live editing use case (as the article says), and then that means you have 2 analyzers: one used with editors and one used for offline tools like Sourcegraph, DXR, etc. That bifurcation means each one gets less community involvement. The ideal IMO is to always target the live editing use case and add caching and optimizations that use language-specific knowledge, so that in both cases you’re querying a live server but it can compute the answers quickly. I haven’t worked with Rust specifically, but it’s interesting to see that they came to a similar conclusion.
BTW Sourcegraph is interested in funding this effort and building better Rust support for code browsing and search into our product using https://docs.sourcegraph.com/extensions. I just reached out to the author.
The thing is, if you target only live editing case, you can become slower for "batch compilation" case. For source browsers this probably isn't too important, b/c one can add caching using something like LSIF. However for command line compilers a similar caching won't work: in the majority of cases, you compile new code. But adding fast "batch mode" on top of "live" editing seems easier than going in the opposite direction.
Well with a good build cache, CI can do things like incremental compiles too. You can even put the cache on the network and have most of your CI use incremental compiles.
I've worked on projects that took over an hour to compile. I would not like lower performance for CI. Many workflows require successful CI runs before one can merge for example.
It's really only Rust and C++ that ever take that long, right? IMO, that's simply the compiler being unacceptably slow. I also can't see a slightly less efficient parse tree being a big factor in compile times this kind of length...
Eclipse does make this work for java with ECJ. It gains total and correct knowledge of the code base by instrumenting a compiler designed for incremental compilation. But it's also usable for batch compilation.
There's a whole class of problems around this that I'd like to see get some more attention. It shows up for all kinds of static analysis in code editors, including things as simple as autocomplete.
For instance, if I have a file that compiles and I start typing in the space between two function bodies, I'm probably implementing a new function. In fact that probability is so high that you should just assume that to be the case, even if the code I've written so far doesn't have balanced nesting yet. All decisions and advice should be based on the speculation that the code will eventually represent a new set of blocks, not a reorganization of the remaining ones.
You can't make the same assumption about if statements inside of a function, or about incomplete multiline comments, but you can assume that anywhere from zero to all of the sibling blocks are about to have their scope changed (but not ancestors).
In short, the gap between how I'd describe a commit and how the diff tool represents it also prevents live code analysis from being able to work as well as it might.
That's not a particularly hard problem, all tools which try to solve it (for example, IntelliJ), solve it easily. For example, here's the syntax tree that rust-analyzer builds for this example:
fn foo() { }
fn
fn bar() { }
Syntax tree (whitespace nodes elided for clarity):
When llvm was young, one of its design goals was to be useful for live code analysis, to prevent that problem of having two parsers that might not agree.
It seems like parsing both forwards in the normal way and backwards from the end of the file would be helpful? That is, take the longest suffix that parses correctly and assume any syntax errors are before that for error-reporting purposes.
Most programming languages aren't designed for parsing backwards, though!
Microsoft told us they definitely did not plan to make any kind of language server registry, so we saw a community need for an informational site to exist with more details than the wiki page they had. We’ve heard it’s useful, but we don’t want to create duplication. Want to file an issue on the langserver.org repo for discussion?
I don't see why one should be well versed in stupid hollywood-isms.:)And hackernews is a derogatory term that I will have you know I am very offended by. Better only than web-boy
It would be interesting to develop Xi and rust-analyzer in tandem, and see how much of an IDE Xi could become while leaving it still basically a text editor.
Language features are definitely in the near-term plans for xi, and depending on what stage Rust Analyzer is at it could definitely be a good initial target for some of that work.
We're still trying to figure out the best way to implement those. There's a prototype, but it hasn't landed in master. It's definitely on the roadmap though.
The major innovation of kakoune, imo, is to combine modal editing with multiple selections. These are two very powerful concepts that haven't been combined well before, to the best of my knowledge. In any case yes, I think kak is a great editor and a definite source of inspiration.
Sounds cool! I don't know what to say about Vim-like. At this point I think it's mostly a dream that one day, something can replace Vim. Not saying it's perfect, it's just so entrenched, and full of features.
That said, I don't think I'd be after a perfect Vim copy. I think I would like an editor that uses the same approach, doesn't have to be exactly the same keybindings.
I find building solid foundation and tooling really important, and it seems where are right there at this point with Rust. But IDEs have always been controversial beasts. Some might say Visual Studio at one point was one of the best representations of good set of trade-offs that made great code-writing environment. Now it's different story. But what I am far more interested in in terms of Rust and it's IDE ergonomics is what can be achieved differently? Rust by itself is different beast. Unique language that pushed different way of thinking and structuring code. So can that be used as an inspiration for something different in terms of IDEs and developer UIs?
I am not sure I understand the criticism of RLS. It feels to me that the Language Server Protocol does allow for incremental parsing etc. In particular, the approach taken by rust-analyzer
> Analyzer maintains a "database" of input facts (source code + information about project structure) and derived facts (syntax trees, name resolution information, types). The client then can change input facts, and query derived facts at the current state of the world.
seems that it could also be adopted by a language sever for Rust.
The criticism is not directed at the LSP. RLS just implements the specification, but as you said LSP doesn't create these constraints.
One outcome of all this might be that RLS internally gets replaced by something like rust-analyzer, which is something that the maintainer of RLS has mused in a recent blog post as one of many possible options for the future of RLS:
>The only fundamental problem with IntelliJ Rust is that it is not written in Rust, and I would really love to have a pure-rust solution for IDEs. Why? The main reason is that Rust is the perfect language for building these sorts of tools
I assume you are talking about the last bit? This is of course a personal opinion anyone is free to disagree with. However in practice many popular programming languages are implemented in C or C++ (clang, gcc, swift, CPython, V8, SpiderMonkey, HotSpot) and in theory the best language for compilers is [Oca]ml. Rust is a nice synthesis of theory and practice.
As an end user, one thing I care a lot about is if the dev env would be truly integrated. I want a one-click/command install of everything that I need for my vim to be a rust IDE. I tried in vscode and had to install two different plugins (one of which had a similar name to another and I had to go look up which one to install) python and then lldb using admin permissions which failed on my school computers .
Side: Has emacs been losing traction slowly? It's a third of vim!
For me XEmacs was the go-to environment while I didn't had access to Borland like IDEs on UNIX environments.
Nowadays XEmacs seems to have died, I have plenty of IDEs to choose from for my favourite languages, even on GNU/Linux and Emacs seems to still lack some of the nice features of XEmacs.
As for VIM, I learned VI on Xenix, but never could bother to actually use it more than for profile files, or as the go-to editor over telnet/ssh sessions when nothing else was installed by default.
I assume many on the IDE camp might share a similar experience.
[1]: https://github.com/matklad
[2]: https://ferrous-systems.com
[3]: https://github.com/rust-analyzer/rust-analyzer