We (I work at Sourcegraph) actually started off with a similar model to kythe: https://srclib.org/ This project existed before kythe was published, but was based on the ideas talked about pre-dating kythe. If I remember correctly Steve Yegge had some blogpost about solving the MxN problem, which turned into him doing grok at Google.
We ended up switching away from that model for a few reasons:
- At the time it was very costly to essentially index the whole OSS world, when most commit indexes would never be read.
- It was slow to index a codebase for a commit, and most of the work was wasted since a developer would often only look at a handful of files.
- Getting incremental indexing working usually required pretty deep integration into the build tool, so was a lot of work per language which didn't scale.
- A lot of tools using an "indexed" model end up only indexing the master/trunk branch, and you don't get your code intelligence features for PRs/etc.
At some point LSP came onto the scene. We were early adopters making a bet that this would take off. It has, and the list of community created LSP servers is large: https://langserver.org/ It also allowed us to switch from an upfront indexing model, to a model which just encodes user intent. So the underlying LSP server can be as lazy as it wants to be with respect to how it responds to the user == increased perf and reduced resources.
Things like cross-repo references doesn't come with LSP. Many LSP servers assume the user sets the build up correctly. There are also quite a few more assumptions LSP authors make which don't easily translate into an automated server environment. So we have done quite a bit of work to smooth that over / contributed some LSP servers for popular languages.
Seems like you are digging in so you might have more questions. I'll try keep track of this thread, but also feel free to email me keegan at sourcegraph.com or file issues/questions on our repo.
Thanks for the clarification. The approach makes sense. I am actually impressed by the lack of NIH syndrome, it is rare to see open source software integrated so effectively. That is
a +1 for protocol standardization
We ended up switching away from that model for a few reasons:
- At the time it was very costly to essentially index the whole OSS world, when most commit indexes would never be read.
- It was slow to index a codebase for a commit, and most of the work was wasted since a developer would often only look at a handful of files.
- Getting incremental indexing working usually required pretty deep integration into the build tool, so was a lot of work per language which didn't scale.
- A lot of tools using an "indexed" model end up only indexing the master/trunk branch, and you don't get your code intelligence features for PRs/etc.
At some point LSP came onto the scene. We were early adopters making a bet that this would take off. It has, and the list of community created LSP servers is large: https://langserver.org/ It also allowed us to switch from an upfront indexing model, to a model which just encodes user intent. So the underlying LSP server can be as lazy as it wants to be with respect to how it responds to the user == increased perf and reduced resources.
Things like cross-repo references doesn't come with LSP. Many LSP servers assume the user sets the build up correctly. There are also quite a few more assumptions LSP authors make which don't easily translate into an automated server environment. So we have done quite a bit of work to smooth that over / contributed some LSP servers for popular languages.
Seems like you are digging in so you might have more questions. I'll try keep track of this thread, but also feel free to email me keegan at sourcegraph.com or file issues/questions on our repo.