Hi! I am the author. I will be glad to answer any questions.
First of all, I don't want another Emacs rewrite, much less in Guile. Mixing languages is not good for power-use, which requires ease-of-use, or at least conceptual simplicity. I talk more about it in the article in the Project's Philosophy/Homogeneity section in [1] The Power of Structure.
I am proposing we need to really start considering a different paradigm, and attempting to do it right, and that's structural editing. People are wondering if it's possible to be writing better structural editors. We all know there have been attempts to do those, and, well, lo and behold, those were janky too.
But they don't have to be. When people start thinking of structure-editing, they immediately jump to the "how do we do C++".
Well, in fact, I could tell you how we could do exactly that: you could start small. You start with what you know. And you know that you could, say, start with structuralizing the {} brackets. That's a semantic unit. So, that's a start. Even without getting down to the compiler level.
But I am not arguing I am about to do wonders for something as complex as some mainstream langauge in terms of a structural editing. I believe it can certainly be attempted, though, and certainly improved, peacemeal. And what you can't do: mix it with the traditional string-based editing. Or take python: that one would structuralize pretty nicely by indentation. Would it accomplish everything? No. But it would certainly help.
The gist of it boils down to the fact that you don't need to start at that very complex level, you can do things piecemeal and still get many benefits. Ask yourself this: can you edit a /list/ structurally, i.e. edit like in a string-based editor while maintaning an actual list behind the scenes? Sure, you can. A tree? Absolutely. Look at Paredit.
THERE'S NO REASON FOR THAT TO BE JANKY. NO reason why that wouldn't work.
It can absolutely be done.
And, really, structural editing like I am proposing /subsumes/ string-based editing, because you can just write a specialized editor for general strings, and use that for things you don't know how to structure yet. And yet, even those string-based editors can be specialized further, as some semantic units like words, expressions and even characters are often immediately apparent.
What does that give us? At the very least: object identity and programmatic access, and having the ability to pick your own data structure.
This kind of small things are what's actually going to be very useful for stuff like note-takers and computational notebooks and REPLs and what not. We don't need to start with programming languages (though I am going to do a Common Lisp IDE).
Please, ask me anything! Let's talk!
PS Another very important point is that ambiguity can be localized. Look at the Alchemy section in [1] where I discuss dealing with the reader (but I also talk about it in Rune).
PPS And thank you for posting this. It's exciting to be reading comments. Truly.
> Well, alright, I don't see no CSV-mode to arrange everything into a pretty table and then let me filter/sort/edit the damn thing.
`csv-mode` is right there on ELPA, the default package source for Emacs. You went to MELPA, which only has packages from people who don’t want to license their code the same way as Emacs. Install `csv-mode`, then type `C-c C-a` to format it into columns, `C-c C-s` to sort by a field, and `C-c C-n` to sort numerically by a field.
> A table editor within emacs will be janky, it will be a slow heap of cowdung, outspreading and dispersing, channeling the fumes.
This is completely wrong. There are half a dozen Emacs packages that give you variations on the theme of a table, and none of them are slow or janky in my experience.
> Well, scrolling that thing was not fun, I will tell you that. Neither was getting spammed with:
All the errors you report look like they were caused by packages you have installed, not Emacs itself.
> Timed out waiting for property-notify event
This one is actually caused by your clipboard manager rather than an Emacs package.
Didn't know about that one. Jesus, Elpa, really? Not that it changes anything.
>> A table editor within emacs will be janky
> This is completely wrong. There are half a dozen Emacs packages that give you variations on the theme of a table, and none of them are slow or janky in my experience.
Haha, have you used org-made lately? Have you tried formatting a large table? Is it fast, you think?
Look, make a 100colx100000row table that you can edit (although, something like 10 by 20 will do the trick too). Not just view, but edit. It's not a question of whether it will be janky, it's a question how long you have to wait before exiting the damn thing in a furious fit of anger. That's just what you get with plain-text, plain and simple. I don't see how this is even an argument.
And then, why do you think it is that there are half-a-dozen packages there for it anyway? Is it, perhaps, because no one of them really does the job?
> This one is actually caused by your clipboard manager rather than an Emacs package.
I take a rant license on that one.
> Etc.
But, you see, that's the problem: these problems don't end.
I’ve used various large `org-mode` tables with great success and no rage, but a table that large means that you need a database, not a spreadsheet. Use sqlite for it instead and you will be much happier. Use the right tool for the job; nobody is forcing you to use Emacs for every single thing.
I find that whenever people complain that Emacs is slow or janky, it is usually because they have configured too much stuff to run all of the time. They got excited by the possibilities and downloaded every single shiny package they could find. After a few years of downloading shiny packages Emacs just isn’t fast any more, and so they blame Emacs.
Exactly, a custom structure, a custom object. That's my point. And that point is: the ability to specialize on custom structures, which are unavoidable if you are to keep your sanity.
But why are you complaining about that. Who told you that it would be a good idea to use Emacs instead of a database? Why did you ever come to that conclusion?
Because it's often claimed than Emacs can do everything. Or, if not claimed, than assumed. And if not assumed, people go and try to do this stuff anyway. I gave an example of an object-oriented spreadsheet in my article.
You can't stop people from trying, but that's just the wrong paradigm to try those things. My point: we can still have an environment that can do all of these things. Embed a spreadsheet and interface a database textually. Why, is what emacs giving us really the end of it, you think? Because I don't.
> But they don't have to be. When people start thinking of structure-editing, they immediately jump to the "how do we do C++".
What does that mean?
> start with structuralizing the {} brackets. That's a semantic unit.
So, vim.
Getting simple structure done is not the problem. We have them everywhere and everyone can build their own tools in proper environments. Supporting the big picture and custom structure is the unsolved problem. The best we get in that realm would be support for XML, Lisp, maybe also JSON and YAML and the likes. But all those are hyper specialized tools, optimized for those specific cases. What we lack is something good which generalize this.
> The gist of it boils down to the fact that you don't need to start at that very complex level,
No, there is, there always is. Because if you start simple, you always end up with the wacky unsatisfying solutions at some point. Nothing scales well to infinity. Micro-managing and macro-managing are different scopes with different solutions. Simple is good for the micro-levels. Complex is good for the macro-parts. Think about the text-editing of advanced text-editors, and the abilities of vim. Advanced text-editors are simple, and not bad. But compared to the complex editing of vim it still cannot be compared.
That means they are trying to solve some very difficult, general problem first. But, you see, that's exactly the problem: you don't want to go general. You want to go: specialized. And here's the key: then you want to mix and do the interplay for your simple, specialized well-working elements. And, indeed, you can define some general interface properties for that, once you have it.
> Getting simple structure done is not the problem.
I agree!
> What we lack is something good which generalize this.
I can't emphasize this enough, but trying to find a general structure and fit it for everything is a path to failure, and I am arguing vehemently against such structures or approaches [1]. Strings are such a general structure. Although there were others proposed, like Ted Nelsons zig-zags. That's where you do not want to go.
But again: generalization is not the problem, the problem is the ability to specialize.
But, yes, you are right: there has to be an overarching system, and that's exactly what I want to do. A system where the simple parts can interplay. And importantly: embed.
The simple structures I listed, and a few others? Those will be enough for the applications that I want to do. And that will be plenty useful to me, already at that.
See, the fact of embedding itself lets you manage the complexity, because that's where a lot of complexity lies within: in hierarchies. And then, when you are doing editing operations, you have full and easy introspection into all the structures that you are operating on, so, doing them right will be possible.
PS I have been using vim for quite a few years, and I don't see it as some kind of complex editing (other than bindings and modality which takes getting used to). Maybe I am misunderstanding what you mean by this exactly, though.
PPS I am sorry for responding slowly, there are quite a few comments.
> trying to find a general structure and fit it for everything is a path to failure
Depends on whether you consider everything literal. That nothing can solve everything and your mother is obvious. But solutions which can handle majority, or even 99.99% of relevant cases of your domain, I think we already have that today for structures. XML for example was optimized for decades to handle structures. Not all parts are good for the majority of cases, but the general ideas and concepts are universal enough that flavors of them will always appear when tackling such problems.
So instead of build yet another new solution, maybe just look at the existing tools and how we can make them more useable for your usecases.
> PS I have been using vim for quite a few years, and I don't see it as some kind of complex editing (other than bindings and modality which takes getting used to). Maybe I am misunderstanding what you mean by this exactly, though.
Normal editing today is: you press a key, something happens, that's all. Sometimes you add a modifier, but that's mostly it. Vim on the other side, you have modes, you have parameters for key presses, you compose commands to create new commands on the fly. That all is several steps more complex than just a simple key press. Vi-Input is a whole language in itself, highly complex on a cryptic level, all just to macro-manage micro-tasks.
> So instead of build yet another new solution, maybe just look at the existing tools and how we can make them more useable for your usecases.
Existing solutions don't integrate within an environment, and not within each other.
> XML
I go into much trouble to explain my position on general data structures in the article I have linked previously. You can't, in general, really optimize general data structures for specialized use-cases. That's why they are called general!
You wouldn't use a map where you could use a vector, right? But a map would still do the trick, no?
If you start doing all that, you are then trying to fit logs through a meatgrinder. You then have bloat, inefficiency, and all that other good stuff. I mean, you see, this is exactly why we have so many problems right now: people trying to use a general solution for everything. But all those solutions aren't good enough for one reason: they can't specialize. And you can't make them so. (That's the whole point of the topic article, too, just applied to strings.) And they can't interoperate.
> even 99.99% of relevant cases of your domain
Currently, that's more like 50% and the rest is unattainable without access and the knowledge of structure. And the first 50% doesn't even work that well either. I list a few applications on the website, so, don't take this as a bunch of empty claims: those can't be done efficiently (or practically) via the general means.
> Normal editing today is: you press a key, something happens, that's all. Sometimes you add a modifier, but that's mostly it. Vim on the other side, you have modes, you have parameters for key presses, you compose commands to create new commands on the fly. That all is several steps more complex than just a simple key press. Vi-Input is a whole language in itself, highly complex on a cryptic level, all just to macro-manage micro-tasks.
It may be complex in comparison to notepad, sure, but, I mean I haven't seen what's so cryptic about it, not really.
> Existing solutions don't integrate within an environment, and not within each other.
Integration is surface, the interface to the user. Not the concepts or libs.
> I go into much trouble to explain my position on general data structures in the article I have linked previously.
Sorry, but not reading that, not after the first article. Too long, not enough meat, too unfocused. You should write shorter articles and stick to the topic when you want to reach people. Life is too short and full of content.
> If you start doing all that, you are then trying to fit logs through a meatgrinder. You then have bloat, inefficiency, and all that other good stuff.
And if you don't work with a solid foundation, you will start accumulating bloat of small pieces.
> I mean, you see, this is exactly why we have so many problems right now: people trying to use a general solution for everything.
That's wrong. Problems will always exist. Everything is a compromise of tradeoffs. It's not possible to have no problems. Well, except to no do it at all. The main reason why people tend to use more generalized solutions, is because it allows them to move faster, and specialize later if necessary.
Structures are also such a compromise. It enables you to handle more and press it into a form, but you will lose liberty to use forms outside your structure. Understanding and managing the tradeoffs is relevant here. There is a reason why we have types in the first place, and not write everything directly in memory like on a canvas. Similar there is a reason why we not have a thousand different types doing slightly the same thing, but use inheritance to build specialized types from a handful of basic types. XML is such a basic type, for tree-structures. And it is a good starting-point to build a foundation for structured editing. It is not the only type, but at the moment, the problem is, that there is not even this.
> It may be complex in comparison to notepad, sure, but,
Notepad is not an advanced editor.
Is there even any editor which has a more complex editing than the vi-family?
> I mean I haven't seen what's so cryptic about it, not really.
Then maybe you just don't understand the problems tackled here at all.
> Integration is surface, the interface to the user. Not the concepts or libs.
That's the kind of viewpoint you get after years of using Unix or Windows.
> Sorry, but not reading that, not after the first article.
Ok. That one is not a rant, but whatever, you are arguing against its contents, and I don't see why I should be repeating myself. Sorry it's too long.
> And if you don't work with a solid foundation, you will start accumulating bloat of small pieces.
Exactly: you need a solid foundation to give you all the flexibility & meta-flexibility that you need. Otherwise meet bloat.
> That's wrong. Problems will always exist. Everything is a compromise of tradeoffs
< XML is such a basic type, for tree-structures.
Hey, look, you keep arguing with the article you have decided not to read, and you are arguing with a strawman. I don't know what you think I am trying to do.
> Structures are also such a compromise. It enables you to handle more and press it into a form
You know what real flexibility means? It's the ability to decide when you want structure and when you don't. And this is the kind of structural editing I am proposing: you can simply recreate a string-based editor within it, for any of your elements. At the higher-level, it can be reused. No structure where none is required? Sure. I am all for it. But there's much more structure out there that meets the eye.
> Then maybe you just don't understand the problems tackled here at all.
Or maybe you don't? Because, explain your point better, please.
The idea is to sync changes in the text to a tree structure, then have all the structure manipulation functions built on top of it. See the gif here for a visual representation: https://github.com/nvim-treesitter/playground
I like the fact of make people's lives easier. It's a universal solution which may be used in many editors: can't argue against reuse.
But is it perfect? I don't think so. Whatever is responsible for structural analysis should also be responsible for editing and all the extension capabilities, if you are to have a truly powerful system.
For me, the real problem lies in the integration with the client environment. Tree-sitter is fundamentally separate from the client application. That places questions on the introspection capabilities.
I mean, suppose your comment has structure. Suppose it's a markdown document, that comment in your code. And suppose you want to edit it that way: as a markdown document (or anything more complex). So you may want a markdown parser for that. Suppose you can write that parser in tree-sitter. Suppose you can even do structural-embedding. But how are you going to control and customize it? How are you going to write complex interaction policies between different structures? Embedding is where such seperation starts running into limits.
Why does this happen? I believe it's simple: APIs aren't enough. What you want is for the user to have access to the building blocks of whatever is being manipulated. And, so, keeping those blocks away from the user is never going to be that.
What's more, tree-sitter is still not really structural. That means it can't handle ambiguity well. For instance, in a lisp expression, you can insert a stray parenthesis. Paredit or a tree-sitter solution is going to tell you that your structure is malformed. A truly powerful structural editor is going to simply assume that parenthesis doesn't disrupt the structure around it, and so, it will simple keep track of it, and maybe it will find a match within the same expression at some later point in time.
Yes: a structural lisp editor doesn't have to care about a stray parenthesis. It just keeps keeping everything around it intact. It's a local ambiguity.
And, at last, the very important idea of structural editing is not even editing: it's the ability to treat structures as objects. It's the ability to present objects using a textual interface. Suppose you have a table structure, multiple layers of it. You need direct access to all that. Editing it should be easy, but it's really secondary.
Tree sitter doesn't deal with this.
I still see tree-sitter as useful, because you still need to deal with all the plain-text files out there. Perhaps, it could be used to import plain-text into a structural-editor representation, and then it may be, again, used on small subexpressions to identify their nature, incrementally, at runtime (but not the whole document, like what a usual incremental parser would do). So, that's the value of it as a parsing tool.
So, yeah, it's a useful tool, but when it comes to some advanced points of power-use and extension, it's not really enough.
Thanks for the question, it's thought provoking in many ways.
Do you think managing CL packages in a way similar to 1968's Grail (https://www.youtube.com/watch?v=2Cq8S3jzJiQ) could work? I'm thinking of a visual overview of packages were a user can connect packages together (maybe by drawing lines or by holding a key down and clicking on packages to link them), and can afterwards click on the vertices connecting the nodes to configure which symbols get imported.
It might be a bit UML-ish, but that's probably because I spent a bit too much time playing around with Umbrello.
Man, I love these old demos. And I haven't seen this one before. I now understand much better what Alan Kay meant when he said: I felt like sticking my hands right through the display and actually touching the information structures directly. This is the first system I have used, and practically the only one since that I would call truly intimate. [1] Thanks for this link.
Well, in terms of UI, I see no problem whatsoever in building such a workflow. The way it will work is: lenses (aka editors) are cells (aka widgets). So you will be able to position things freely, anywhere. You could even connect and see actual symbols get imported and such.
As for the CL side, unimporting a symbol (aka removing a connection) is possible, so I see no problem there either : )
Basically, when you say 'structural editing', do you mean making a a parse tree for every kind of input and having a modal command language that permits traversal and editing of that parse tree. Like what sapling https://github.com/kneasle/sapling is attempting to do ?
Nope. Parsing isn't even necessary for most things I want to do. Moreover, I don't really care for trees: any objects, any structures, any source can be used. There will be a common interface for it (like jump to the next semantic unit or search a lisp object). But any object can have it's own programmatic interface, and not all objects need to implement all of the interface (e.g. a read-only object).
I have expounded the point about structural editing (tangentially) on some other comments here, in this top-level thread (unless you want to read the article [1]).
In fact, I think every output should have its own buffer (or a place to end up in).
On the one hand, this is a UI problem, so that's just how I would prefer it. When you have a lot of output (like thousands of lines in a batch), you really don't want to go looking for where that thing begins.
On the other hand, you can't easily access the latest execution run programmatically. So, then, the input part shouldn't be there as well.
That doesn't mean you couldn't select a run of input-output dialogue, though. Visually, the run of UI input-outputs could be the same, it's just that how you work with all that stuff that has to be structured.
> First of all, I don't want another Emacs rewrite, much less in Guile. Mixing languages is not good for power-use, which requires ease-of-use, or at least conceptual simplicity.
This is a huge problem with GNU. Lisp is an extremely elitist language, and extremely divisive. It's not taught that well in GNU documentation either, in my opinion.
The least elitist language today is probably JavaScript. We already had Atom, and then things like VS Code, that make heavy use of JavaScript.
But does using JavaScript as the scripting language imply that the editor must be Electron-based? Surely you can have a thinner JavaScript-based app for desktop? And who cares whether it runs in a browser.
First of all, I don't want another Emacs rewrite, much less in Guile. Mixing languages is not good for power-use, which requires ease-of-use, or at least conceptual simplicity. I talk more about it in the article in the Project's Philosophy/Homogeneity section in [1] The Power of Structure.
I am proposing we need to really start considering a different paradigm, and attempting to do it right, and that's structural editing. People are wondering if it's possible to be writing better structural editors. We all know there have been attempts to do those, and, well, lo and behold, those were janky too.
But they don't have to be. When people start thinking of structure-editing, they immediately jump to the "how do we do C++". Well, in fact, I could tell you how we could do exactly that: you could start small. You start with what you know. And you know that you could, say, start with structuralizing the {} brackets. That's a semantic unit. So, that's a start. Even without getting down to the compiler level.
But I am not arguing I am about to do wonders for something as complex as some mainstream langauge in terms of a structural editing. I believe it can certainly be attempted, though, and certainly improved, peacemeal. And what you can't do: mix it with the traditional string-based editing. Or take python: that one would structuralize pretty nicely by indentation. Would it accomplish everything? No. But it would certainly help.
The gist of it boils down to the fact that you don't need to start at that very complex level, you can do things piecemeal and still get many benefits. Ask yourself this: can you edit a /list/ structurally, i.e. edit like in a string-based editor while maintaning an actual list behind the scenes? Sure, you can. A tree? Absolutely. Look at Paredit.
THERE'S NO REASON FOR THAT TO BE JANKY. NO reason why that wouldn't work.
It can absolutely be done.
And, really, structural editing like I am proposing /subsumes/ string-based editing, because you can just write a specialized editor for general strings, and use that for things you don't know how to structure yet. And yet, even those string-based editors can be specialized further, as some semantic units like words, expressions and even characters are often immediately apparent.
What does that give us? At the very least: object identity and programmatic access, and having the ability to pick your own data structure.
This kind of small things are what's actually going to be very useful for stuff like note-takers and computational notebooks and REPLs and what not. We don't need to start with programming languages (though I am going to do a Common Lisp IDE).
Please, ask me anything! Let's talk!
PS Another very important point is that ambiguity can be localized. Look at the Alchemy section in [1] where I discuss dealing with the reader (but I also talk about it in Rune).
PPS And thank you for posting this. It's exciting to be reading comments. Truly.
[1] https://project-mage.org/the-power-of-structure