Thanks for your interest in my old article! That version is a draft update of this article: Brad A. Myers. "Visual Programming, Programming by Example, and Program Visualization; A Taxonomy," Proceedings SIGCHI '86: Human Factors in Computing Systems. Boston, MA. April 13-17, 1986. pp. 59-66. http://www.cs.cmu.edu/~bam/papers/chi86vltax.pdf
By the way, something I always meant to ask you, Brad:
How does "C32" fit into your acronym theme of gemstones and rocks? Is it a teeny tiny 4x4x2 carbon atom block of diamond? How many carats would that be?
>Brad Myers wrote several articles in that book about his work on PERIDOT and GARNET, and he also developed C32:
>C32: CMU's Clever and Compelling Contribution to Computer Science in CommonLisp which is Customizable and Characterized by a Complete Coverage of Code and Contains a Cornucopia of Creative Constructs, because it Can Create Complex, Correct Constraints that are Constructed Clearly and Concretely, and Communicated using Columns of Cells, that are Constantly Calculated so they Change Continuously, and Cancel Confusion
>Spreadsheet-like system that allows constraints on objects to be specified by demonstration. Intelligent cut and paste. Implemented using Garnet.
That's a great acronym, but IMHO the acronym I came up with for the first iteration of FBP was also pretty good: DOORMAT (Data-Oriented Organization Running Multiple Asynchronous Tasks)! However, it was deemed to be lacking in gravitas!
C32 doesn't follow the general convention of my group of naming after gemstones or kinds of minerals, but I think it is fun. Very few of our systems strive for gravitas!
My transparently paned contribution of "GLASS" for "Graphical Layer And Server Simplifier" was technically a mineral, but not a very precious one. But it was great for making windows! Too bad it kept shattering because ugly X11 was on one side of it and beautiful Lisp was on the other. But at least you can cut "GLASS" to make fake gemstones of "PASTE".
I do try to follow the field. There are lots of modern VPs and some commercial successes. The key "large-scale" VPs are LabView from National Instruments, and OutSystems. There are a lot of what nowadays are called "No Code" or "Low Code" environments, that are mostly VPs as well - see https://en.wikipedia.org/wiki/No-code_development_platform
LabView seems constrained to the electronics/signals world. For that sort focus-area interface, Blender has a similar 'flow' graph based programming mechanism for its filters. I think this is common in digital audio authoring apps too. I was not aware of OutSystems, it seems like it's basically VB for webapps.
Hey Brad! Can you believe it's been 29 years since we worked together when I was visiting the Garnet group at CMU? Thank you for that wonderful opportunity, I really learned a lot from it. It's such a rare treat to actually be paid to write Lisp code!
About four years ago I mentioned Garnet on HN and linked to an article I wrote about it called "Constraints and Prototypes in Garnet and Laszlo", and your "All the Widgets" video, here:
>Yay Garnet! ;) I worked on Garnet with Brad Myers at CMU, on the PostScript printing driver. Brad is really into mineral acronyms, and I came up with an acronym he liked: "GLASS: Graphical Layer And Server Simplifier".
Constraints and Prototypes in Garnet and Laszlo (updated link)
Just yesterday I cited Richard Potter's "Just-in-Time Programming" paper in "Watch What I Do: Programming by Demonstration" that you co-edited, as being relevant to playing Factorio!
>Playing Factorio requires starting out by performing a few tasks by hand, then thinking about how to incrementally automate factories with trains, drones, combinator circuits, and blueprints, with high level modular repeatable patterns. And estimating where the break-even point is between completing a few tasks by hand, and spending the time to automate more common tasks.
>That evokes an article that Richard Potter wrote about "Just-in-Time Programming" in Alan Cypher's classic book (which is now online for free), "Watch What I Do: Programming by Demonstration", about when "the user attempts to write a program for a task that is already in progress":
>Watch What I Do: Programming by Demonstration. Edited by Allen Cypher. Co-edited by Daniel C. Halbert, David Kurlander, Henry Lieberman, David Maulsby, Brad A. Myers, and Alan Turransky. 1993. The MIT Press, Cambridge, Massachusetts, London, England.
>Chapter 27: Just-in-Time Programming. Richard Potter.
[...]
I was lucky to have the pleasure of working with Richard Potter at Ben Shneiderman's Human Computer Interaction Lab at the University of Maryland, while he was developing an early version of Triggers, which he also wrote about in that book:
Morgan Dixon and James Fogarty, who were working with James Landay at the University of Washington, also did some wonderful work along those lines on Prefab, which I've written about here:
Also just yesterday, I enjoyed watching Dan Ingalls' talk on "Pronto: Toward a Live Designer's Notebook", in which he demonstrated how he's been exploring Programming by Demonstration in Lively!
I really enjoyed this paper “A Taxonomy of Simulation Software: A work in progress” from Learning Technology Review by Kurt Schmucker at Apple. It covered many of my favorite systems.
It reminds me of the much more modern an comprehensive "Gadget Background Survey" that Chaim Gingold did at HARC, which includes Alan Kay's favorites, Rockey’s Boots and Robot Odyssey, and Chaim's amazing SimCity Reverse Diagrams and lots of great stuff I’d never seen before:
I've also been greatly inspired by the systems described in the classic books “Visual Programming” by Nan C Shu, and “Watch What I Do: Programming by Demonstration” edited by Alan Cypher.
Brad Myers wrote several articles in that book about his work on PERIDOT and GARNET, and he also developed C32:
C32: CMU's Clever and Compelling Contribution to Computer Science in CommonLisp which is Customizable and Characterized by a Complete Coverage of Code and Contains a Cornucopia of Creative Constructs, because it Can Create Complex, Correct Constraints that are Constructed Clearly and Concretely, and Communicated using Columns of Cells, that are Constantly Calculated so they Change Continuously, and Cancel Confusion
Danny Ingalls, one of the developers of Fabrik at Apple, explains:
"Probably the biggest difference between Fabrik and other wiring languages was that it obeyed modular time. There were no loops, only blocks in which time was instant, although a block might ’tick’ many times in its enclosing context. This meant that it was real data flow and could be compiled to normal languages like Smalltalk (and Pascal for Apple at the time). Although it also behaved bidirectionally (e.g. temp converter), a bidirectional diagram was really only a shorthand for two diagrams with different sources (this extended to multidirectionality as well)"
Does anyone feel we are still in the silent film era of computer programming? Will we ever advance from text editors into something else and leave Vim,
Emacs etc. forever behind for good?
The 80-column IBM punch card was invented in 1928 and here we are almost 100 years later writing programs as a sequence of 80-column lines of text. It seems pretty obvious to me that we're stuck in a local maximum. I don't think visual programming is the solution but there must be another alternative. Sort of like how databases are based on text but are much more than text.
The proper response to "640K ought to be enough for anybody" is virtual memory, not 1280K.
People used to complain about Macromedia Director only having 24 layers, so they increased it to 48. But then what are you supposed to do if you want to make a map of the United States: leave out Alaska and Hawaii?
Think about it: why are you writing a textual comment right now and not sending a picture, video, sound clip of your message ? That would be much less "silent film"-ish, no ?
If you ask me : because I can "scan" in 10 seconds a page of different replies, something which would be impossible to do visually or aurally.
This is why 10-minute videos of "how to" in programming always enrage me. Though of course it makes a lot of sense for visual tasks such as 3D animation.
Software is currently described by written text that is transformed by other process (compilers/interpreters) to a different state and then executed. Often the execution does not work as expected due to undefined conditions or states within the "mathematical language", or misunderstanding of the scope or memory allocation or release, etc.
There certainly could and should be visual representations of state, memory utilization, registers, variable values and execution sequence based on the textual conditions described. Contrasts of expected execution vs actual.
Perhaps using GPT-3 that is taught each programming language and then a designer can specify the behavior and GPT-X* can build a solution that can be formally verified as provable. Then generate test cases and show the flow and proof of correct execution based on test conditions.
> Often the execution does not work as expected due to undefined conditions or states within the "mathematical language", or misunderstanding of the scope or memory allocation or release, etc.
nah, 90%+ of bugs are already present in the "paper" specification in my experience, without any programming language involved (and I'm mostly writing in C++ which is prone to the issues you mention).
I do! I am currently in the very early stages of creating a VR-based language. I believe that if we can represent programs more intuitively to our spatial, visual, and tactile senses, we can unlock more programming potential in more people.
The language from scratch, yeah. I have experience with UE4, so I'm going to be using that. Too early for screenshots, I'm still in the planning phase.
Basically I'm spending my time on a whiteboard thinking up simple programs (fibonacci sequence, graph traversal, etc), then drawing them out how I imagine they could be represented in a 3d environment. From that, I am distilling ideas of how to represent data types and algorithms in a way that doesn't look like spaghetti.
I hope not. The 'advancement' from text editing to something else would in all likelihood not be an advancement at all but a retardation instead. There is a reason that the invention of writing is the basis of civilization. The only way for leaving text editing to be left behind is if all these big companies forbid ordinary users to do any programming besides a very dumbed-down/graphical version of it. I am sure they would very much like this but it would be a form of tyranny.
"Plain text" writing using a "text editor", is much more limited in its expressiveness than writing on a piece of paper. There is no reason to believe that a character grid somehow is the pinnacle of programming.
Cool stuff. Wish I found this a few years ago when I dug into this topic for a thesis. Back then I analyzed this incredible resource (warning: clicking this link will load looots of images): http://blog.interfacevision.com/design/design-visual-progarm...
and came up with my own provisional taxonomy where major categories were "Line-connected block-based", "Snap-together block-based", "List-based", "Enhanced text", and a few others.
Looking back I still think that the most promising direction when it comes to real-world applications of visual programming is the hybrid approach: make a language which has "real-time" interchangeable text and visual representations -- you get the best of both worlds this way (that was basically the topic of my thesis[1]).
I thought it was an original idea when I started out with my research, but sure enough there were attempts in the past. And towards the end (4+ years ago) I discovered a fresh one which seems to be still going strong, albeit rebranded: https://github.com/enso-org
Last time I checked they were making what I consider a bad design choice by leaving the structuring of the visual representation entirely up to the user, but maybe that's fixed now.
Anyway I think it's a cool topic and worth keeping an eye on.
@djder thanks for referring to us (Enso)! The topic you mentioned - leaving the graph structure (layout?) entirely up to the user - why do you feel it is a bad choice? In our experience (strongly influenced by VFX tools, like Sidefx Houdini), people are designing the 2D workspace similar to how you structure your code. For example, "on left top corner we will have database filtering", "on right bottom space we will have credit risk analysis". This way authors of workflows are able to navigate trough hierarchical graphs really, really fast, knowing where are things they have designed. With auto-generated layouts, the layout can change drastically when you apply even small semantic changes (like when you pass a new parameter between two nodes), and thus, it will break this mental space-mapping between nodes positions and their responsivities.
Anyway, we are building Enso in a completely open way, allowing third-party auto-layouting algorithms to manage the graph visual layout, and we are very open to investigate it further - that's why I asked this question :)
Soon after starting work on what later became Flow-Based Programming (FBP), I decided that I should leave the block layout entirely up to the application developer... A few years later, James Martin (1981) was working on what he called 4th-generation languages, and the blocks were prone to moving around unpredictably as the text was changed... Very confusing, as people do like things to stay where they are put! This actually fits in with the idea of "loci" alluded to elsewhere in this page. In FBP, "stepwise refinement" is a key part of the design process, so you get a hierarchy of simple structures... I agree that you could add a function to "prettify" your diagram(s), but this should still be under the developer's control. And if you make it simple to move blocks around (as in DrawFBP), it's probably not worth the trouble!
Paul, its such an honor to see your reply here! I'm a big fan of your work and your Flow-Based Programming book. It was one of the most important inspirations to what I'm doing in my life now (Enso: https://www.youtube.com/channel/UC4oMK7cL1ElfNR_OhS-YQAw). I truly love the arguments that you use there and how you describe how visual programming can unlock the communication between users of different backgrounds (business people, developers, etc).
To be honest, I was thinking about contacting you, Paul, for a long time already, however, I always felt like we are not yet in the stage of the development to bother you. Right now, when we are very close to Enso 2.0 public release, I would truly love to spend an hour with you on a call – I'll write an email to you about it and I hope you'll agree to chat with us! :)
Thanks for the kind words - it's a real boost when someone talks about how my work has influenced them! I would be happy to have a chat - are you in the UK? If so, we are 5 hours behind you, so we can plan a chat time accordingly... We could start the ball rolling by you pinging me at jpaulmorr@gmail.com ...
I took a look at your first (?) video, and IMO the human factors are amazing! That said, however, our areas of application seem a little different: at least your video is mostly talking about numbers, and I assume a mostly synchronous world view, whereas my work uses data objects ("Information Packets") travelling between asynchronously executing components... Given your orientation, it makes sense to be able to flip back and forth between visual and text representations - not sure it makes as much as sense in my environment! However, Chuck, IIRC, in one of the projects I worked on, built a network of around 200 processes, without drawing a single picture!
I see it as analogous to structuring source code text. The sweet spot IMO lies in the place where the editor offers guidance for common use-cases (think auto-indent, on-demand auto-format), but allows you to arrange code in a non-standard way when appropriate. That I think would be ideal.
Not all good programmers care about maintaining perfect visual order and I think that's fine. They would appreciate the automation and would be discouraged by the lack of it. Same as in text languages/IDEs.
Of course it's totally fine if your audience does not care about such features or they are not a priority.
But for a good general-purpose system I think that would be a must.
Now the sweet spot is not trivial to achieve and I haven't completely solved it, but I feel it can be done and I'd invest in researching it.
I fully agree that the visual environment should arrange the newly created graph nodes in a nice way automatically. In the Enso 2.0 version we have a very early support implemented exactly for that. However, there is a lot to be done yet to have it working fluently in the case of bigger and complex graphs. It is a fascinating research topic for me, TBH.
>The method of loci (loci being Latin for "places") is a strategy of memory enhancement which uses visualizations of familiar spatial environments in order to enhance the recall of information. The method of loci is also known as the memory journey, memory palace, or mind palace technique. This method is a mnemonic device adopted in ancient Roman and Greek rhetorical treatises (in the anonymous Rhetorica ad Herennium, Cicero's De Oratore, and Quintilian's Institutio Oratoria). Many memory contest champions report using this technique to recall faces, digits, and lists of words.
>The Method of Loci is an ancient technique that used to be taught for thousands of years as a standard part of a classical eduction, way back when people needed to remember things before the invention of smartphones and printing presses. But in the middle ages it was banned for being immoral! Apparently, some bad apples were abusing the Method of Loci to remember "immoral" things they shouldn't be thinking about, using "fabulous" images they shouldn't be imagining.
>>Remember to use physical objects in these palaces since they have easily imaginable traits; when you are dealing with more abstract or untranslatable ideas it is best to convert them into objects based on the way the words sound, so Valmur becomes Val Kilmer, Les Preuses becomes purses, etc. Additionally, you don’t need to be concerned with reality when making these memory palaces. The more slapstick, unique and vivid they are, the easier they will stick. Raunchy imagery always works well, to the point where some religious orders in the middle ages banned the practice because it was deemed immoral.
>Memory Palace techniques have been known as the Mind Palace, Method of Loci, and Memory Journey, Art of Memory, Ars Memorativa, Memorative Art, Mnemotechnics, Architectural Mnemonic, Graphical Mnemonic, and Textual Mnemonic.
Eric Krokos, Catherine Plaisant, and Amitabh Varshney published an interesting paper about memory palaces in virtual reality, called "Virtual memory palaces: immersion aids recall" in Virtual Reality volume 23, pages 1–15 (2019).
>I visualize and remember code that way. For me, it's hard to forget somewhere I've been, even if I only imagined being there.
>Each function is a little building like an office or a shop, which has a sign out front telling what services or products it sells, and contains everything inside you need to solve some kind of problem or produce some kind of product or service (where equipment in the room is like references to other objects and functions and imported libraries).
>You're standing behind the front counter, just about to receive a customer though the front entrance door with the parameters you need for one particular instance of that problem.
>You go into the back room, solve the problem, then deliver the results out the exit door at the back of the building (or through any of the other earlier emergency exits, if you had to exit prematurely or throw an error and run away).
>The front/back flow is a metaphor for the top/bottom flow of control through a function.
>>A Nassi–Shneiderman diagram (NSD) in computer programming is a graphical design representation for structured programming. This type of diagram was developed in 1972 by Isaac Nassi and Ben Shneiderman who were both graduate students at Stony Brook University. These diagrams are also called structograms, as they show a program's structures.
>If you squint you can see the example Nassi-Shneiderman diagram in that article as a map of a building, with its front at the top, and exit at the bottom.
>You can have internal hallways and rooms for branches and loops, like a Nassi-Shneiderman diagram. The "Sub to Determine Wiki-Article" room is like the front entrance lobby of a theater where buy your ticket. The "Select Favourite Genre" room is like the stage of The Price is Right, and you get to pick what's behind door #1 (History), #2 (Science), or # (Geography), or else choose Other. They each have one or two rooms behind them with your rewards, and then they all finally exit out to the same back stage loading dock, where you take your wonderful prize (or consolation donkey) home.
Ben explained that "Nassi-Shneiderman Diagrams have peaked out a couple of decades ago, but there were a huge success story for a 15-minute invention, with hundreds of software tools, dozens of textbooks, and 1000+ papers based on them."
He has a funny story about the incredibly harsh rejection letter he received when he and Ike Nassi submitted their first paper about "Nassi-Shneiderman Diagrams" to Communications of the ACM in 1972, that should serve as an "inspiration for anyone whose new ideas are rejected by some respected authorities".
>Our submission of the structured flowcharts to the Communications of the ACM was quickly rejected, on October 4, 1972, by the Programming Languages Editor David Gries of Cornell University. He included a single anonymous reference letter which is on paper that has a Cornell University watermark. I assume Gries gave our paper to one of his colleagues (you can play the guessing game too), who wrote the most brutal rejection letter I have ever gotten.
>The reviewer wrote: "I feel that the best thing the authors could do is collect all copies of this technical report and burn them, before anybody reads them." As graduate students, this stinging rejection shocked us, but we kept getting enthusiastic responses from people around us. We sent the paper to the unrefereed ACM SIGPLAN Notices, where it was published in August 1973. It didn't take long for others to produce extensions, software tools, and applications of structured flowcharts.
>The next problem was theft of the idea. I had sent a draft to respected colleagues, and soon others published slight variations. One of these respected colleagues was Ned Chapin, who greatly troubled us by publishing what he called 'Chapin Charts.' A friend of mine sent me his published paper with a note encouraging me to sue. For several years I feared that Chapin's reputation and his frequent professional seminars would wind up leaving the idea tied to his name, but as the decades passed, the ending has proved to be a happy one. We called the idea 'structured flowcharts, but they are widely known as Nassi-Shneiderman Diagrams.
>Another problem was the appearances of patents for variations on our idea, but these have not limited the widespread recognition we have gotten over the years.
>I wish every graduate student or young inventor would have the pleasure of seeing his/her ideas spread so far and influence the work of so many people. I also hope that the story of the bold rejection of our novel idea and its eventual international success, is an inspiration for anyone whose new ideas are rejected by some respected authorities.
>DonHopkins on July 13, 2019 | on: I was wrong about spreadsheets (2017)
>Google sheets (and other google docs) can be programmed in "serverless" JavaScript that runs in the cloud somewhere. It's hellishly slow making sheets API calls, though. Feels like some kind of remote procedure call. (Slower than driving Excel via OLE Automation even, and that's saying something!) Then it times out on a wall clock (not cpu time) limit, and breaks if you take too long.
>A CS grad student friend of mine was in a programming language class, and the instructor was lecturing about visual programming languages, and claimed that there weren't any widely used visual programming languages. (This was in the late 80's, but some people are still under the same impression.)
>He raised his hand and pointed out that spreadsheets qualified as visual programming languages, and were pretty darn common.
>They're quite visual and popular because of their 2D spatial nature, relative and absolute 2D addressing modes, declarative functions and constraints, visual presentation of live directly manipulatable data, fonts, text attributes, background and foreground colors, lines, patterns, etc. Some even support procedural scripting languages whose statements are written in columns of cells.
>Maybe "real programmers" would have accepted spreadsheets more readily had Lotus named their product "Lotus 012"? (But then normal people would have hated it!)
Excerpt from "Taxonomies of Visual Programming and Program Visualization", by Brad A Myers, 1990/3/1, Journal of Visual Languages & Computing, Volume 1, Issue 1, pages 97-123:
Spreadsheets, such as those in VisiCalc or Lotus 1-2-3, were designed to help nonprogrammers manage finances. Spreadsheets incorporate programming features and can be made to do general purpose calculations [71] and therefore qualify as a very-high level Visual Programming Language. Some of the reasons that spreadsheets are so popular are (from [43] and [1]):
1. the graphics on the screen use familiar, concrete, and visible representation which directly maps to the user's natural model of the data,
2. they are nonmodal and interpretive and therefore provide immediate feedback,
3. they supply aggregate and high-level operations,
4. they avoid the notion of variables (all data is visible),
5. the inner world of computation is suppressed,
6. each cell typically has a single value throughout the computation,
7. they are nondeclarative and typeless,
8. consistency is automatically maintained, and
9. the order of evaluation (flow of control) is entirely derived from the declared cell dependencies.
The first point differentiates spreadsheets from many other Visual Programming Languages including flowcharts which are graphical representations derived from textual (linear) languages. With spreadsheets, the original representation in graphical and there is no natural textual language.
Action Graphics [41] uses ideas from spreadsheets to try to make it easier to program graphical animations. The 'Forms' system [43] uses a more conventional spreadsheet format, but adds sub-sheets (to provide procedural abstraction) which can have an unbounded size (to handle arbitrary parameters).
A different style of system is SIL-ICON [49], which allows the user to construct 'iconic sentences' consisting of graphics arranged in a meaningful two-dimensional fashion, as shown in Figure 5. The SIL-ICON interpreter then parses the picture to determine what it means. The interpreter itself is generated from a description of the legal pictures, in the same way that conventional compilers can be generated from BNF descriptions of the grammar.
Visual Programming and Program Visualization are interesting areas that show
promise for improving the programming process, especially for non-programmers, but
more work needs to be done. The success of spreadsheets demonstrates that if we find
the appropriate paradigms, graphical techniques can revolutionize the way people
interact with computers.
>lmm 9 months ago | on: Maybe visual programming is the answer, maybe not
>If there was a visual programming language with anywhere near the popularity of Ruby, I'd be willing to consider that maybe the idea has some merit.
>DonHopkins 9 months ago [–]
>Excel.
>I could turn your argument around: If Ruby were anywhere near as popular, widely used, and successful as Excel, I'd be willing to consider that maybe the idea that Ruby is a viable programming language has some merit.
>But I won't, because whether or not something is a visual programming language isn't up to a popularity contest.
>Can you come up with a plausible definition of visual programming languages that excludes Excel, without being so hopelessly contrived and gerrymandered that it also arbitrarily excludes other visual programming languages?
[...] (TL;DR: he couldn't, since he was under the mistaken impression that Excel not programmable, and was less popular than Ruby...)
That thread was on an earlier discussion about a blog posting from 2020 about the same 1989 paper by Brad Myers that we're currently discussing.
>Maybe Visual Programming is The Answer. Maybe Not
>Whenever discussing problems with programming today and potential solutions, invariably someone will pop up and declare that the problem is obviously the fact that programs are linear text and if only programming were visual, all problems would immediately disappear in some unspecified way.
>I understand the attraction of visual programming, particularly for visual thinkers. However, it's not as if this hasn't been tried, with so far very limited success. Brad Myers, in his 1989 paper Taxonomies of Visual Programming gave, along with the titular taxonomy, a non-exhaustive summary of the problems, starting with visual languages in general:
