We've been waiting to announce the news for a while now. Can't tell you how happy we are to finally get it out.
We're tremendously excited about the potential we have with Eve and everyone we've talked to about it so far leaves with their head reeling. There's been an undercurrent brewing in the programming community that we're approaching a sort of transformational point in the way software ends up built. We hope to be part of that transformation.
I don't talk about it a ton in the post, but Eve isn't just about giving the Excel generation a seat at the table, but it's also explicitly designed to get us back to what we do best: solving problems. There are some pretty amazing properties that fall out of simplifying things as much as we have: concurrent by default, a real shot at auto-parallelization, a debugger that can tell you when, why, and how a value came to be, diffs not of code but of the output, the chance for true real-time collaboration on software, introspection into every part of the system, and so on. We'll be talking more about how it works over the next couple of months, but I can promise you now: there's some exciting stuff coming for all of us.
As always, in the interest of sharing with the community who helped us get here, I'm happy to answer any questions!
I've always felt that the tuple oriented view of the world of database was closer to what we should be doing instead of objects (or more lately functions?), although fp definitely gets you closer.
My one comment is to please make eve keyboard oriented from the start. I really don't want to use a mouse. I should be able to navigate every entity logically with a keyboard. For example, I should be able to hit a key to select available link endpoints and another to connect them.
I like visual, but I love keyboard oriented a lot more. Dragging and scrolling are way too overused ui interactions.
1. Powerful keyboard-based navigation/manipulation will be extremely helpful.
2. RDBMS-style tuples seem weird to me; why not objects? Quite natural to a layperson, IMO. If a "cell" can hold an object ({name: "Brown shoes", price: { amount: 254, currency: "USD"}} rather than just a scalar value like $2.54) a spreadsheet can be a lot more powerful.
Take a look at http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.113.... . Object databases, XML databases and network databases were around long before relational databases. The key problem the relational databases solved is not having to bake the access paths into your data-structure.
With a sufficiently rich type system (with composites and sets as domains), there's not a lot of difference when you are talking about pure data (instead of how the data structure relates to behavior) between tuples and objects.
Suggestion: if you want this to succeed, constrain your scope. Say No to touch-screen & voice! (IMO they are crippled interfaces compared to the keyboard)
If they want to reach a billion people then they're going to have to live in a world of touch screen (and maybe voice).
At NDC Oslo this year there was a keynote by Luke Wroblewski on user experience on the web. There's a video on Vimeo: http://vimeo.com/97305189
The key takeaway was that a significant percentage of people are using their smart phones with either only their thumb or their forefinger. Many people don't even have a computing device with a traditional keyboard and full-size screen. That talk got me thinking about this question: what would programming look like if you only use your thumb?
I don't think you can write off these interfaces, rather I think we may need to start to really think about separating creation of core components of software and then orchestrating the interaction of these components as a different aspect to developing software.
I've often wished I could program even a RAD proof of concept kind of thing on my phone. Many ideas come to me while I'm not at a computer (our desks are not always the most inspiring place we find ourselves) and as a consequence, are gone before I get back to work.
If you're wondering how programming with just your thumbs might work, look at Touch Develop from Microsoft Research.
It's worth noting that for programmers with some degree of RSI or a similar injury, a keyboard can potentially be a far less painful interface than a mouse or even touch.
What domain-specific editors are you planning to include in public release? Something for UI construction on the web?
Is it planned that users of Eve will routinely construct new or modify existing domain-specific editors to the task at hand?
Since first focus of Eve is on web apps (and it's hosted on javascript, correct?), will it cover both client and server applications?
Will there be an FFI?
If I remember correctly, you're planning to use datalog+time+constraints and plan to use materialized views extensively, is it so? Will ability to work with changes as a stream and aggregate over them in more interesting ways than "the last always wins" be builtin in Eve?
Have you tried to show a prototype to children and how do they respond in learning such system if so? What was the reaction of academic friends who were not too familiar with standard forms of programming (like you did with the first prototype of Aurora)?
If I'll ask more questions will it be all right and not too much trouble? :)
> Is it planned that users of Eve will routinely construct new or modify existing domain-specific editors to the task at hand?
Not sure at this point. For now we are focusing on the core interactions and we'll build out the rest once we've put tools in peoples hands and seen how they work.
> What domain-specific editors are you planning to include in public release? Something for UI construction on the web?
The table, rule and function editors for managing data and logic. A UI editor, hopefully with some constraint-based layout model. Various debugging/tracing/understanding tools.
> Since first focus of Eve is on web apps (and it's hosted on javascript, correct?), will it cover both client and server applications?
It will have to. Not sure yet how that will work. We can do websocket-style stuff at the moment but we will want to provide something simpler on top of that.
The editor and compiler are mostly written in Eve. The core language is tiny but very extensible - since execution order is not specified explicitly but determined from data-flow you can drop new logic in anywhere to add new compiler optimisations or new editor commands.
The language is also very data-centric - this week I was playing with programs that modify their own code by writing to the ast tables, triggering the incremental compiler to update the dataflow plan.
> If I remember correctly, you're planning to use datalog+time+constraints and plan to use materialized views extensively, is it so? Will ability to work with changes as a stream and aggregate over them in more interesting ways than "the last always wins" be builtin in Eve?
We're still trying out different models of time. The ideal semantic model is append-only, never forget. Implementing that on a real machine needs some restrictions to prevent exploding. Dedalus (http://db.cs.berkeley.edu/papers/datalog2011-dedalus.pdf) attaches epochs to each facts and only allows rules to query the current or last epoch. Edelweiss (http://db.cs.berkeley.edu/papers/vldb14-edelweiss.pdf) allows querying any epoch but uses static analysis to figure out when data can be safely deleted without changing the results.
> Have you tried to show a prototype to children and how do they respond in learning such system if so? What was the reaction of academic friends who were not too familiar with standard forms of programming (like you did with the first prototype of Aurora)?
No children yet. We test things on adult non-programmers regularly. That lead to a couple of surprising changes like getting rid of nesting / scoping (no matter what we did visually, people just couldn't figure it out).
> If I'll ask more questions will it be all right and not too much trouble? :)
It would be interesting to hear how much of EVE code has it taken to implement compiler or editor. And is there any interesting bottlenecks you would need to optimize out before release?
And how was the experience of writing a lot of EVE code? Do you feel yourself a lot more productive? Have you become accustomed to a new way of writing software very fast and does it feel natural to you now?
Regarding models of time. I'm a bit familiar with Dedalus and I've skimmed over Edelweiss. I'm interested in how would you use Edelweiss in non-distributed settings (I assume Eve is not targeted at that kind of applications right now). Would you try to use it to maintain transactional visibility during re-flows of data across views and afterwards discarding dead data?
I remember you're mentioning datomic in a podcast, so I think you'd want to keep dead data at least for some time, to have an ability to run time-travel queries later. Is this correct?
Would it be possible for developers to control when facts should be garbage collected? For example, CORFU log (which is a good implementation of event log abstraction with flexible views above it) allows clients to control trimming http://muratbuffalo.blogspot.ru/2014/09/paper-summary-tango-... (also I've had a good experience implementing underneath a similar interface for a queue system, it lead to a similar range compression optimization as in Edelweiss).
All the details you've already mentioned make me dream at night wishing a sooner release of Eve to experiment with it. I hope your next answers won't make me kidnap you and beg for an earlier access :)
> It would be interesting to hear how much of EVE code has it taken to implement compiler or editor.
The current iteration of the editor is 350 lines of our text-based Eve dsl plus 450 lines of helper js which will eventually become our UI and IO libraries.
Stratification is about 30 lines. There used to be more going on in the compiler but now that we use a constraint solver for all the query plans there isn't much left for it to do. http://p2.berkeley.intel-research.net/papers/EvitaRacedVLDB2... has some interesting ideas that we will be exploring later.
> And is there any interesting bottlenecks you would need to optimize out before release?
We've managed to get pretty surprisingly far with really dumb code. We are halfway through a big rewrite so the current build of the editor is using a runtime that is missing half of it's features. Each table is stored in a big array, we do sequential scans for constraint propagation, the incremental evaluation has not been ported yet and we don't have any Edelweiss gc. It can still recalculate and rerender in 20-40ms. That's too slow for fluid animation but plenty fast enough button clicks etc.
The lesson here is that computers are really fast and we squander most of that speed on unnecessary runtime abstraction. I'm pretty excited to see how fast it will get when we have real indexes and incremental evaluation.
> And how was the experience of writing a lot of EVE code? Do you feel yourself a lot more productive? Have you become accustomed to a new way of writing software very fast and does it feel natural to you now?
There are ups and downs.
The main upside I've noticed is how much time I normally have to spend worrying about where to put data, how to access data, how to keep data consistent and what order to do things in. In Eve you just put it in a table and let the runtime worry about the rest. I feel like I now spend a lot more time thinking about the actual problem instead of shepherding data around.
The main problem we're wrestling with at the moment is identity. Since the editor is built in the Edelweiss style as a pure function of the input events, we have to be able to generate stable, unique ids for anything that has a continuous identity. In the Bloom model we could just generate a random UUID and store it. At the moment we build ids by mashing together strings but I feel like the language should be helping us out here.
There are also a whole bunch of problems that come down to immaturity. We don't have a debugger yet so we rely on console dumps a lot. The compiler doesn't have much error checking so bugs in the editor tend to percolate a long way before surfacing symptoms. Implementing integrity constraints is pretty high up on my todo list right now.
> I'm interested in how would you use Edelweiss in non-distributed settings (I assume Eve is not targeted at that kind of applications right now).
There is a section in Out Of The Tarpit where the author argues that the ideal way to specify an application is as a purely functional view over all the inputs it has ever received. He then concedes that in practice you are forced to throw things away to avoid running out of memory. Reading the Edelweiss paper convinced me that we could program as if we were living in Mosely's ideal world and then separately specify a garbage collection strategy. It's just another instance of separating specification from optimisation which I believe is key to managing complexity.
This is probably the most risky part of our plan but we can always fall back to the Bloom model instead. The actual language implementation doesn't really vary between them.
I find the details you are giving here interesting, and quite promising.
> That lead to a couple of surprising changes like getting rid of nesting / scoping (no matter what we did visually, people just couldn't figure it out).
I worry about that -- I can see where its probably a huge gain for initial usability but my concern with it is that it shows signs of one of the big problems with spreadsheets in that they make it easy for non-programmers to start with something, and add complexity to it until not only they can't maintain it, but calling in a professional programmer doesn't make it maintainable either, and you essentially need to do a ground-up reanalysis and rewrite to get it into a state where it is maintainable with the complexity it has grown to.
Scoping is basically a way of mapping human-readable names to uniquely addressed code objects. When I say "foo" the compiler walks up the lexical scope until it finds a declaration for a variable called "foo" and replaces the human-readable name with the corresponding unique id.
Instead of having a set of rules to map the human-readable names, we generate unique ids for everything at edit time and tag them with the human-readable name. So when you type "foo", you get an intelligently sorted auto-complete box asking you which foo you mean.
This is one of the advantages of not directly editing the program text - we can insert all kinds of useful metadata directly into the AST.
> ...they make it easy for non-programmers to start with something, and add complexity to it until not only they can't maintain it...
Like most things, the problem can't be solved with technology alone. We can improve the tools to make it easier to organise code and to make it easier to understand other peoples code but we will also have to find ways to scale up the teaching aspect. It's the 'pit of success' - we have to design the experience so that people naturally end up doing the right thing. That includes building a community where people learn from each other and are exposed to better ways of structuring their thoughts.
> Instead of having a set of rules to map the human-readable names, we generate unique ids for everything at edit time and tag them with the human-readable name. So when you type "foo", you get an intelligently sorted auto-complete box asking you which foo you mean.
> This is one of the advantages of not directly editing the program text - we can insert all kinds of useful metadata directly into the AST.
I suppose I'll have to see how it works when there is something available to poke on, but that description doesn't allay my concern about it making a programming environment that makes writing easy, but reviewing, understanding, and maintaining previously-written programs hard in much the way that spreadsheets do.
If the information needed to disambiguate references is buried behind the scenes and not readily apparent in the primary interfaced used for editing, that seems problematic.
There is also much less stuff to disambiguate. The only names you ever refer to at the moment are tables. Most databases get by just fine with no namespacing at all.
As Jonathan Edwards says in his subtext work "names are too valuable to waste on talking to compilers." Making naming more of an interface concern rather than a part of the language is a great IDE and can greatly complement the way we use lots of search to write programs anyhow.
> Most databases get by just fine with no namespacing at all.
True, but most databases are very limited components of the program(s) that use them, which do use namespacing, not the complete platform for the program.
We also get to have arbitrary names rather than trying to stuff everything into single word. "People who disagree with me on the internet" is a valid name for an Eve table.
It's a valid name, albeit on that requires quoting, in pretty much any SQL based database, too. A lot of organizations have coding standards that prohibit names requiring quotes, and/or long names, but most databases support them just fine.
That said, I'm quite interested in getting a hold of eve.
Liked what I saw with Juno[1] (Julia IDE based on Light Table) and hope sometime after the big Eve reveal next year that Juno can easily migrate. The Juno team clearly felt that Light Table was one of the better and more extensible IDE platforms for Julia to work within.
This is as good a place as any to ask, but why aren't you piggybacking on the work that's being done in the Clojure community on these issues?
I'm working on something similar but much more constrained myself [1], but initiatives like Brandom Bloom's EClj and Reid McKenzy's KISS are already on their way to building the foundation for what you need with Eve but within the Clojure format/community.
Taking that route would also give you a natural path to keep LightTable alive and going and eventually get it bootstrapping/self-hosting.
Programming suffers heavily from path dependencies. There are lots of suboptimal things that we are stuck with because of all the other suboptimal things. We are trying for a clean break.
We're tremendously excited about the potential we have with Eve and everyone we've talked to about it so far leaves with their head reeling. There's been an undercurrent brewing in the programming community that we're approaching a sort of transformational point in the way software ends up built. We hope to be part of that transformation.
I don't talk about it a ton in the post, but Eve isn't just about giving the Excel generation a seat at the table, but it's also explicitly designed to get us back to what we do best: solving problems. There are some pretty amazing properties that fall out of simplifying things as much as we have: concurrent by default, a real shot at auto-parallelization, a debugger that can tell you when, why, and how a value came to be, diffs not of code but of the output, the chance for true real-time collaboration on software, introspection into every part of the system, and so on. We'll be talking more about how it works over the next couple of months, but I can promise you now: there's some exciting stuff coming for all of us.
As always, in the interest of sharing with the community who helped us get here, I'm happy to answer any questions!