The response by Isaac filled me with rage as soon as I got to 'No one actually writes code like this'. The guy provides a real example from his application, describes how alternative solutions stack up, and how he'd like to solve the problem, and gets called out for presenting a 'strawman'. Sigh, newsgroups...
I personally share the original author's perspective on this and was disappointed to see more sophisticated methods of writing concurrent code (futures, coroutines, etc) dropped from Node in favor of raw callbacks. In my experience, chaining raw callbacks is too easy to get wrong for it to be considered a reliable way of writing all your asynchronous logic, even if it is a great primitive to build a framework on top of. In particular, the effort involved in propagating exceptions all the way through a callback chain is rather significant, and making a mistake in a single function undermines the reliability of the entire chain (usually resulting in work silently being dropped on the floor). I wish more of an effort was made to address this problem.
They're still riddled with boilerplate (anonymous functions) and are quite horrible to read and write (bracket nesting).
I sometimes wish node.js (and js in general) would adopt co-routines. They'd seem to be the perfect middle-ground here; async code that looks almost like threaded-code, and no more callback-hell.
I wholeheartedly agree with you that the comment "No one actually writes code like this" was inaccurate and not useful for the discussion.
I see people doing this exact same style of code in EventMachine in ruby, and have done for years. I am continually having to advise newcomers to the evented space that they need to write state machines inside objects to maintain state and process flow, and advising them to wrap callbacks into message sends to these process containers.
There is nothing lacking in javascript to do this (create state and function containers), just as there is no lack of OO in ruby. Despite this, newcomers to evented programming always do the same thing. They revert back to a primitive style of programming, writing very long, very tab-deep functions. This is bad practice, you don't do it in an imperative synchronous environment (after some experience), and you shouldn't do it in an async environment.
You do not need to use closures for state passing. You shouldn't use closures as general state containers. They're even worse than using stack frames, as they're even less easy to debug (at least walking stack frames is easy with a decent debugger). They're really useful for the minor things, but to encode a complex process, they're completely inappropriate for reasons already discussed (error handling is an ever so common one in the network space).
I think this may be what the person was trying to get at with their comment, but they completely lacked substance, which was no use to the OP.
Everyone seems to end up discussing some new async-useful algorithms or datastructure that will "save the day", but in reality, until you start encoding application logic processes into decoupled, testable containers, you're always going to fight some complexity. This really has nothing to do with async, the principles of encapsulation and coupling are absolutely well discussed and documented, and merely apply even more vehemently to the async space. The unfortunate fact is that many of the users of these libraries do not factor code in this manner, and as such this discussion comes up again and again.
I honestly don't know of any good open source examples of complex async applications in either language (ruby or javascript) that are well structured. I can point you time and time and time again at libraries that are poorly constructed based on the "trivia" examples that "don't require" any structure and so just use a single object/scope and some closures to achieve a minimal use case.
There are many possible reasons for this lack of good examples. The reason I haven't written one is because it still takes some care, mostly in architecture, and no one will pay me to write such open source. There probably are some good examples in C, based on libev, libevent, glibs loop, etc, etc. I'm not intimately familiar enough with any example code bases to be able to call them out off the top of my head.
write state machines inside objects to maintain state and process flow, and advising them to wrap callbacks into message sends to these process containers
Any chance you could give an example of this refactoring / change of perspective (or link to a blog post, etc)? I can almost imagine what you mean, but it sounds like useful enough advice that I'd really like to grok it, and a concrete example would help a lot.
I'm currently working on another approach to reduce the complexity and error-proneness of evented programming. I'm working on a combinator library to abstract and automate the more common patterns of wiring up callbacks and propagating exceptions. Have you seen any success (or pitfalls) from this kind of approach?
I honestly don't know of any good open source examples of complex async applications in either language (ruby or javascript) that are well structured.
So, at first you claim there is clean way to do it, but then you admit nobody is doing it the clean way?
That sounds a bit strange, doesn't it?
I think the reality is that there simply is no pattern to escape callback-hell in complex evented applications. Or why else would everybody still be writing the same spaghetti-code with the logic scattered over at least two
callbacks (success/errBack) per unit of work?
I'd go as far and claim the spaghetti-pattern is inherent to evented programming, unless you wrap it all up in something sane like co-routines.
I think what raggi said is that there are no good _open source_ _complex_ async _apps_, mostly because when people are writing open source they tend to write libs, not apps, or end up not working anymore at the problem before it gets really complex.
I've worked on a commercial, complex, async codebase with raggi and have to agree that there is a way other than callback-hell, even for complex evented applications.
In my personal opinion, even more so on ruby than javascript, but I don't want to get to that really.
edit: changed asterisks to underscores because I didn't know they'd be italicized
the reason it hasn't surfaced as a best practice yet is probably because best practices for async programming haven't been a hot topic yet either.
and the point is exactly that a snippet doesn't fit the bill. its easy to abstract complexity on a snippet. its only with the full fledged solution that you start to see best practices emerging and making a difference.
Back in 2002 there was a lot of buzz about 'single-process' web servers like lighthttpd, but when I evaluated the open source products that existed at the time, I found that all of them had problems with reliability and data corruption. Matters are probably better today, but it underlines that it's much harder to make async programs correct than to make programs that use blocking I/O correctly.
I've spent some time building RIAs that are a bit more complex than what you usually see in the web using Javascript, GWT, Silverlight and other environments that all use async comm. The reason you don't see RIAs this complex often is that nobody knows how to write them.
A key architectural principle is that GUI elements need to register their interest in changing data... If you try to hand-code the paths that feed information in the GUI elements, you'll eventually reach the point where all hell breaks loose when you make a little change.
Another principle is to design your communication protocol around the application rather than the other way around. Programming in a single address space, the common practice is to build complex functions by composing simpler functions. That just doesn't work in distributed systems, not if you care about performance or reliability. If you can write one RPC call that does the work of 20 RPC calls, you eliminate a lot of complexity in your app and typically speed it up dramatically.
The new C# 5.0 proposes to bring asynchronous keywords, which I believe works like macros to turn imperative code into continuations.
The “await” operator used twice in
that method does not mean “this method now blocks
the current thread until the asynchronous
operation returns”. That would be making
the asynchronous operation back into a
synchronous operation, which is precisely
what we are attempting to avoid. Rather,
it means the opposite of that; it means
“if the task we are awaiting has
not yet completed then sign up the rest of
this method as the continuation of that task,
and then return to your caller immediately;
the task will invoke the continuation when
it completes.”
async void ArchiveDocuments(List<Url> urls)
{
Task archive = null;
for(int i = 0; i < urls.Count; ++i)
{
var document = await FetchAsync(urls[i]);
if (archive != null)
await archive;
archive = ArchiveAsync(document);
}
}
Microsoft has been very lucky to have someone like Anders Hejlsberg, who has steadily brought a stream of features to an enterprise programming language.
Yes, what is true is the fact, that they are finally working on a better ways to support async-code in a more frictionless way... (yes, Hejlsberg, F# team and Async Team rock, as does Erik Maijer)
They are still not where they would like to be though i guess:
just check out what subtle issues you can run into by using this quite intuitive way of handling async resp. auto continuations:
`Bind` with named methods goes a long way to cleaning up nested callbacks. Here's a (quick+dirty; I know I'm missing some `do`'s in there) version of the same code in CoffeeScript:
_ =
fileMenu: ->
mainWindow.menu "File", (e,f) => this.onFile(e,f)
onFile: (err,file) ->
if err then throw err
file.openMenu (e,m) => this.onOpenMenu(e,m)
onOpenMenu: (err,menu) ->
if err then throw err
menu.item "Open", (e,i) => this.onOpen(e,i)
onOpen: (err,item) ->
if err then throw err
mainWindow.getChild type('Window'), (e,d) => this.onDialog(e,d)
onDialog: (err,dialog) ->
if err then throw err
#...
_.fileMenu()
( `=>` is CoffeeScript for a function bound to the current scope )
We've adopted a way that allows us to do basically this:
mainWindow.menu "File", (err, file)->
openMenu err, (err, menu)->
getItem err, "Open", (err, item)->
click err, item, (err)->
getChild err, getChildType("Window"), (err, dialog)->
if err? and err instanceof NoFileError
# do stuff
if err? and err instanceof InvalidClickError
# do other stuff
if err?
throw err
# do stuff with dialog if no err
Each function takes as first argument an error and as last a callback. If there's
an error, the function just passes the error up the callback stack until it gets
dealed with.
fun = (err, args..., callback)
You can't pull that off if your code is heavily object oriented though:
mainWindow.menu "File", (err, file)->
# If mainWindow.menu errors, file is undefined
# and the call to openMenu throws an error
file.openMenu err, (err, menu)->
# ...
Breaking your code/api up with modules instead into classes fixes that.
This is why I always find it sad that people are averse to learning about "abstract, impractical" concepts like monads. Monads are abstract, but they are far from impractical: they solve problems which you are already encountering, and even already identifying as problems.
The best solution would be to add monadic syntax to javascript. You would get CPS, coroutines, error propagation, early failure, and several other capabilities all using the same syntax. Shit like generators or coroutines are just ad-hoc special-cased versions.
I'm not convinced that monads give the best tooling story. If you have a nice syntax for passing the operations into the monad, that means there's going to be quite an aggressive transformation of the source syntax; and since the control flow - i.e. the calling of the functions passed into the monad - is determined by the monad, it's harder to analyse at a higher level.
If you have a powerful type system that's open enough to encode further guarantees in it, so that the library itself can supply higher-level proofs of properties (if not analyses that can e.g. be queried and manipulated in an IDE context), that's great; but it doesn't help in an environment like Javascript, or even Java or C#. And I'm not sure it helps in larger scale situations, e.g. debugging multiprocess interaction. With so much done in libraries, you run the risk of tools not knowing enough about the program to help.
I guess what I'm saying is that the case for ad hoc special cases is quite strong outside of situations where all your programmers are highly trained and have deep knowledge of all domains intersecting with their problems.
I'm completely convinced that the growing ubiquity of asynchronous and multithreaded code will all drive us to embrace Object Oriented Programming 2.0 AKA Functional Programming. Can't come fast enough for me.
where foo includes a definition of bind and return (return should be renamed, there is already a return in js). Subcalls like baz(x) would be interpreted within the same monad.
Basically you would just make semicolon a function with dynamic extent, rather than a language builtin.
I don't know if it would be possible to monadize functions which use break, continue, or return. While and if would have to be special cased.
But wouldn't that require that all calls inside the with_monad block is of the same monad type? Wouldn't the syntax need a way of distinguishing between monadic and non-monadic operations (like using "let" rather than "<-" in Haskells do-syntax)?
Using async.js (http://eligrey.com/blog/post/pausing-javascript-with-async-j...), you can do away with callbacks and still have asynchronous code by using the yield statement. The following is an example (taken from my blog post) of asynchronous code using async.js:
yield to.request("/feedback", "POST", (
yield to.prompt("What are your impressions of async.js?")
));
yield to.inform("Thanks for your feedback!");
// do more stuff here
The equivalent code with callbacks is as follows:
async.prompt(
["What are your impressions of async.js?"],
function (response) {
async.request(
["feedback", "POST", response],
function () {
async.inform(
["Thanks for your feedback!"],
function () {
// do more stuff here
}
);
}
);
}
);
It makes it very easy to handle DOM events, as demonstrated below.
Since you wrote it: any idea what browser support of Javascript 1.7 is at? Wikipedia[1] currently thinks only Firefox has past 1.5, but I can't find anything else that isn't ancient (in web years), and Wikipedia's tables of things tend to get inaccurate pretty quickly.
Only Mozilla supports JavaScript (which is a mix of JavaScript-only features and ECMAScript; it's a little complicated), and everyone else implements ECMAScript. Some JavaScript features are in discussion of being added to ECMAScript, but no current ECMAScript engines (e.g. V8) have implemented them yet, barring the partial destructuring assignment implementation supported by Opera's Futhark ECMAScript engine.
There have been some different attempts to expand JavaScript syntax to make this style of programming more elegant.
Narrative JavaScript http://www.neilmix.com/narrativejs/doc/ has a single construct, the arrow which turns everything after the arrow into a closure which is passed as an argument.
The really-real problem that Node.js has is that once they admit that this is a good idea, huge swathes of the Node.js value proposition go flying out the window. If Javascript is not up to the task and you need a source rewriter or a really fancy, convoluted library to make non-trivial Node.js apps actually feasible to build, then... why are we in Javascript at all? Why not use a language and runtime where the support for this was baked in from day one and the compiler is the one doing all the hard work, as it should be?
This is why the Node.js hype was such a bad idea. (Don't miss the word "hype". Node.js was not a bad idea, except inasmuch as it encourages one into dangerous areas.) A useful, if limited, project built its community around a foundation of unkeepable promises. Truth is, the need to deal with this problem, and indeed far worse, the email in question should be considered an exemplar of the problem, just wait until you have three sequences of twice the size interacting with each other, is the inevitable outcome of any serious large project written in the manually-decomposed-cooperative-multiprocessing style, not an anomaly. Human brains are not meant to deal with this. It ceases to be a significant advantage over conventional threading pretty quickly.
Basically what this style comes down to is that instead of letting the runtime schedule your threads, and potentially create nondeterminism that kills your program, you manually schedule your threads instead, and instead choke the programmer on the geometric complexity of the scheduling problem. It makes beautiful small examples and demos, and makes easy things easy, but it makes hard things effectively impossible.
It runs in browsers and on V8, and it can access the CommonJS library ecosystem, and it's a nice language in a lot of other ways.
> This is why the Node.js hype was such a bad idea.
Except that the hype wasn't a bad idea. A lot of programmers are playing with something new and interesting, being exposed to new techniques and philosophies, and creating cool things. That sounds like a good idea to me.
> ... it makes hard things effectively impossible.
Those are very cool projects, but the main issue is that their source transformation is pretty aggressive. Until someone comes up with real, usable debugging facilities for those, they're not going to catch on.
I recently attended a presentation on StratifiedJS and it is really cool, but as you say they do some rather aggressive source transformation. You are basically writing in a different language and loading in an interpreter in JS to handle the new language. ObjectiveJ does the same thing. Take JS, extend it and run it through your own interpreter. It still sort-of looks like JS, but isn't.
Not only do you have the overhead of loading in an interpreter but you are also left without good debugging facilities.
CoffeeScript doesn't have this problem because it compiles down to readable JS, so if you're familiar with both CoffeeScript and JS you can just use native JS debugging tools to go at it. It does, however require the programmer to be familiar with both languages, so it's usually not the right choice for writing projects that you can hand off to another team at some later point.
The point about not being able to use 'normal' JS debugging facilities with StratifiedJS (or other 'higher-level' languages compiling to JS) is certainly true, but IMO it is not nearly as bad as it sounds.
Debugging highly asynchronous programs (whether written in 'straight' JS or in a higher-level language) with a 'normal' JS debugger isn't really much help: the callstacks that the debugger spits out in no way correlate to the asynchronous actions that are going on.
Having an abstraction that sits on top of JS, as it is the case with StratifiedJS, gives us the opportunity to write a debugger that spits out stack traces that actually reflect the true causal state of your program logic.
Granted, we don't have such a debugger yet, but we will in the not-too-distant future. What is already working is that StratifiedJS will amend exceptions with the correct linenumber and module name of the original StratifiedJS sources that threw the exception. So you don't have to manually correlate the generated source code with the original SJS source code, like you would need to do in systems that do a more or less 1-to-1 translation to JS.
Then there's jwacs (http://chumsley.org/jwacs/), which does CPS conversion on JavaScript without inventing weird new terms or requiring runtime libraries.
It would be nice if people stop saying 'async' when they mean 'callbacks'. Twisted's inlineCallbacks, Corotwine, Gevent... are all plenty async, and there's not an explicit callback in sight. (I suppose you could argue that the yield in inlineCallbacks and Monocle delimits a callback. You wouldn't be wrong.)
Of those I know gevent, which is indeed very nice, as it is 'implicitly' async. It has the performance benefits of being async, but doesn't require you to write callback-based macaroni network code.
Corotwine is exactly alike. Personally, I prefer inlineCallbacks. Explicit is better than implicit... Basically I can say, okay, so, here something slow is going to happen, and I'm willing to yield (hey look it even uses an appropriate keyword) control over to the reactor. My main point is still this: okay, yes, sometimes callback code can get ugly (with the apparent implicit assumption plain old "blocking" code is always pretty...), but that has nothing to do with the async backing.
Plain old blocking code is not always pretty, but it is much more clear what happens, and how errors are handled; all steps are in one place. Also, in my experience, it is usually much shorter.
Does anyone know if there are any plans for coroutines or similar mechanisms in the new versions of ecma/javascript? Python's async frameworks like monocle and diesel work around issues like that by returning another item from a generator and receiving the result again, which works quite well and makes the code look the same as the synchronous one.
I know Brendan Eich has mentioned that he would like to bring python-like generators into the language. They're in Mozilla's engine, were in ES4, and they're a ES Harmony strawman.
Otherwise, I'm not really aware of anything. I know there's been efforts around Promises but I think those have been restricted to libraries and I think that even that's stalled because the CommonJS list is split between Promises/A and Promises/B.
I've done web programming in Lua (w/ WSAPI); Lua is generally like Javascript's non-glue-huffing older brother, and coroutines/iterators remove a LOT of issues.
I was working on an event-loop-based web server+framework in Lua, but then I got into Erlang and realized how utterly lost the whole endeavor was. (But who wants to program in Erlang? I hear it's bad at strings, or something.)
I really like JS and it has excellent closure support, but it's really obvious that heavy async programming like we're doing nowadays with AJAX is not what it was designed for.
The 'nested callback' problem is one I've encountered over and over. I've figured out a relatively nice way to work around it. The solution goes something like this.
- Take the individual actions that you want and put them in closures.
- These closures take 1 argument, which is always a callback function that takes no arguments (this is the convention you structure everything around)
- Put these closures in an array, in the order that you want them executed
- Pass the array to a function that executes them for you (in Haskell it's called a sequence)
Some code on github up here, https://github.com/michiel/asynchelper-js - the sequencer.js function is verbose, if you read the next SO link I've included a three line example of implementing it.
And another one as a gist on github here, https://gist.github.com/604730 (for rendering a massive table from JSON data).
There's a lot more to this sequencer pattern. You can nest sequences, you can have the sequencer run each callback using setTimeout(callback, 0) to release control back to the main loop and get out of 'script is doing too much' popups, etc.
Need to make an async call? Make a closure that performs the async call and invoke the callback argument in your async callback.
Need to add some random JS code to a sequence? Wrap it in a closure and call the callback when you're done. If the random JS code requires an async call, see the previous paragraph.
Error handling, of course, is still awful. You can extend the sequencer convention to add error closures as a second parameter, you can do everything inside the closures in try/catch blocks and finally always call the callback, etc. But that just tends to make a mess of things. So the examples just assume everything is going fine.
It's not an ideal solution, but it's native JS and works everywhere.
It allows you to structure your code into functional blocks and work on them separately instead of making everything one big nested callback mess.
It's lets you create pseudo-threads, you pretend you're calling a sync function, and get a promise back, which you can call methods on (giving more promises) etc.
It needs a rewrite to allow objects in different pseudo-threads to interact - i.e. the threads to cross. Currently it's just a queue of actions to perform which doesn't allow this.
I can't help feeling that all of the examples contain too much copy and pasting. This is dealt with by taking a higher order approach to the problem. Which is what async.js is for.
doSomething = (c) ->
async.waterfall [
(c) -> mainWindow.menu "File", c
(file, c) -> file.openMenu c
(menu, c) -> menu.item "Open", c
(item, c) -> item.click c
(c) -> mainWindow.getChild type("Window"), c
(dialog, c) -> dialog.doSomething c null
]
It's not perfect, but I think it's pretty readable and succinct.
Perhaps what we need is something like CJSR (Common JavaScript Runtime). Languages that compile to JavaScript generate standardized debugging information as JSON (CoffeeScript has been pushing for this). These languages can then use a shared set of in-browser debugging tools (JSDB?). The fact that Safari, Chrome, and FF all support common browser extension architectures as this makes this solution even more compelling.
Agressive source transformation becomes less of an issue.
I tend to use Node for specific tasks like running a process that may take from 0 to 20 seconds without having a thread block. Using it for discrete tasks that it's suited for and then doing the bulk of my programming on regular vanilla synchronous server processes has worked for me as a way to manage complexity.
Until a standard model of async emerges I think programming large web apps on node will feel a little like hiking with a stone in your shoe.
There is a huge gap for a server side implementation of JS and node is not the one the masses are asking for, despite all the hype surrounding it.
Just take python, ruby or php, and make JS on the server on par with them. Forget about creating servers, async stuff and coroutines, 90% of moms and pops websites don't need them.
There is a huge momentum behind SSJS but it is all being wasted waiting for node to be the mother of all js apps. It will never be.
I felt the same way, but thankfully Javascript lets you write really cool things that handle the flow of your code in ways that correspond to the way you think of it.
let async_copy (in:Stream) (out:Stream) = asnyc {
let buf = Array.create 8192
let rec copy = async {
return! match in.ReadAsync(0,buf.Length,buf) with
| 0 -> ()
| x -> do! out.WriteAsnyc(0,x,buf);
return! copy; }
return! copy; }
There's an async copy routine, not much more overhead than a sync copy.
let sync_copy (in:Stream) (out:Stream) =
let buf = Array.create 8192
let rec copy =
match in.Read(0,buf.Length,buf) with
| 0 -> ()
| x -> out.Write(0,x,buf);
copy
copy
Is there a reason that events don't work in the situation? The async result handler fires an event 'getXFinished' and any function that needs to respond does so. When they are finished, they fire events. And additional functions are called. And so on.
I personally share the original author's perspective on this and was disappointed to see more sophisticated methods of writing concurrent code (futures, coroutines, etc) dropped from Node in favor of raw callbacks. In my experience, chaining raw callbacks is too easy to get wrong for it to be considered a reliable way of writing all your asynchronous logic, even if it is a great primitive to build a framework on top of. In particular, the effort involved in propagating exceptions all the way through a callback chain is rather significant, and making a mistake in a single function undermines the reliability of the entire chain (usually resulting in work silently being dropped on the floor). I wish more of an effort was made to address this problem.