Rival node.js RPC project (dnode) author chiming in here.
This is pretty neat and I like the emphasis on getting stuff going in as few lines as possible but I'm not convinced that return values and implicit result callbacks are the way to go for asynchronous requests, which is how I understand this project to work. Often when you want to make a remote call you'll be doing some I/O on the server side, which in node is asynchronous so you can't return right away. Passing along a callback from the client side works much better when the I/O actions have some values to call the client back with.
The implicit callback is definitely my least favorite part in building NowJS. You can still pass in an additional callback to do any asynchronous I/O. We're trying to find ways to make this experience better for the next version
The solution I settled on for dnode is that a deep traversal finds all the functions in your argument list, pulls them out, and wraps them. This part of the protocol is a reusable module if you want to go with it:
It's written for continuation-passing style but it shouldn't be too hard to retrofit it for other call styles if you want to preserve explicit returns for instance.
I've just written JSON-RPC over SocketIO (for another project). I was going to open-source it. Is NowJS a superset of this? If so, I don't want to create noise by releasing a less-functional alternative.
Blending this with browserify + backbone + redis will be really easy to do. I think it will definitely simplify the dev of our app ; allowing us to share even more code between the client and server so that we can maximize code reuse.
On that note, (browserify author here) I would love to see projects like nowjs start to abuse the "browserify" field in the package.json for client distribution like dnode v0.6 does (presently in staging until socket.io-node v0.7 lands). With this field you can `require('pkgname')` in your browser-side code just by specifying 'pkgname' in your require list, even though pkgname might have both server and browser-side components.
Great stuff!!
So, do you open up 2 channels? One for the client to make calls to the server (this could just be normal HTTP) and one for the server to make client calls?
Depends. We're built on top of Socket.IO (http://socket.io/) which uses a websocket connection when possible. It falls back to assorted other transports if websockets are not available.
Nowjs seems to have a replication system baked in with harmony proxies so when you update data structures these changes get pushed out to the clients. I chose to have dnode just make copies but I do plan on writing a replication middleware at some point with the .use() functionality that landed in dnode v0.5 so I'm interested to see how well this replication approach turns out with nowjs.
Sure: dcom, corba, rpc, soap. The lesson I learned is that you can't abstract away the network barrier; to the contrary - that barrier must be the central point of the design of storage and communication components of the app.
I suspect that previous attempts haven't been successful for a number of reasons but especially because they tend to be synchronous. Node is a great platform for asynchronous rpc because the asynchronous character is already designed to make the programmer have to think about where the high-latency barriers are. I ran into this problem with drb in ruby for a job queuing system and had to write my own thread pool and polling system to compensate for it. Handing a callback off to your remote method does seem to work reasonably well at addressing these common rpc pitfalls.
"I suspect that previous attempts haven't been successful for a number of reasons but especially because they tend to be synchronous."
No. Node.js has no special contribution to asynchronicity, which has been in common use for something like 20 years now. Every UI-based program you have ever used is asynchronous-event based, even if it has synchronous components, to say nothing of the many many other Node.js-like libraries that have been written over the past decade in other environments, many with more capable languages that Javascript. None of these have been able to fix RPC, and many of the biggest and most well-funded attempts like CORBA and COM have come from smack in the middle of some of the largest piles of event-based code in the world.
The reason trying to pretend a network transaction is a function call fails has to do with the different semantics of a function call versus a network transaction. Latency differences even when everything is working correctly certainly are a factor, but are really less interesting than the problems of the unreliability of the network and the fact you are crossing a semantic boundary. Every network transaction can fail. Every network transaction might succeed, but with unacceptable levels of latency. Every network transaction might initially succeed but cutoff halfway through, or dribble its results in one byte at a time, or send you a gigabyte unexpectedly, or any of a variety of other failure cases you must at least be ready for, even if you can't "handle" (because in some cases there is no "handling" them). Furthermore, every network transaction incurs a serialization step, in which the semantics of the local program must be re-enforced on the data, possibly with failures thereto. For instance, you might get back JSON that specifies an integer greater than 5 billion in size where your environment is still only 32-bit; no local function call can do that. (Which isn't to say local function calls are therefore perfect, it is just that their failures lie in other places. For instance, your overlarge number got truncated somewhere else in your program, but it wasn't a function call that did it, it was some actual math computation somewhere. The point is that there is a different sort of semantic failure that can occur with an RPC vs a local function call and this inevitably leaks out of any abstraction you could try to wrap around the RPC.) And that was simply one tiny example, not the totality of the issues you can encounter, the vast majority of which are far more subtle than that.
RPC (where "procedure" is an old synonym for function) should by design not deal with these issues if it's really going to be "RPC", because by definition of RPC it really ought to look like a function call. Therefore it must handle all of these issues implicitly, and since no one answer is adequate for all cases, usually incorrectly. You can't seal over network issues anywhere near well enough to make dealing with a network as easy as dealing with a function. You must deal with latency issues, but again, these are ultimately the least interesting aspect. You must be able to deal with serialization and semantic issues; if your language forced you to deal with that on every function call you'd never use it. You must have some way of dealing with the various network failures and even throwing various appropriate exceptions only gets you a subset of the actions you might actually want. You must, inevitably, allow these things to poke through somewhere, at which point your are "configuring" your RPC call, at which point it is really no longer a "procedure call" at all, it's something else.
That it has historically imposed an additional point of synchronous behavior in some cases is because languages up to this point have also been largely synchronous, but that has nothing to do with RPC's far more fundamental failures as a network communication metaphor. It is also the case that if you've absorbed too much Node.js hype that you may underestimate the world's understanding of the problem; take a moment to search for "asynchronous COM", for instance. The first hit I get is an article from April of 2000. The synchronous problem is easily and trivially solved by turning RPC calls into futures objects instead, which has been done, and has not salvaged RPC because it is not the core problem RPC has. And there are other solutions, too, which also don't work.
Regrettably, calling things over a network must be more complicated than a local procedure call; all attempts to make it otherwise have indeed failed to live up to their promises.
I feel bad for you having type this well-reasoned argument only to have one person appreciate it (beside me, didn't need any convincing to start with).
Couple of things I'd like to add are versioning problems (on the net old and new versions of code talk to each other, which almost never happens in local calls), and time-travel problem where an operation succeeds, only to have all of its effects reversed because the remote server failed and was restored from a day old backup.
Interesting thing about rpc is that it is so very sexy, and it is tempting you to think that you will add error handling later. So it hides 99% of all problems from you under a sexy appearance.
n00b question here, but I installed it with npm and I'm trying to run the chat sample, but node says it can't find ../lib/nowServerLib.js
I'm still trying to figure out node so pardon my ignorance, but am I supposed to run this from a specific location? anyway, the project looks pretty cool, looking forward to playing with it.
In the typical node pattern, errors should be sent as part of the callback for asynchronous IO. For exceptions, a "throw" statement will not be automatically propagated to the other client, best practice is to catch the errors and call a client side function in that case. Exception handling will be improved in the future.
This is pretty neat and I like the emphasis on getting stuff going in as few lines as possible but I'm not convinced that return values and implicit result callbacks are the way to go for asynchronous requests, which is how I understand this project to work. Often when you want to make a remote call you'll be doing some I/O on the server side, which in node is asynchronous so you can't return right away. Passing along a callback from the client side works much better when the I/O actions have some values to call the client back with.