> I want a better Erlang than Erlang, but talking yourself into how Go is already better than Erlang isn't going to get you there.
I don't mean to go against this point--it's very true, and I think it's important--but I think this conversation (and most comparisons of Go with Erlang) miss something: Erlang is a platform (it has its own VM, and basically its own OS, just relying on the outer OS as a hypervisor.) Meanwhile, Go isn't a platform, nor is it trying to be. Its designers (Rob Pike and Ken Thompson) already made the platform, first, a long time ago. It was called Unix.
These folks are serious about Unix--they want you to use it. They aren't going to reimplement Unix on top of Unix if they can help it. Unix is already the native set of abstractions they think in terms of[1]. An Erlang "release" (VM + source) should be compared to an entire Unix VM with a disk containing some Go binaries; not just a blob of Go source on its own.
Which is to say, the equivalent of the OTP exists for Go, but it isn't in Go--it's in Unix. OTP services are fundamentally "platform-level" things, and Unix provides them. Where are supervision trees? They're in upstart(8). Where is logging? rsyslog(8) will do it. And so forth.
There's nothing wrong with treating an individual Go process as equivalent to an Erlang process (other than overhead, but that's a problem with your Unix implementation, not with Unix as a platform). Make each Go process (that is, Go binary) have a single responsibility, so it can crash on its own. Then, give it a supervisor who can restart it with the right state. Some processes will still need to be larger, of course--goroutines still serve a purpose--but when the process crashes, you'll lose all of that, so don't put everything in there.
Since you now have multiple Go processes running, you'll need to do IPC. It's Unix: do it with sockets. What do you send on them? You can import a raw struct-specifier header file (or a whole client stub library, like in Erlang) from the include/ directory of each other process that specifies types it will understand, and then speak that "protocol" to it. Or you can use a ProtoBuf spec, to make your process more friendly for third-party use. Or, you can use plain text, like most Unix processes.
A single goroutine in each Go process should manage reads from this socket, deserialize the messages coming from it, and stuff them on channels relevant to their meaning. Then, the other parts of your process that care about external messages can receive on those channels when they wake up. Sounds a lot like an Erlang process inbox, doesn't it?
And so forth.
I think a lot of people are used to languages that provide their own insular inner-platform (Ruby, Python, Erlang, C#, Java, etc.) with its own implementation of everything from process scheduling to message passing to bytecode format to exception-handling, and think that Go is another one of these. Go is not this. Go is, despite all its modern trappings, "better C", and like C, it is heavily bound to Unix for most of the operations stuff that is important to high-availability et al.
Try Go with Unix--it might change your opinion on how many decades of hard-won experience Go is leaning on :)
---
[1] Well, okay, their abstraction-set probably hews closer to Plan 9 these days; Go even uses the term "runes" to refer to Unicode code-points and so forth. Still, all the ideas are backportable without too much of a fight.
> Meanwhile, Go isn't a platform, nor is it trying to be. Its designers (Rob Pike and Ken Thompson) already made the platform, first, a long time ago. It was called Unix.
This is great, and I think many of Go's advocates - including devs at Google - underplay it. I don't understand why, particularly given the pedigree of Pike and Thompson.
My approach to building large systems in Go is based around processing pipelines, and tries to be as UNIX-y as possible. The interface types in the io package particularly fit the everything-is-a-file model, where processes can do one thing well while having their inputs and outputs connected to pipes, files, sockets, named sockets, devices, etc.
In short, building complex systems with Go components has made me a better UNIX programmer, a level that I could never quite reach in C due to all the distractions of memory management and unsafety.
I have been playing around with Go on Windows- I really like it (especially the built-in concurrency types) but after reading these comments I feel like I would be better off trying it on UNIX (my mac will do). It probably does not help I am still somewhat lost on UNIX, either, but could you give an example of what a 'processing pipeline' approach would look like? That sounds very much like functional programming to me is that correct? Any advice on tackling UNIX and Go at the same time would be much appreciated!
A well-engineered UNIX-y "pipeline" is not unlike an impure functional program, yes. Well-behaved processes share no state (i.e. they don't contend over the same files), and messages and data passed over UNIX pipes are immutable, much like data in a functional program.
For example: one system I implemented needs to take a few hundred very large CSV files every day, aggregate and sort them, perform some complex processing on them, and output a result in a very different format to many different output files. It's an extremely complex system, and each component is in Go, performing a specific task, e.g.:
* combining the many source files
* cleaning the source files
* performing some aggregation on the stream
* splitting the stream into many parts that other components can read from in parallel ('named pipes' in unix make this very nice)
* splitting the stream into many output files
etc. Every component is dumb and does one thing, but a single controlling program is responsible for handling command line arguments that describe the overall outcome and setting up the stdin and stdout stream of all of the components to create the final result.
There's a beautiful simplicity to systems implemented like this, and it means you can take advantage of existing tools like grep, awk, sed, sort, cut, etc. to do a lot of the heavy lifting more reliably and quickly than you could probably implement yourself, while still coding the overall system at a reasonably high level of abstraction. Go doesn't lead to this approach directly, but it's very pleasant working with it as a citizen of this wider environment.
Fine. Then the UNIX platform is a great deal weaker than the Erlang platform on the supervision tree front, and if you'd like to fix it there, fine, but it's still not comparable as of this moment. Killing a Go OS process is not even remotely similar to killing an Erlang process, and restarting a Go OS process is not even remotely similar to restarting an Erlang process.
In fact, let me underline that... the UNIX platform is a GREAT deal weaker than the Erlang platform on this front. It has faint shadows of what Erlang supports, which are incredibly heavyweight, far less reliable, FAR less granular, and effectively can not be used the way Erlang's can be. It's an answer, sure, but it's not even in the same league in this particular way, so don't fool yourself otherwise.
Of course UNIX has other advantages, but, well, that's why I run Erlang on UNIX, so....
I don't mean to go against this point--it's very true, and I think it's important--but I think this conversation (and most comparisons of Go with Erlang) miss something: Erlang is a platform (it has its own VM, and basically its own OS, just relying on the outer OS as a hypervisor.) Meanwhile, Go isn't a platform, nor is it trying to be. Its designers (Rob Pike and Ken Thompson) already made the platform, first, a long time ago. It was called Unix.
These folks are serious about Unix--they want you to use it. They aren't going to reimplement Unix on top of Unix if they can help it. Unix is already the native set of abstractions they think in terms of[1]. An Erlang "release" (VM + source) should be compared to an entire Unix VM with a disk containing some Go binaries; not just a blob of Go source on its own.
Which is to say, the equivalent of the OTP exists for Go, but it isn't in Go--it's in Unix. OTP services are fundamentally "platform-level" things, and Unix provides them. Where are supervision trees? They're in upstart(8). Where is logging? rsyslog(8) will do it. And so forth.
There's nothing wrong with treating an individual Go process as equivalent to an Erlang process (other than overhead, but that's a problem with your Unix implementation, not with Unix as a platform). Make each Go process (that is, Go binary) have a single responsibility, so it can crash on its own. Then, give it a supervisor who can restart it with the right state. Some processes will still need to be larger, of course--goroutines still serve a purpose--but when the process crashes, you'll lose all of that, so don't put everything in there.
Since you now have multiple Go processes running, you'll need to do IPC. It's Unix: do it with sockets. What do you send on them? You can import a raw struct-specifier header file (or a whole client stub library, like in Erlang) from the include/ directory of each other process that specifies types it will understand, and then speak that "protocol" to it. Or you can use a ProtoBuf spec, to make your process more friendly for third-party use. Or, you can use plain text, like most Unix processes.
A single goroutine in each Go process should manage reads from this socket, deserialize the messages coming from it, and stuff them on channels relevant to their meaning. Then, the other parts of your process that care about external messages can receive on those channels when they wake up. Sounds a lot like an Erlang process inbox, doesn't it?
And so forth.
I think a lot of people are used to languages that provide their own insular inner-platform (Ruby, Python, Erlang, C#, Java, etc.) with its own implementation of everything from process scheduling to message passing to bytecode format to exception-handling, and think that Go is another one of these. Go is not this. Go is, despite all its modern trappings, "better C", and like C, it is heavily bound to Unix for most of the operations stuff that is important to high-availability et al.
Try Go with Unix--it might change your opinion on how many decades of hard-won experience Go is leaning on :)
---
[1] Well, okay, their abstraction-set probably hews closer to Plan 9 these days; Go even uses the term "runes" to refer to Unicode code-points and so forth. Still, all the ideas are backportable without too much of a fight.