Hi all, a quick warning: this is very rough and early, and not yet ready for production use.
What we're aiming for is Unix-style composition. Because libswarm services communicate over unidirectional message streams (similarly to how goroutines communicate over channels [1]), they can be combined in a way similar to unix pipelines. This makes composition much easier, and is an elegant solution to the problem of hardcoded dependencies in service-oriented architectures (for service A to combine service B a C, it needs to know specifically B and C. Instead what you want the logic of A to be decoupled and re-usable with another component D, without changing it. That's what unix-style composition allows).
TLDR: libswarm allows network services to be composed like unix commands. But it's still very early!
The author of Fleet have started contributing a Fleet adapter :)
libswarm exposes a standard interface to systems like Fleet, Mesos, Consul etc. It also has adapters for popular infrastructure providers, and for the Docker API itself. That way you can compose your architecture from standard building blocks, and change the building blocks over time without being locked in.
There was huge demand for this in the Docker system because there are so many orchestration tools being announced and competing for the attention of developers and sysadmins. Just at Dockercon, New Relic announced "Centurion", Spotify announced "Helios", Google announced "Kubernetes". Having a standard interface allows for these tools to continue competing without the risk of fragmentation and lock-in.
Of course the interface has to be thin enough to not get in the way. That's why we're focusing on keeping the codebase small, and the API as simple as possible.
fleet and etcd are orthogonal to the problems that libswarm is trying to solve.
libswarm has a future as a nice way of doing composition of networked services that are represented as objects in pipelines. The first demo is a `swarmd` daemon that exposes a docker API endpoint on one end and aggregates between a lot of "real" Docker endpoints.
The big open question is how this sort of composition works when the objects have different interfaces. So, how do we expose a "swarm" API that can be shared between different consistent data stores like etcd, zookeeper or consul. Or how do we encapsulate things like a runtime configured load balancer like nginx, apache2 or vulcand using a unified interface.
My hope for the project is to look at the interface model of Go or the DBus protocol and try to get consensus around the interfaces that make sense and hide complexity.
This is a layer "above" etcd/fleet/geard/shipyard/etc.
The idea is to standardize the way that administrators/users/developers "talk" to the various underlying containers. The intent is for it to have plugin support for etcd and fleet and all the other tools in that layer.
This alludes to the fact that you can do this within the same process, simply by running goroutines and communicating over Go channels. Later, when you actually need to scale, your services already support the interface necessary to be exposed over unix sockets, websocket or spdy/tls: you just need to change the plumbing.
1: We plan on porting libswarm to every major language.
2: The idea is to use socket passing, so you can use libswarm for service discovery. Place your postgres service behind a libswarm proxy, which implements the "connect" verb: that allows a client to use libswarm for discovery.
What we're aiming for is Unix-style composition. Because libswarm services communicate over unidirectional message streams (similarly to how goroutines communicate over channels [1]), they can be combined in a way similar to unix pipelines. This makes composition much easier, and is an elegant solution to the problem of hardcoded dependencies in service-oriented architectures (for service A to combine service B a C, it needs to know specifically B and C. Instead what you want the logic of A to be decoupled and re-usable with another component D, without changing it. That's what unix-style composition allows).
TLDR: libswarm allows network services to be composed like unix commands. But it's still very early!
[1] libswarm is built on a small library called libchan: https://github.com/docker/libchan