Except it's... not quite correct. getent performs NSS resolution, not DNS resolution, so its output depends on the value of the hosts field in your /etc/nsswitch.conf, which could be set to something fun like LDAP or something else.
I don't know Kubernetes, so it's very possible that Kubernetes enforces that name resolution is always DNS and only DNS (a "hosts: dns" line in /etc/nsswitch.conf, without any additional stuff like files which would mean resolution via /etc/hosts), in which case getent is indeed correct, but in the general case, there is no system-intrinsic tool that tests DNS only, and you have to use something like dig.
And if you look under the hood, you will see that it's only possible because of a bunch of ad-hoc hacks, which is exactly what my proposal was about: level the playing field and make it possible to have alternatives _without_ the ad-hoc hacks.
Not at all. I'm referring to the structure of the Alpine packaging, especially around boot scripts, that hardcodes busybox in a number of places and makes it difficult to package an alternative without workarounds - which accumulates tech debt.
"Avoiding spinning up new processes" is incorrect characterization of s6. Processes are not a scarce resource; spawning a process is not a costly operation in the context of process supervision, even on embedded systems. s6 focuses on optimizing some metrics that are indeed important to embedded systems, like RAM use and code path length, but "spinning up new processes" isn't one of these metrics.
It is not, and has never been, necessary to learn execline, the scripting language you're speaking of, in order to use s6. execline is used _internally_, and you can also use it in your own scripts if you so choose, but it is not a requirement.
"Sending a polite shutdown signal, waiting for some time, and sending a harsher shutdown signal" is a matter of exactly one command: s6-svc -d. That is precisely one of the benefits of s6 over other daemontools-style supervisors: it handles this sequence natively.
I welcome fact-based criticism of s6. I do not welcome FUD-based criticism.
You need to read the archives of the supervision mailing-list at list.skarnet.org, as well as http://skarnet.org/software/s6/overview.html - there is a lot of value in process supervision, it's just that most people who whip up their own solution end up doing it wrong. Like crypto, in a less dramatically noticeable way.
I agree with Jonathan on everything here, with the additional caveat that the Go runtime is particularly heavy, and thus not really suitable for low-level system software (which should use resources sparingly, leaving them for production applications). So I don't think Go is a suitable language choice for a supervisor in the first place; and apart from the language choice, all of Jonathan's criticism applys.
I strongly recommend the author of immortal to join the supervision mailing-list on skarnet.org, follow the discussions there and ask questions if needed. We have about 19 years of experience with process supervision, there has been a lot of prior art and successive refinements, and we hold extensive knowledge about what constitutes good practice. If someone aims to design a process supervisor, even if it's a completely new take with a specific spin, it would be a mistake not to tap into our pool of experience. We always want users to benefit from good, rigorous software.
Hi, so assuming immortal was created using the best in the class programming language for its purpose, and let's say was following all the existing best practices, I still have one question that never really found a proper solution, which is:
For cases in where is required to monitor services like unicorn/gunicorn/Nginx that demonize them self and have the capability to fork how to make aware the supervisor of does changes and to continue monitoring the new process without entering into a race condition, to be more clear here is an example of this problem https://asciinema.org/a/80371, and work around's here http://serverfault.com/a/587065/94862.
Type=
If set to forking, it is expected that the process configured with ExecStart= will call fork() as part of its start-up. The parent process is expected to exit when start-up is complete and all communication channels are set up. The child continues to run as the main daemon process. This is the behavior of traditional UNIX daemons. If this setting is used, it is recommended to also use the PIDFile= option, so that systemd can identify the main process of the daemon. systemd will proceed with starting follow-up units as soon as the parent process exits.
PIDFile=
Takes an absolute file name pointing to the PID file of this daemon. Use of this option is recommended for services where Type= is set to forking. systemd will read the PID of the main process of the daemon after start-up of the service. systemd will not write to the file configured here, although it will remove the file after the service has shut down if it still exists.
There is no good solution for servers that, mistakenly, background themselves; the practice of backgrounding itself comes from a time where supervision was not as widely known as it is today, and nowadays there is no excuse for a daemon to fail to at least provide an option to avoid backgrounding itself.
The best thing to do is to contact the daemon's authors and insist they change that. In nginx's case, they think it's ok if they perform their own supervision, but it would be nice if they also could integrate with existing supervision systems.
In the meantime, as a workaround, the accepted practice for supervising forking daemons is known as "fghack": maintaining a process keeping track of the forking daemons via a few pipes that close when the daemons die. s6, for instance, provides such a tool: https://skarnet.org/software/s6/s6-fghack.html . AFAIK, so does nosh.
Still in active development too. :) And no, Jonathan, I haven't heard anything from TrueOS people.
Which is a shame, because if a distribution is being built today, with no existing legacy service script base to cater to (or if it has to be converted anyway), going for anything else than s6 or nosh is staying in the past.
Honestly, the people who use Busybox are probably aware of the problems - political as well as technical - with systemd, and most likely not using systemd in the first place because it's not exactly a good choice for embedded systems.
Also, for busybox syslogd, it does not matter at all whether you can hold its socket. syslog isn't a reliable mechanism anyway, and busybox is light enough that the non-ready period is really short.
So, on the technical side, the impact is quite minimal.
On the political side, however, it looks like Denys made a splash, and I'm not going to complain about it. :)
The significant technical aspect is, as I mentioned, that now there's no mechanism to pass an already-open socket to the daemon. nosh has several service bundles supplied out of the box for providing syslog service. They provide the various combinations of two different syslog tools over /run/log, UDP port 514, and /run/systemd/journal/syslog. Each operates in the UCSPI(-like) way, opening the datagram socket specific to the service being run, dropping privileges (in the syslog-read case), and then invoking the daemon program. All sorts of fairly obvious (to those familiar with the daemontools way of doing things) consequences ensue, like separated streams for local clients and remote clients, and control of whether and whose remote client service is provided that is as straightforward as taking the individual services up and down.
Rainer Gerhards rsyslogd and the nosh toolset's own syslog-read both support this. The BusyBox syslogd used to be usable in this way, as well. udp-socket-listen and local-datagram-socket-listen have a --systemd-compatibility option that would have interoperated quite happily with the BusyBox code as it was.
But thanks to BusyBox syslogd now being less than it was before, the systemd compatibility won't work and there's no mechanism for this in BusyBox syslogd. In taking a sideswipe at systemd people, Denys Vlasenko has made BusyBox less operable with other systems that are not systemd. That's a shame, in my view.
> it's not exactly a good choice for embedded systems.
I don't develop embedded systems, but systemd is actually popular in that space because of its watchdog capabilities and its inclusion in projects like GenIVI and Tizen.
I see nothing there that can't be achieved by a simple supervision suite. s6, nosh, even runit provide those features; the extra complexity of systemd isn't needed at all.
If even embedded developers are getting pulled in by the sirens of systemd because OH MY GOSH IT HAS WATCHDOG CAPABILITIES, then we definitely need to work more on raising awareness about supervision.
You seem to dislike the idea of asking for notification support in the daemons themselves. Granted, it's impractical, but it's the only way to avoid race conditions which are rare for most services, but very real for a few programs that may take ages to get ready.
How would you address the problem of slow starters without support from the daemons, while keeping a "the dependency graph is always respected" invariant?
Right, if the problem is slow starters (say, RabbitMQ), then restoring from an image where initialization has already been complete would resolve it. For more complicated cases where there is a race with acquiring some resource, then a readiness notification mechanism could be supported as a niche feature. I'm still not convinced of it in the general case.
But how would you go with restoring such an image? It won't be instant, there will still be a window where the service manager is continuing its work but the image hasn't been restored yet. One way or the other, you need a synchronization mechanism, so how would you design such a mechanism without support from the daemon?
You don't need a complicated case to have a race with acquiring some resource. A server opening a socket is providing a resource; clients connecting to that socket are using it. Clients should not be started before the server is listening to the socket. How would you solve this case - are you advocating socket activation by the service manager? ;)
It's important to realize that "a replacement for systemd" is neither achievable nor desirable.
systemd has grown fast and conquered large market shares because it was backed up by a company, which put a lot of money and manpower into it - to write it, to integrate it, to advertise it. The only thing that has not been correctly funded in systemd is research and design.
systemd is only Open Source if you read the license; all its other aspects are proprietary - it's software made by a company and aiming to capture a market. It is impossible to compete with systemd on the same grounds, because no real Open Source developers will have as many resources as Red Hat.
And even if it was possible, the result of such an attempt would simply be another integrated behemoth, powered by money and marketing instead of good technical decisions. (Or even worse, powered by ideology - can you imagine a systemd-like controlled by the FSF?) In the end, the situation for the users would be even worse than it is today. You don't fight Goliath with Goliath; you don't fight Microsoft's hegemony by buying Apple products.
About interface compatibility: the author of the DnE article (vezzy-fnord) has written uselessd, and finally abandoned the project because the systemd interfaces are so tightly integrated with the systemd design in the first place that it's impossible to be compatible without simply being a systemd clone, which he did not want uselessd to be. No, interface compatibility is not an option, because it would simply acknowledge the validity and superiority of the systemd architecture, and nobody wants a copy of systemd.
I believe that the way to provide an alternative to systemd is to provide all the functionality that the systemd users like, but in a technically better, less integrated, more unixish way.
With s6, s6-linux-init and s6-rc, I now have respectively a process supervision system, a simple init process and a service manager, which should be sufficient for a majority of applications. The next important thing that sysadmins like in systemd seems to be cgroup management, so I'd like to study the thing soon and assess what needs to be done next; but for now, I believe that the s6 family of programs is now viable as a serious alternative to systemd, and I would love to give it a broader audience.
> systemd has grown fast and conquered large market shares because it was backed up by a company, which put a lot of money and manpower into it - to write it, to integrate it, to advertise it. The only thing that has not been correctly funded in systemd is research and design.
> systemd is only Open Source if you read the license; all its other aspects are proprietary - it's software made by a company and aiming to capture a market. It is impossible to compete with systemd on the same grounds, because no real Open Source developers will have as many resources as Red Hat.
This. The reason systemd is hated so much by "old unix guys" is because it shows that Red Hat basically owns linux the way Microsoft owns Windows and Apple owns OS X.
Many people protested against systemd but it was still pushed by force. It was made specificaly incompatible to make experience on BSDs worse and it sucessfuly removed Gnome from there.
So yeah. Some "haters" talk about unix this and complex that, but I believe it's mostly about scummy company showing off it owns linux now.
I would say it's more like RedHat has actually stepped up and made a "Linux-derived OS" the way that Android is a "Linux-derived OS" or OSX is a "BSD-derived OS." They've thrown the "basic Unix principles"—the ones that make Linux distributions effectively interchangable commodities, but with software having to target a very low lowest-common-denominator of functionality—to the side, and instead adopted a complete "base system API", in the same sense that CoreFoundation+Cocoa is for OSX, and Win32 is for Windows.
As with those other APIs, you can now "target systemd" in the same way you could "target Win32" or "target Android"—and, because of this, we'll probably see a virtualization library (like winelib is for Win32; like ARC is for Android) spring up, giving other OSes the ability to provide the "systemd API" without needing to run a Linux kernel.
From RedHat's perspective, they were just trying to copy OSX and Android: making a "real OS" atop Unix-y foundations. The interesting thing is that a large number of other providers in the Linux space has started to agree with their design decisions—effectively unifying on the "systemd platform" rather than on the "Posix platform" they were focusing on previously.
I think this has been coming for a long time; the large OS providers already think of Desktop Environments as the GUI equivalent to a "platform" rather than as application software for the OS; thus why Ubuntu, for example, has a separate distribution for each GUI software suite.
In a discussion of systemd history held here previously, it was pointed out that back in 2007-or-so, there was a large amount of interest expressed by various distro makers in adopting OSX's launchd as the basis for a new platform standard, if only it had been FOSS at the time. Early systemd was a launchd clone; upstart also started as a launchd clone before mutating.
I have a feeling that although RedHat was the one to step up and create a platform, all of the major distro providers were actually waiting to hop on the first available FOSS "OS platform", no matter who it came from. If, for example, Apple had created "Apple Linux": a version of Linux that runs the OSX GUI and OSX apps—then I can bet you right now that every other distro would have standardized on whatever "platform" gunk was in-between that GUI and the Linux kernel; CoreFoundation et al would have become to Linux distros as WebKit is to mobile web browsers.
As it stands, it's only an accident of history that this didn't happen with the Android platform. (If the Android Runtime ran native-speed binaries from the start, and the GUI was developed five-or-so years later such that it was developed from the start with Win8-like unified desktop/mobile support in mind rather than getting type-cast as "only for mobile devices", I could bet that Linux would "be" Android now.)
---
With all that said, while all the big, commercially-backed Linux providers have been waiting on tenterhooks for a "platform" to latch onto, all the little distros are looking to remain Unix-y, and that's a wonderful thing.
Think about it: when you've got the POSIX platform "for desktops and servers" and the Android platform "for mobile", and nary the twain shall meet, it makes little sense to care about cross-compatibility; why would you want to run Android apps natively on your desktop? But if you add in a third platform that's also for desktop—the systemd platform—suddenly kernel developers and distro makers and whatever you call the FreeDesktop people start thinking about how to create a Linux-based OS that natively supports multiple platforms, so that it can run code targeting systemd, and code targeting Plain-old-POSIX, without making a mess. And once you have that infrastructure in place... why not let it run code targeting Android too? Why not first-class Wine support? Etc.
One thing Windows has had forever (though lately in a degraded state) is multi-runtime (or, given the above discourse, multi-"platform") support. Windows can be Win32 to one program, OS/2 to another, and POSIX to a third. OSX had the capability for multi-runtime support back when it was called Rhapsody (the "Red Box, Blue Box, Yellow Box" architecture) and also co-supported Cocoa and Carbon apps for about a decade. And these are just single companies trying to meet small needs. Linux, the OS with a kernel that supports every device a nerd ever cared to packet-probe, could easily become "the multi-runtime OS", if we care enough to make it that way.
If we don't, though, at least systemd is decent. :P
In a discussion of systemd history held here previously, it was pointed out that back in 2007-or-so, there was a large amount of interest expressed by various distro makers in adopting OSX's launchd as the basis for a new platform standard, if only it had been FOSS at the time.
I don't think there ever was a large amount of interest. In fact, the only party that studied launchd at the time (circa 2006) was Canonical. It was indeed licensing issues that stalled further research, but there was also another contender at the time being considered called initng, which Canonical ended up rejecting and went on to write Upstart instead, led by Scott James Remnant. See SJR's proposal and introduction to Upstart. [1]
Early systemd was a launchd clone; upstart also started as a launchd clone before mutating.
Upstart was never a launchd clone to the best of my knowledge. launchd was likely a spark that influenced Canonical to take action, but the design is pretty different.
Your musings about the "multi-runtime" convergence that systemd will allegedly enable do not appear to pan out. systemd is nothing like the OS X Core frameworks or like the Windows Runtime, it's much lower level than that. It's more of a middleware than a runtime platform (think Hurd, not Core Foundation).
There is also absolutely nothing implying that GNU/Linux will ever target Android code. The divergences are plentiful, with my article about an Android init porting attempt listing only a few. [2] Nor is there any expressed interest from any Android vendor to have any serious GNU/Linux convergence to begin with.
Not sure what "first-class Wine support" is supposed to mean. Wine is pretty self-contained.
> systemd is nothing like the OS X Core frameworks or like the Windows Runtime, it's much lower level than that. It's more of a middleware than a runtime platform
You're not considering the full scope of the thing labelled "systemd." If you only use parts of it, it's middleware, yes. If you "work with the grain" of systemd, though, then you're packaging services in nspawn containers and so forth, which does constitute a separate "platform target", in the same way that e.g. CoreOS is a (mostly-ignored attempt at a) "platform target."
Basically, I'm talking here about a Linux equivalent to Solaris's "Branded Zones": within a container boundary, an app can be made by the system to think it's running on "systemd Linux", or on "POSIX Linux", or on "Android Linux", or on BSD or Win32 or Cocoa or whatever else. Runtime-virtualization is done at the system container-management level (with more or less help from the kernel), rather than expecting applications to "port" themselves by applying their own proprietary virtualization wrappers ala Cider for OSX.
> Not sure what "first-class Wine support" is supposed to mean. Wine is pretty self-contained.
I mean, basically, support to the level of DOS applications run in Win32 VMM containers, rather than to the level of X11 applications run on OSX: management of Wine sandboxes as OS-level "runtime containers", such that you could run and maintain Wine apps alongside other apps, in production, using the OS's maintenance tooling. A Linux "ReactOS Runtime" equivalent to Windows's "POSIX Runtime."
> Nor is there any expressed interest from any Android vendor to have any serious GNU/Linux convergence to begin with.
I don't mean to suggest convergence. This is something very different: making "running an Android binary" a primitive action of the system (not the Linux kernel), where "using an Android virtualization layer" is then an implementation detail of how the system accomplishes this. Think of Linux's binfmt_misc and its ability to execute e.g. JVM code by booting a JVM, but with an upcall to the init daemon to decide how to implement the policy of running a particular binary format. If that init daemon is effectively a container-manager that understands how to instantiate and manage different "branded containers" for each runtime it supports, Linux multi-runtime support just falls out.
nspawn containers are yet not really used as a platform target, and currently Docker is leading. Though nspawn has since acquired some notion of the Docker image format, and even in light of the OCI standard, I do not foresee it becoming a primary container solution in its present form. rkt being nspawn-based may or may not take hold.
Indeed, systemd is as of yet not comprehensive enough to be a POSIX-parity target. There is no "systemd Linux" as such, it's systemd/GNU/Linux or Linux with the GNU and systemd suite. Android is a top-down integrated system on the other hand and does not linearly track GNU/Linux.
As of yet, there is nothing like system call emulation or similar in nspawn to have branded zones.
Nor does Red Hat's present actions imply something like this. The GNOME project, affiliated with Red Hat, is working on a poor reinvention of Nix called xdg-app to enable the "app frameworks and runtimes" design that Lennart Poettering wrote about in "Revisiting How We Put Together Linux Systems," but that too is firmly specific to GNU/Linux as the host.
Red Hat is also leading a container OS called Project Atomic, however nothing like branded zones is seen there as a goal, either. Instead, they've made a simple meta-framework for running various Linux container images over several orchestration frameworks and PaaS, called Nulecule. It's firmly a layer over Docker and Kubernetes, however, so it is limited to that.
A Linux "ReactOS Runtime" equivalent to Windows's "POSIX Runtime."
That would be quite a feat in of itself, systemd or branded zones aside. ReactOS isn't Wine, but even with Wine it would be a sizable integration effort.
If that init daemon is effectively a container-manager that understands how to instantiate and manage different "branded containers" for each runtime it supports, Linux multi-runtime support just falls out.
The init daemon is not a container manager in this case, but an object-oriented resource management with a transactional job scheduler and some limited execution environment modification that work with namespaces and cgroups, but the container framework is outside. As it should be. An init daemon as your container manager sounds dreadful and horrifying, though I hear RancherOS boots from Docker as PID1...
But as there is no such init daemon yet nor anything like branded zones, I still have to say you're a dreamer. This might be a long-term strategy, but with acts like Project Atomic it really doesn't look like it. I still think it's more middleware than Core runtime. I pray it is...
Note that I wasn't suggesting that systemd is itself an attempt at an "init daemon with branded zones"—instead, it's a particular runtime that will increasingly differentiate itself from POSIX Linux, and I believe that that friction will eventually cause developers to want to create an init daemon with branded zones to supercede systemd, where emulating systemd-as-it-stands (for apps built to expect it) would be one of that multi-runtime system's goals. With the POSIX people and the bigcorp Linux providers pulling in opposite directions, a multi-runtime Linux would be the only Nash equilibrium. (Though it'd need someone like FreeDesktop to suggest it and start a working group for it, because neither "side" would care about it on their own.)
That may not be the way things play out, certainly. Things might just diverge and stay that way, if nobody cares enough to change things. But there are unknown unknowns that can give things a very hard shove in that direction. For example: imagine that systemd decides to integrate deeply with GNOME, to the point that you now have an integrated "systemd+glib+GNOME platform" with a unified API, the way that OSX is a "launchd+CoreFoundation+Cocoa platform". That would be an extremely divergent target in the Linux ecosystem introducing a lot of friction to everyone else's development efforts—similar to the early stages of the Android project—and it would get a lot of people's hackles up. But it's not out of the realm of possibility for RedHat and Debian to agree on something like that.
instead, it's a particular runtime that will increasingly differentiate itself from POSIX Linux
Ah, correct.
I believe that that friction will eventually cause developers to want to create an init daemon with branded zones to supercede systemd, where emulating systemd-as-it-stands (for apps built to expect it) would be one of that multi-runtime system's goals. With the POSIX people and the bigcorp Linux providers pulling in opposite directions, a multi-runtime Linux would be the only Nash equilibrium. (Though it'd need someone like FreeDesktop to suggest it and start a working group for it, because neither "side" would care about it on their own.)
It's a hypothesis, then. Plausible, certainly. I assume you're presenting this as a probable way out of the portability concerns from GNU/Linux becoming more of a closed environment to the point of instigating a massive developer schism? This still raises the question of why it would need to reside in the init daemon, but I suppose crass architectural decisions are not a stranger to Linux developers.
For example: imagine that systemd decides to integrate deeply with GNOME, to the point that you now have an integrated "systemd+glib+GNOME platform" with a unified API
Such a decision might just backfire quite tremendously and prove to be systemd's overturning, though it does sound like a similar integration is very much in the spirit of the project. Then, yes, radical surgery would occur, but it may not be quite as dramatic as branded zones and simply an abandonment of systemd and related Freedesktop-ware.
RedHat and Debian to agree
Debian aren't really that large of a player, more of a passive target platform. Being a non-affiliated foundation, they're generally steered into things by whoever is on the committee and seldom ever lead themselves. So, no agreement strictly necessary.
Also, given http://skarnet.org/software/sdnotify-wrapper.c it becomes possible to both use s6's superior notification approach in code and to be compatible with how systemd handles notifications.
This is exciting to me and I would like to encourage you to make it way more obvious, given I've been playing with s6 for a while and still only managed to notice it existed because of http://homepage.ntlworld.com./jonathan.deboynepollard/FGA/un... - and while perhaps that's due to my own stupidity, likely there will be other similarly stupid people who'd also value this information ;)
As Jonathan said: the reason for the compilation phase is efficiency. The information in the compiled database is organized for fast loading and immediate access to the complete dependency tree.
And yes, efficiency is important. The point of s6-rc is to provide a dependency-based system manager with minimal overhead. Time a s6-rc run against an equivalent openrc run, and you will see a notable difference. strace both runs and you will understand why: OpenRC performs all its dependency analysis at run time, every time you run a script, and it has to read the whole database of scripts to do so. s6-rc performs the "read a zillion scripts, do the dependency analysis" part only once, and offline.
There are also other benefits: for instance, simple checking - when you write an invalid set of services, the error is detected at compile-time, not at run-time when it could prevent your machine from properly booting.
The database format is not proprietary. It's undocumented, which is not the same thing; and if people insist, I may document it. In the meantime, the code that reads it is in src/libs6rc/s6rc_db_read.c, a function that you can use in your own code, or that you can analyze to understand the format. It's not a complex one.
Going out of sync is not specific to s6-rc. It's a risk as soon as you compile something. It may happen with OpenRC too, if you forget to rc-update. It may happen with your binaries, if you forget to make install. "Things may go out of sync" is not an argument against compiled designs, it's an argument in favour of paying a little attention.
I don't know Kubernetes, so it's very possible that Kubernetes enforces that name resolution is always DNS and only DNS (a "hosts: dns" line in /etc/nsswitch.conf, without any additional stuff like files which would mean resolution via /etc/hosts), in which case getent is indeed correct, but in the general case, there is no system-intrinsic tool that tests DNS only, and you have to use something like dig.