I've always liked the concept behind GoboLinux[1], it makes so much more sense in comparison to the legacy mainframe style file structure other linux distros use. Debian dropping LSB might pave the way for more distros adopting a more modern and sensible file structure.
This is nothing new, of course. NeXTStep introduced things like ~/Apps, /LocalLibrary, and /LocalApps back in the early 1990s.
Daniel J. Bernstein, around the turn of the century, proposed a /package hierarchy (for package management without the need for conflict resolution) and a /command directory. (http://cr.yp.to/slashpackage.html) Their major problem was the idea that one had to register things with what was to most of the world just some bloke on another continent. But they had concepts like a hierarchical package naming scheme ("admin/", "mail/", &c.), self-contained build trees with versioned names, and an "index" directory full of symbolic links.
the problem is exactly that. some group in an ivory tower refuses to (or can't?) move. most of posix is like that. Linux "standard" for gui stuff is still some definition from IBM to normalize DOS apps (reason why F10 open menus, copy paste with ctrl+ins, shift+ins, etc).
then people that can't work together with others (eg Jobs) does retarded things like /localLibrary which benefit nobody, not even themselves. just throw everything up. then if those folks succeed despite that, it becomes the new immutable thing to follow, like gnome is doing with following blindly apple ui specs.
edit: i never tried to contribute to any larger standard. i did develop against most of them, cursed at their anachronistic ways and coded around it praising consistency. so i am pretty sure i am part of the problem described above. I'm not saying the fine people in those ivory towers are the problem. I'm glad we have that little help for world consistency. it's just not enough.
You're taking received wisdom from Wikipedia, and it's wrong. Common User Access was for normalizing IBM applications, across its then hardware platforms: the PC, System/3x, and System/370. Indeed, it's most promiment PC success was 32-bit IBM OS/2, which implemented the later CUA 91 and which most definitely wasn't DOS.
well, whatever the actual reason, i actually (used to) consult it when writing CURSE applications only because it is the closest we have for a curses application usability standard.
where it came from originally is not much of concern. but the fact that it is the best we have, is.
I do not seem to understand, so forgive my ignorance if you will but; Why do we need a /Apps directory isn't that what /bin /sbin and /usr/bin are for? And as for /LocalApps is that not what /home/$user/bin is for? this seems fairly redundant, or needless wheel reinventing?
On NeXTSTEP and OS X, the /Apps (/Applications) directory is for self-contained application bundles that hold a GUI application and all of its dependencies that aren't part of the base OS, and they can be installed and removed by dragging and dropping that one bundle (which is a directory, but appears in the graphical file manager as a single object). There was also a distinction between locally-installed stuff and network resources which made things a little more complex but more manageable if you were actually working in an office full of NeXT boxes. OS X still has /usr and it still has pretty much the same meaning, but GUI apps live in a separate and more user-friendly domain.
Known on Windows world as xcopy install, due to the similar way MS-DOS software used to be bundled and everyone used xcopy instead of the pretty basic copy that didn't do recursive copies.
But sadly it doesn't present the nice object concept NeXTSTEP and OS X have.
Not advocating for one of the other, but they're not the same. /Apps is based on the application that supplied the files, /usr vs / is based on whether necessary to boot the OS, /bin vs /lib vs /share based on the file type.
Probably just more intuitive more than anything else. Someone without prior knowledge to the operating system (or even computers in general) and things like /Apps is really obvious where as /bin and /sbin understanding is really going to depending on how much someone knows about computers.
As far as I know, /Apps is for packages that ship with their own versions of shared libraries and everything else the program needs to run; all files are installed in one directory under /Apps, instead of where they actually belong in the Unix directory structure.
/bin and friends are for binaries only. For a full package you also want libraries, man pages, configuration files etc... Those are usually bundled as a .tar.gz which would ideally be the best way to bundle and distribute a program.
This sounds somewhat similar to /opt (albeit more complicated), wherein programs are located in /opt/package or /opt/provider. For example, the entirety of Sublime Text is stored in /opt/sublime_text and Google programs are stored in subdirs of /opt/google.
I have been saying for years our Linux Package System is broken and antiquated. When we had 40 MB hard drives or no hard drives at all these systems made sense. Today we can have a system that just works like this or Gobo Linux. I have hopes that with containers that we will see some major distro switch (Looking at Arch Linux) to a non-shared everything libraries.
The point of the FHS is not to save disk space, although that's a nice side effect. The point is to not have multiple copies of the same bug or slightly different bug across each package.
Security updates perhaps: If e.g. OpenSSL is shared, a single security update will fix all applications using it. And a single update will most likely be out more quickly than updates to a dozen or so applications.
The thing is that the libraries are not shared, and every application has its own version of library in its application directory. So if you would want to update OpenSSL you would have to find openssl in every single application and update (or more likely wait for vendor to release new version with more recent openssl).
At that point the shared libraries are useless, they don't provide security (update in one place and have all aps using it patched), doesn't save RAM (each app loads their own version of library, instead of multiple apps sharing the same code), and you get worse performance (there's overhead for using shared libraries).
What lsaferite suggested makes sense, if the library is used only by given app, why not compile it statically? You have everything in a single file, and at least you're not losing performance by using shared objects.
The scheme that Canonical seems to be playing with is that larger projects (DEs, central libs) are put into what they call frameworks.
These are packages that other packages can refer to rather than massively duplicating the content within themselves.
These frameworks will in turn be versioned, so that if package X wants framework Z, while Y wants Z+1 they can both be happy without descending into dependency hell.
Note that Gobolinux gets around this to a degree by using age old soname and symlinks. This in that ld and similar is pointed at a directory containing symlinks.
libxyz.so.1 will be pointing to whatever is the active version of libxyz, while something like libxyz.so.1.1 will be pointing to that specific lib version.
Disabling a specific lib version is done by removing the relevant symlinks.
Gobolinux provides tools for doing this, but they are basically shell scripts. And so in a pinch someone can do all that manually.
I guess it will depend on the size of the distro project backing it. Gobo, by "virtue" of being a small group, can't provide a large store of pre-compiled packages. Thus one at present have to rely on compiling on ones own.
That said, once compiled a binary package can be produced by simply creating a compressed archive of the content of /Programs/app-name/app-version (an official binary package is a bit more involved to ensure compatibility with the latest released iso).
Beyond that i am not quite sure what you are asking.
It's funny you should say that. I've been using Android for several years after using iPhones. And I find myself wishing all the time for a phone that simply runs Debian Stable.
Tight security? Well, how about Stagefright? Cyanogen no longer supports my phone. Even when they did, it wasn't well-supported: each new release had a high chance of breaking something fundamental. Right now I'm stuck on a "milestone" release that isn't even the newest one (it's over a year old now!), because later ones all have some unacceptable problem, like GPS not working or cell data not working. I can't just install an updated browser component package, because Android is monolithic. On Debian it'd be in a package like libwebkit, and it'd be a simple "apt-get upgrade" away.
So I'm stuck with a vulnerable Android version. If I try to upgrade to fix it, I'll lose GPS or cellular data. The only way to get a fix is to buy a new phone.
Now Google is trying to push more of the components into its Google Play Service app/framework. So, great: more parts can be upgraded independently of the monolithic core. But that also means that Google can push out updates to more components without my knowledge or consent: not great. That means more instances of feeling something warm in my pocket, pulling out my phone, and finding that Google Play Services has kept my phone awake for 10 hours and my battery is at 12%.
This has happened to me on multiple occasions: one day my phone works fine. The next day it eats the battery like candy. I didn't change any settings or install any updates--Google merely decided to push out an automatic update to its APKs without my knowledge or consent, and Google Play Services constantly crashes every 5 seconds, popping up a dialog box that I have to manually dismiss. I can't even do a simple task like send an SMS or check my email, because every 5 seconds it pops up another "Unfortunately..." box. There's no practical way to roll it back, nor even to report the problem. The best I can do is find people reporting the same problem on Reddit and hope that Google gets its act together so I can actually have a usable phone again; maybe one morning I'll wake up and my phone will not be "sick" anymore. In the meantime I have a really dumb "smartphone" that keeps my leg warm.
I have had high hopes for Android, but I now consider it to be a mess, and consider Google to be a highly irresponsible steward. I don't want an iPhone, but if someone wants a reliable smartphone without nasty surprises, they are probably better served with one.
I really miss maemo, it was pretty much what you're describing. Being able to rebuild packages directly from the debian archive if I wanted some new software, or patch some system software and put it in my own APT repository, are things I really liked.
In addition to the problems you describe with google moving more core functionality into its "Google Apps" packages, it also means that running android without the google bits is getting increasingly hard. It's missing some core functionality that other non-google apps expect.
Public build service is https://openrepos.net/ and it already has lots of packages that you know from a normal Linux desktop and you can also host your custom stuff there. High-level package manager is packagekit or zypper.
I would also recommend using something like AOSP with a very minimal GApps install. Or bake your own ROM if you really need fine control. I even had to build a modified TWRP recovery in order to fully support my latest device. Additionally, use a firewall and Xposed with XPrivacy to tightly reign in apps. However the reality is that this is no fun, and not something you can expect the average consumer to do. It is quite unfortunate and I have similar sentiments.
I never enable automatic app updates. However, Google Play Services is unaffected by this setting--it updates whenever Google wants it to, without any indication to the device owner.
Sure, I could use Android without Google Play Services. That would also mean not being able to use the vast majority of apps in the Play Store--and I don't mean games. That would mean not being able to use, e.g. Google Maps.
So to use Google Maps, I should have to give Google free rein to push updates to its software to my phone whenever it wants, including untested updates that cause breakage, crashes, and battery draining causing high temperatures?
This is what's wrong with Android.
And you can't seriously claim that such drastic breakage happens on a regular basis with Debian Stable.
Why is /usr/bin being adopted as the place for binaries with /bin linking to it instead of the other way around? Since they're essentially the same directory under this setup why not favor the short path?
In short: because having things in /usr is equally compatible, and makes some useful things like atomically snapshotting /usr to snapshot all executables, mounting /usr readonly, etc. possible.
It's an interesting question. One part of the the answer is that the idea is for there to be a single subtree off the root where "the operating system" lives. Where MacOS 10 has /System, OS/2 has \OS2, Windows has \Windows, and Digital Research DOS has \DOS, systemd Linux has /usr.
Of course, another part is that this is an old idea, already implemented once in the commercial Unices a quarter of a century ago. They merged /bin into /usr/bin, with a symbolic link from the former to the latter.
A lot of those designs predated initramfs. I have yet to hear of a setup where there's a good reason for not mounting /usr in the initramfs. (Though, yes, I have heard of setups where you can, it would just be a fairly involved change. It is a legitimate problem for these sysadmins to be compelled to rearchitect things, although I'd personally blame that more on the distro model where all parts of userspace are tied together than on systemd itself.)
That looks fine for embedded systems or maybe some end-user systems, but doing that in a server context would be scary. The original filesystem hierarchy was designed so that /usr, /etc, /bin, and so on would have their own partitions and in the event that something tried to eat the filesystem, you might still be able to boot into single user mode and repair the damage. I've had to do this on a handful of different servers where /usr is boned and won't mount, but I still have enough stuff available that I can fix it.
Plus there are a handful of tools with smart option that take advantage of the multiple partition approach, like rm and rsync's --one-file-system option.
Except, of course, that /etc/init and /bin/sh being on "their own partitions" is the opposite of what was actually the case in the original filesystem hierarchy. (Hint: Yes, /etc/init not /sbin/init .)
I believe that what JdeBP is refering to is that, before /sbin existed, the binary which is now at /sbin/init used to be at /etc/init, if I recall correctly what I once read in an old FSSTND rationale.
The original filesystem hierarchy was designed when 1MB was a large amount of space.
If you want to have a separate read-only partition for system programs (and require going into single-user mode to modify that), go for it. But the rest of the /bin /etc /usr /usr/local nonsense is just because it's easier than changing to something better for most people.
Someone should've posted a polite correction instead of downvoting you. So:
The thinking behind the file system layout had nothing to do with the available disk space or having a read-only file system. It was, originally, entirely a hedge against corruption; before journaling file systems and all that jazz, if your system was hit by a power outage and it was in the middle of some operation or another it might corrupt whatever file system it was working in.
So, you would create multiple partitions and have different directories as the mount point for each partition. Say for instance that, as a cautious sysadmin, you put /sbin on its own partition. Excellent! Since /sbin rarely gets touched, if everything goes pear-shaped, the system will still boot into single user mode and you can use tools like fsck and mount, which are under the uncorrupted /sbin partition and are statically linked. If you couldn't for some reason recover the file system on a particular partition, then you might be looking at having to load, say, /etc from backups, instead of everything under /.
Over time, other software started to take advantage of multiple partition layouts to protect you from getting hurt. You could mark some partitions noexec, nosuid, or restrict operations like rm or rsync to specific partitions without having to worry about whether or not they'll crawl out of that partition and eat the world.
But that's pretty old-school now. As file systems improved and corruption was less of a problem and everybody sort of gave up caring about what options rm accepted, more and more systems just mounted everything under / so that you would be maximally boned if ever there was a filesystem problem. Progress!
So that's why I'm not a fan of "novel file system layouts"; the previous layout was designed and existed for some good reasons and I have a hunch that the people that are happy about novel file system layouts don't know about those reasons.
If my filesystem gets corrupted, I boot into a live cd, just like everybody does.
With btrfs and integrated parity, checksums, cow and scrubbing on the horizon filesystem reliability is only a minor thing to care about. With subvolumes we can snapshot as many things atomically as we want.
Simlicity is now what trumps your arguments overwhelmingly. The filesystem hierarchy has to be intuitive, consistent and simple, which it is currently not.
Booting a LiveCD would not have fixed affected partitions.
You're not the first on HN to say in so many words that the sometimes expensive lessons I've learned over 30 years of computing are suddenly irrelevant now. I used to argue that, but I don't see much point to it anymore.
One of your arguments was to put /bin and /sbin onto a seperate partition so that if something happens, it’s unlikely that /bin and /sbin are affected. In reality however, nobody cares about /bin and /sbin at all.
Another argument was more fine grained permission control. You can do that by other means, for example with subvolumes.
So, after all, possible corruptions are not alleviated by complex partition struture, or only with a bad trade-off for complexity. Instead, do backups. Easy and reliable. Everbody is happy, without many partitions.
Simplicity is crucial and it's one thing the free software ecosystem is consistently getting wrong. I see it as the major showstopper for many people to use it more intensively.
The filesystem hierarchy is just one culprit, but a fundamental one.
Take a look at at some fundamental pure mathematics lectures or books. You can immediately see that this was designed to be as easy as possible. Software must be the same.
> If my filesystem gets corrupted, I boot into a live cd, just like everybody does.
And that is what makes someone take the road trip to the server farm at 2am because a ge-wiz dev thought the exact same and never considered a "rescue" mode with network capability.
Your use case is a tiny minority which doesn’t provide enough justification to increase the complexity of the default, for complexity being _very_ expensive.
And this is why Linux user space is going down the tube.
Basically that exact capability is why people adopted Linux in the first place to run the servers that run the net.
I don't see how having a / that can get up and running even when the larger system housed in /usr and such is down for the count, is anything more complex that the piles of VMs and containers that is being advocated today.
You are free to set up your system as you wish. I also don’t advocate for removing options. I’m asking for a simple default. Simplicity is very important. If you can’t explain what you are doing to your parents, it is not yet simple enough.
Don’t misunderstand simplicity for triviality for they are not the same. We need to make non-trivial problems simple.
I run a multiuser shell system for hosting tmux/ssh/irc type usage. It's limiting but it's a labor of love for friends. For anything that matters I use a modern (piles of VMs and containers!) design.
Let's ignore Amazon, Google, Microsoft, Facebook, and so on, each of whom are by themselves rumored or reported to have over a million computers in their DCs. Let's further ignore the long tail of the second tier of providers, who are thousands in number and have merely hundreds of thousands of computers each in their DCs. Let's go on to ignore all of the self serve corporations in the hundreds of thousands of data centers across the world, running Linux or a derivative on thousands to hundreds of computers each.
Let's also ignore all the routers, switches, file servers, and miscellaneous other networking equipment running Linux or a derivative.
Let's further ignore all of the televisions, media players, microwaves, appliances, and other consumer goods running Linux or a derivative.
Let's look only at Android phones, which run an embedded derivative of Linux. The phone market shipped 341.5 million year over year in 2015 Q2 (http://www.idc.com/prodserv/smartphone-os-market-share.jsp). Android had an 82.8% marketshare, so that's 282.77mm.
The number of linux computers that have the ability to boot "a live cd" is 6 thousandths (.6%) of one segment of one part of the linux embedded and server market.
Linux on the desktop has not only been dead for the last 18 years, it's barely a rounding error on a single segment of a single part of the embedded and server market.
By the way, "the complexity of the default" has been fine for over 30 years, and it's still fine. Just because you don't understand it, doesn't mean it's not the right answer.
Thanks for the link! That article and the comments support your argument better than mine and I have to defer to ChuckMcM's expertise -- it predates mine by a lot, my first experience with Unix-like operating systems was in 1996 and his memory is better than mine anyway.
I'm 100% certain I remember the instructions for partitioning being related to file system corruption, but that may well have come long after the original design of the FSH.
I nuked /usr on one of my servers for cleaning purposes some 5-6 years ago when I was still quite fresh to Unix. After reboot, I was unable to log in even as root. That was the day I learned not to change the shell of root to anything outside of /bin (I had set it to /usr/bin/bash).
I was able to recover the machine with a LiveCD but still, it was annoying.
One thing about Gobo is that it does not try to enforce any specific layout outside the distro itself. Heck, the last version basically have a legacy layout inside /System/Index.
This so that it is less likely that anything with hardcoded paths will throw a hissy fit.
Frankly Gobo demonstrate the massive flexibility of classic unix tools, as most of it is shell script and symlinks. There is perhaps a few binaries, purely for performance (and those are third iteration, the first being shell script, the second Python).
Well Gobo basically don't care, it maintains a replica FHS so that minimal effort is needed to get existing software running.
Heck, it originated as one guy's solution for managing self-compiled programs within his home dir on the university mainframe (or some such).
You can still find Rootless (the bootstrap tool for using Gobolinux in the home directory) on the Gobolinux site, but i can't comment on how well maintained it is these days.
We package our software for CentOS, Debian, Ubuntu, and have occasionally supported a few others. We never even considered targeting the LSB. It is so much simpler to target the defaults on the system, build for the top three or four or five distros (which actually ends up working on several spin off systems, by virtue of similarity), and when problems arise we know we're not the only ones running into it because we are using the same libs as the rest of the system.
The idea of one package for all distros is cute but not at all realistic. And it isn't pleasant for the end user, who wants everything in their native package manager and things to be configured in the same ways as everything else.
LSB was a well-meaning idea, but it doesn't actually do anything to improve openness or interoperability.
It is possible to build one binary that run on all reasonably popular distributions, even for quite complex software; there is just pretty much no documentation on how to set up a build environment to do so (there is no "GNU/Linux SDK").
A project I was familiar with some years ago used essentially a NFS mount containing a subset of a 4 year old "enterprise distro" and then built with GCC --sysroot plus some vars like PATH, PKG_CONFIG_PATH etc set appropriately.
One of the problems is, beyond the few things listed in LSB (so it's not entirely useless!) it's not at all obvious what libraries you can expect to be on the system in an ABI compatible way - if you want to use something as unstable as OpenSSL you have to bundle it.
We support and manage all of the major init systems (initscripts, systemd, Upstart, BSD, etc.) so we don't care about that (or we do, but the software already has configuration and code for all of the possibilities).
About a year back I was tasked with porting an installer for a proprietary product to work on Linux. I wanted to show the log files to the user in the graphical installer and I ended up using xdg-open, which as it turns out opens text files in Firefox, even though light weight editors are present on the system.
I also found that some scripts had the hashbang line as /bin/env which works on RHEL, but not on SuSE.
I have also been trying to write some scripts that work across multiple distros. Detecting distros is wierd. "lsb-release" is not present in most distros. "/etc/issue.net" is present in some, but not on ArchLinux.
No wonder ISVs find it hard to support Linux. I suspect the technical side of this problem is probably trivial, the leadership side, not so much.
/bin/env was always a bad idea. Way before Linux was even a thing, every Unix used /usr/bin/env - and that was the point of it being used to find an interpreter. Red Hat (where I worked at the time, man years ago) realised the mistake, but alas didn't want to break compatibility.
'lsb-release' was always hilarious: the one thing about Linux that is almost standard is that the LSB tools are never present.
IIRC not all distros will put executable binaries in /usr/bin/, but even those that don't will usually provide /usr/bin/env (even as a symlink), which will then search your path for python. In summary: /usr/bin/env is more portable. I'm unsure whether "#!/usr/bin/env python" is equivalent to #!python.
> xdg-open, which as it turns out opens text files in Firefox, even though light weight editors are present on the system.
It opens text files in Emacs here – did you configure your system to associate the text/plain mimetype to Firefox? (The following mostly for anybody who is interested):
$ xdg-mime query filetype <file>
will print the mimetype of <file>.
$ xdg-mime query default <mimetype>
will print the default .desktop file associated to this mimetype, i.e. the application that is run if you do xdg-open. That application can be changed using
Maybe that particular distribution was misconfigured to have the default for text files be Firefox, but overall I don’t find the behaviour particularly surprising.
I've never encountered a system where xdg-open did something I didn't expect. It's basically the linux variant of windows' double-click and OSX's open. If I did it on a text file and it would open it in Firefox I would think "Jeez what idiot chose Firefox for text files" not "Jeez xdg-open is useless".
It opens the file. In which particular way it opens that file of course depends on the system in question – I don’t have Emacs on my phone, I don’t want xdg-open to attempt to open text files with Emacs on my phone.
Whether or not the apparent distribution default of opening text files with Firefox makes sense is an issue somewhat orthogonal to whether xdg-open works – in that particular case it may have been misconfigured to open files with Firefox, but that doesn’t mean it a) didn’t open the file (it did) nor b) in a completely unreasonable way (the file was still displayed and such).
xdg-open chainloads a desktop-environment specific loader if available, and only if not, uses a heuristic to determine what program to load. Currently it supports three different loaders for various KDE versions, two Gnome loaders (one of which is shared with Mate, which also has its own loader supported), and one for each Xfce, LXDE and Enlightenment.
If none of those are found, xdg-mime will be used. Relying on xdg-open to work sanely across desktops, much less across distributions, is… optimistic at best.
I worked on some large commercial software for Linux for many years that had to support a wide variety of Linux installs (old and new), I can vouch for it not being easy. The technical side of supporting Linux in this environment can be very difficult.
There are, as you note, issues with the locations or presence of certain files, directories, and scripts. That's one part of the problem, but probably the easier one to solve (we had a whole big function that would try to detect how to open a URL by checking for xdg-open and older variants, checking configuration for KDE/GNOME/etc., and then falling back on detected browsers). This also has to be done for opening arbitrary files using their assigned program.
What's more complicated are library dependencies. Depending on system installs on GTK+ isn't always enough. You may need features not available on that install, due to version issues. You may have to work around versions of system libraries that have a bug present on one particular distro due to a custom patch.
Themes and icon sets are a problem if you hit any version compatibility problems. Your UI can appear broken if you reference icons that aren't registered on that system (generally an issue on older installs). This actually gets complicated quick in some cases.
So as an ISV, you have a couple of options for how you approach all this. You can limit the distros and versions you support, putting out builds for each. This is a perfectly fine strategy if you can get away with it, but not everyone's going to be able to use your program in that case.
Another option, which is what we did, was to basically ship everything we needed along with the program. Libraries (glibc, gtk, x11, everything we were forced to ship for compatibility), icon sets, themes, etc. You then try to dynamically figure out which modules you can load from the system, and which you need to load from shipped modules. You also need to handle setting up paths for anything like icon sets and fallback themes you may need, and deal with the compatibility issues that can occur from that (older versions of the XDG icon theme spec didn't have a properly-functioning fallback system when specifying multiple icon theme directories).
If you ever need to do anything more specialized (support for hardware devices, anything using D-BUS on older installs, more specialized window management abilities), you run into a whole new set of issues.
These days I focus almost entirely on web-based software, so that sort of wide range of compatibility problems are entirely behind me. (Ahem.)
It would be great to still file bugreports for anything you think could be improved in e.g. a distribution. I help out with Mageia. There's often various things which are simply overlooked or people aren't aware of. Obviously your bugreport could be ignored or not dealt with, but it doesn't have to be. It's a small investment to hopefully (years later) have less things to deal with.
I do the same when packaging upstream software. Sometimes there are build problems due to configure(.ac) not checking for dependencies properly. I'll then open a bugreport that these checks should be added. All that small stuff helps quite a bit.
A lot of the problems we hit were due to having to support many versions of many distros, not so much a particular bug in one distro. That said, we filed bugs, provided patches, and even drafted XDG specs for some of the issues we were dealing with.
The problem is, even if we filed a bug report with a patch for a 3.0 release of some distro, and it was fixed in 3.1, we still had to support 3.0. That meant code that worked with both the buggy 3.0, and the fixed 3.1. That's not a big deal when isolated to one distro or one component, but we're talking all the major distros, various third-party libraries, etc.
We may have been a bit nuts, trying to keep such a wide variety of environments supported, but that said, we also got really good at it, and it meant we didn't have to force customers to make changes to their system that they didn't want to make. The products ran on distros so old that they still only shipped GTK 1.2 (though we eventually finally walled off some of the older ones when we started supporting only 64-bit processors).
> I have also been trying to write some scripts that work across multiple distros. Detecting distros is wierd. "lsb-release" is not present in most distros. "/etc/issue.net" is present in some, but not on ArchLinux.
The impression I get is that the main point of LSB was to combat FUD from commercial Unix vendors about Linux being fragmented and nonstandardized and lacking binary compatibility. Once Linux supplanted Solaris in the corporate world this all became unnecessary. Is this right?
As someone shipping proprietary applications on Linux, the lack of binary compat is still not fun at all. In a previous gig, we ended up with 20 odd build slaves ({x86, x86_64} * Common RHEL, Ubuntu, Debian), and making binaries for each, rather than trying to make LSB based binaries.
What kind of problems did you have with trying to make common binaries? The old wisdom is that you can make it work by building against an old enough glibc (and maybe other conservative deps like base X libs) and then ship other libraries along with your binary.
I was very excited until I saw: "Supporting graphical applications such as those based on GTK, Qt, SDL, OpenGL etc is outside the scope of this project."
To the point, the rest of the discussion here revolves around init scripts and the filesystem hierarchy, which are the only parts of LSB that anyone actually used. (And they're sticking around in Debian, at least for now.)
In theory LSB meant a vendor could ship an RPM that would work flawlessly on any Linux distro supporting that LSB release (which on Debian meant supporting RPM alongside DEB as a package format). In practice vendors would just target RHEL and/or Ubuntu and call it a day.
I remember back in the day, LSB was announced as a means to avoid the return of UNIX wars in GNU/Linux.
Here we are with each distribution having its own way how to do things, so now we do have GNU/Linux wars, with a few selected distributions just like the old commercial UNIXes, because no one can bare to support X thousand variants.
As you can see it has the big name distributions on it, except for Debian and Gentoo, plus many obscure ones.
However, the list of LSB certified applications is rather remarkably short, which suggests that the real problem is that ISVs are much less interested in LSB than one would expect.
In a follow-up message, he noted that changing the LSB to be, essentially, "whatever Debian as well as all other actors in the FLOSS world are _actually _doing" might make the standard—and the effort to support it in Debian—more valuable.
I think this strategy worked remarkably well for HTML5. Standards is a two-way conversation with the community.
The problem with LSB is that no application used it, because not every distro supports it. I hope that xdg-app application containerization[1] will change that and make it finally possible to write an application that targets all distributions. Well, all distributions that implement xdg-app ;).
Suggest to just ask various distributions nicely if they could package xdg-app. Especially the ones that you target and/or use frequently. I quite like the idea behind xdg-app and I've packaged xdg-app for Mageia Cauldron (think Fedora rawhide). As I did that only yesterday I don't know if it works. But next stable release of Mageia will have it.
I suspect Windows/Microsoft is capable of self-destruction, too. Remember Windows Mobile? It used to be the dominant OS on smartphones before the iPhone and Android came along.
Is fingerprint reading within the scope of what OpenGroup is trying to do? POSIX doesn't specify whether you should make a web page with CSS2 or CSS3 either.
Higher level languages (and libraries) seem to be defining the portable interfaces for a lot of what software needs to do now, which helps but essentially seems to punt the problem at the operating system level.
I guess portable interfaces may just end up needing to be defined over non-portable interfaces, and C may end up not being practical for anything that needs to be portable.
Higher level languages aren't bad, and it would probably be a good thing if a language like Rust became the de-facto for low-level. My feeling is that C is still the simplest to compile though, which I think is an advantage.
> Most compilers I know of do register allocation better than nearly all humans where optimization is important.
Nowadays yes, you are forgetting C has 40 years of research into compiler optimization.
We could easily beat the register allocator in the 90's with Assembly coding, specially in x86.
There is a quote from a famous woman in compiler research from the mid-80's, where she claims C semantics have thrown the optimization research back to the dinosaur age.
Sadly I cannot find it, as I don't remember the exact words.
> Aliasing is a problem, but I don't think it's strictly inherent to C.
Most languages have stricter semantics for aliasing and do not decay arrays into pointers.
Also they tend to have better type information, which can help the optimizer.
Last I checked there wasn't even a portable way to make a new filesystem. POSIX standard is a lot like the C standard, far above and almost completely hardware agnostic.
Actually UNIX kind of plays the role of C runtime, but when the time for standardizing C came, they could only make into ANSI C features that were common across multiple hardware and OSes back in the late 80's.
So what was left out of ANSI C, is what became POSIX. Making any OS with ANSI C + POSIX compliant compatible with the original UNIX systems.
But unlike other programming languages, on C's case the evolution in terms of standard libraries stopped there.
I guess what I was really wondering was why so little has changed with respect to multiprocessing, modern filesystems, resource quotas, virtualization, etc... since '08.
I personally have a number of specific gripes, but more importantly it seems odd the committee hasn't added anything to the standard since '08 (especially since good portable software generally needs more functionality now).
As an easy example, multiprocessing has become more important between now and then. Why are things like processor affinities still not defined?
There has been recent movement in multiprocessing - for example robust and errorchecking mutexes are now in the base spec.
In the case of processor affinity, it looks to me like no-one has ever suggested it. Participating in the Austin Group is pretty open - why don't you put a proposal forward? (Maybe affinity can be added to struct sched_param somehow?)
This was a bit disheartening to read. I didn't see a any problem proper versioning couldn't address. Breaking backward compatibility shouldn't be an issue as long as you generally don't break forward compatibility. I was under the impression this was what the POSIX_C_SOURCE macro was meant for.
I worry this will make things more complicated than necessary in the future. The LSB is one of those things that you don't really miss until it's gone.
Is this a solvable problem via package tools?
Can (and maybe my ignorance isshowing) can say apt-get not interrogate a .deb file and ask "which files re your config?". Ok for this distro, they go under /GOBO/yourapp/config or /usr/etc in that distro.
A final single constants file could be handed back to the app
Saying "CONFIGDIR=/foo/bar/bad"
So it could write the files to any location. But how to tell the application it should now look in a different place ? That's probably impossible to get every developer to do.
LSB is actually somewhat to blame for prolonging sysvinit's lifespan, in its specification of standard initscript functions (which ultimately were seldom used and laughable compared to /etc/rc.subr in BSD-land) and the awkward dependency headers that required a preprocessor like insserv(8) to scan them.
systemd as base system will certainly happen, regardless.
I, and many others, prefer sysvinit over systemd, which I consider overly bloaty and not in-line with the unix philosophy of 'do one thing and do it well'. It's far too involved in the overall systems where, now, some desktop environments require it? That's absolute crap.
I don't know of many people who actually liked sysvinit. It's mostly apathy due to its low surface (though actually sysvinit is still monolithic and overly coupled compared to members of the daemontools family like runit or s6).
It's a red herring, anyway. People have been replacing classic BSD and SysV initialization schemes since 1992 [1], but systemd was the first to penetrate the scene quite massively for a plethora of reasons that no one can seem to agree on.
Responses to software seem to follow patterns. Sysvinit is basically Windows XP. People hated it for its lack of sophistication until it got replaced by Vista with its enhanced permissions, which people found inconvenient because they required slightly different interactions, and then XP became, if not good, at least the best in existence. Systemd will eventually be replaced by a minor revision which addresses the main complaints of the vocal minority, and then when NeoUpstart establishes dominance despite a massive list of bugs, systemd will rise to immaculate status and everyone will brag about how they used it before it was cool.
Well, in fairness, systemd isn't really a matter of changing sysvinit. It's changing the entire low-level userspace, so there's a lot more one can object to, in principle.
Though I'm not going to say I really care for sysvinit - I prefer it to systemd as well. Every interaction I've had with systemd just ends up confusing me until I either successfully get it work _somehow_ outside of logic I can actually comprehend or giving up entirely. A lot of people I know have had similar experiences.
I used to be on the fence. I knew I hated journald with a passion. But for systemd, the last straw was trying to mount a USB drive one day. I would do "mount", and nothing was mounted. I looked through the logs and noticed it was being unmounted on mount. What the fuck, right? Turns out, a prior unmount of a USB stick killed some service that systemd runs. So systemd thinks the service is dead, which means it should not be mounted, which means it keeps umounting the damn thing. Totally braindead.
That's also when I learned systemd was running an entire virtual fstab layer. Because, you know, /etc/fstab just isn't good enough anymore.
This is actually a good thing. You might have found it bothersome, but what it was doing was intelligently trying to reason whether a device would cause problems in your machine. It did not make the correct decision, but the infrastructure is there.
I would file a bug on systemd mentioning this behavior. But I like the fact that that my usb wifi card will not screw my system.
I don't want intelligent tools. The smarter the tool the more difficult it is to fix when it inevitably breaks. The intelligence might save a few minutes every so often, but the failures can cost hours, and at a time when you might urgently need things to work correctly. Predictability is more valuable.
Interesting perspective. My experience has been the complete opposite - writing unit scripts used to be a real pain, but systemd unit files take two minutes and just work every time.
My machines start much faster with systemd too, which is a nice side effect.
Has someone made a resource where noobs like me can go and read about why systemd is "almost comically bad" in a succinct fashion?
I've seen a lot of political stuff, and a lot of claims that sysvinit ain't broke so we shouldn't fix it (which makes little sense to me), but haven't seen a good resource discussing technical merits, or the lack thereof.
I could probably do it, but in all honesty, the "debate" has been so spoiled that I doubt anyone would take it seriously. My relatively technical article about FreeBSD and launchd (which had only one error about where plists were parsed, a relatively minor gripe in the text) was near-universally misunderstood and derailed by commentators who went on rants tangential to what I actually discussed. Same with my init history article.
But, if enough people want to, maybe I'll bite the bullet. Not sure if I'd like to subject myself to it, but I think I might end up writing a rather long treatise if there's demand and I do go with it.
I'd also be interested. I kinda like systemd as far as I've tried it, but as you say, the discussion around it is just inane. I'm sure there are valid complaints buried in there, and you seem to be a reasonable sort about these things.
There are other pages out there claiming to be neutral overviews, but they generally just take a neutral tone while rehashing the same long-debunked myths and spreading FUD.
The specific thing I like about systemd is the init replacement. It's just so much better! Automatically tracking daemon child processes, even detached ones. Supporting all the different modes daemons work, so I don't need to write yet another damn clean-up-all-the-children function in a SysVinit script. It's wonderful.
But systemd is a lot more than an init system and I don't really have strongly formed opinions on those.
Those don't cover the technical aspects, although to be fair, something like 70-90% of the systemd "debate" is about non-technical issues, like "the Unix philosophy" or "change for change's sake"--just read the comments here!
Well no, I've never written a central architectural critique of systemd, only scattered criticisms across comments. I will nonetheless be remedying that soon.
Indeed, that bug report is an interesting example. I can see both sides failing.
1. It was not a good bug report. 'Do not parse "debug" command line parameter' is not a bug. The actual bug is "assertion in systemd-journald" which is a clear regression, but independent of kernel parameters. According to comments 17 and 22, it was promptly fixed.
2. According to projects process, discussions about design are to be done on the mailinglist and not in the bug tracker. Most commenters violate this.
3. No systemd dev explained why they parse the debug parameter.
4. Sievers closed the bug report without explicitly acknowledging the assertion problem.
5. The kernel devs (especially Linus) are too hot headed and misread this. Systemd caused and fixed (!) the assertion problem.
You're quoting a troll as reference for "anyone with a clue". Further, you're referencing that the systemd Debian maintainer quit on the harassment in the systemd discussion because he wanted systemd _in Debian_.
Linus called out on some behaviour. That is not him saying that people should stop developing. It's him saying stop this behaviour.
As shown in the LKML and Debian debates, the systemd infestation
was firmly opposed by nearly everyone outside the 'vested interest'
RedHat filter bubble. And still is.
Some high profile names would be Eric S. Raymond
or Patrick Volkerding (creator of Slackware).
http://ewontfix.com/14/ is one of the classic posts here, but it deserves a good rebuttal, not the straw-man one included. (The starting point I'd make is, if init were really as simple as purported, why isn't it built-in to the kernel? Why not simply skip all the kernel code to reparent children to init, and have children die immediately instead of becoming zombies if their parent has exited? Why not remove all the code to panic if pid 1 dies, and make it just a process like any other?)
There is stunningly little content in that link. There's a lot of broad points absolutely devoid of serious examples of how they are problems under systemd.
You can't just go "attack surface" and have said anything useful, nor can you point at volume of code if that code is actually needed. All criticisms of systemd usually depend on people completely ignoring the very serious problems you can't solve with sysvinit and which do make actual, practical systems less secure (i.e. cgroup process isolation is amazing, and a lack of real service management is a huge problem and any service management is going to be running as root period. Yet there's been a real claim that service management is "what your sysadmin does". Why even bother with computing then?)
I've been a sysadmin for over a decade, and I love systemd.
The arguments against it appear to be philosophical, not practical, and at no time have I encountered any defect in it. After a brief learning period I came to fully embrace it, and in actual use it makes my life easier in several minor but appreciated ways.
It's understandable that people become used to convention, and are resistant to change. However, most complaints about systemd seem to be appeals to tradition.
I actually have a technical rationale for avoiding systemd, among others.
This is a comment I was supposed to send to chris_wot in the systemd v227 announcement thread, but his posts got nuked by dang.
Here goes:
----------------------
systemd tries to force various system resources into the Unit abstraction, to a rather mixed effect. Some are IPC resources (sockets), others are abstract entities which are driven by a lower-level primitive (services that drive jobs), yet others still claim to wrap resources but really drive util-linux binaries (mounts and swaps). The information hiding/encapsulation is inconsistent, too. I'm not shielded from the dirty details of transactions and the seven different job queuing modes, and when they actually blow up in my face, the debugging context is very scarce.
All of this takes place in an ad-hoc graph engine that I cannot really observe. I can look at approximations of it (systemd-analyze), but I have no way of directly manipulating it, only indirectly through normal operations on system and service state of which the effects I cannot reliably gauge as they undergo systemd's heuristics. Moreover, I cannot checkpoint the graph state to disk, so I could e.g. replay and reboot from a known-good system state manifest. systemd snapshots are ephemeral (in-memory) and moreover do not fully preserve the graph info, anyway.
systemd clearly has an identity crisis between whether it wants to be an eager or lazy service manager. Publicly, it recommends laziness through so-called socket activation, but in practice people mostly tend toward using the dependency system. The dependency system, of course, exists mostly to calculate ordering requirements for parallel startup. This however comes at a cost of non-deterministic ordering, whereby a service can be malignantly reordered, breaking the boot sequence (NFS seems especially prone). This is somewhat inherent to parallelism in general, but is particularly undesirable given parallelism serves no other purpose beyond startup speed optimization (for which process checkpointing I maintain is a better solution).
Between all of this intermediate complexity, I can end up in stuck start/stop jobs, cyclical transactions and dependency loops without knowing the exact origin, because the graph engine doesn't supply me with sufficiently good debugging context.
And the worst part is that most of this is not even a reliability/fault tolerance thing. The Unix process model has no semblance of anything resembling a "transaction", nor is the Unit object versatile enough. It's all heuristics on a private graph engine meant to topologically sort the edges, so as to accommodate systemd's dependency system (which conflates hard service relationships with softer ordering requirements, and is generally inflexible)... yet at the same time you're actually expected to lazy load everything, but barely so in the real world.
Oh, and of course, handling anything related to Unit objects in PID1 creates an unnecessary coupling between global system state and service state in particular, on top of all the intermediate complexity of the graph.
Actually, I deleted them. Dang didn't delete anything. I'm not going to comment further on systemd, evidently it touched a raw nerve for myself as I've read all the commentary and found I strongly disagreed with the technical reasons behind systemd hate: I guess I was too overly aggressive as I'm going through a rough time right now (depression and suicidal thoughts), so I thought it best to bow out entirely as I wasn't helping the discussion and caused more heat than light.
P.S. Interesting insights though, you make a few fair points. When I'm feeling better I might ponder some more on them. Peace.
Though you seem to want systemd to start things in the same order each time and have it be predictable, that's pretty impossible. Systemd works more on the basis of "this happened; what should be done now?".
Seems adding something to record how it started things and why it did so would help quite a bit. There's some debug mode where it limits how much it starts at once. Then some way to visualize this easily (not just a GUI/picture).
Whoa, please don't drop things like that casually in the threads. People are quick to believe them, but what you said is false. I didn't touch chris_wot's comments.
We never delete comments outright unless the author asks us to, and only kill individual comments (i.e. mark them [dead]) if we are banning a user for breaking the site guidelines, which was not the case here.
As a sysadmin, I find that every time I rail against systemd here, I get constantly challenged by developers who know far more about, well developing, than I do.
Is it so hard to avoid change for change's sake? I'm not opposed to learning new things, but we're all really busy, and when the standard way of accomplishing things changes in every major release (which you're usually trying to deploy to stay on top of the latest developer demand, or audit requirements or whatever) just adds a massive learning curve. Now I've got to learn something new, rewrite a bunch of tools, all because.. why?
There will always be busy sysadmins; there will also always be new sysadmins. If we agree that systemd is a good thing, the best thing to do is to switch over quickly and bite the bullet on the re-learning problem. I understand why it happened that way and there were excellent technical reasons for it, but Debian's and Ubuntu's slow transitions (through insserv and Upstart + logind, respectively) were probably a disservice in the end, if you somehow magically knew that systemd would be the end state for both distros.
I don't expect it to change again in the next 10-15 years, except potentially to a competitor that is designed very similarly but just plain better.
I would like to see more empathy for the fact that systemd is a complex, underdocumented system. Lennart's blog series is great for the "why," but very sparse on "how" and especially "how do I fix it".
> ... adds a massive learning curve. Now I've got to learn something new, rewrite a bunch of tools, all because.. why?
As a developer I go through that "new tech" cycle every 2-3 yrs. Why? 'couse world's evolving. Some evolution cycles are not necessarily good, but we have to steer the ship forward. It's not always nice, but I don't see the other way...
> I am still to find a sys admin that likes systemd.
Here's another one: I spent 18 mos in Google Site Reliability Engineering (and did a lot of pager work at startups before that) prior to transferring to software engineering, I think I can speak to both perspectives.
Systemd is incredible and is a huge advance in maintaining linux systems. It boggles my mind that anyone could compare it to what we were all doing before and walk away saying: "nah, I'll stick with the stringly-typed shell scripts and the stale pid files, thanks."
Specific technical reasons in favor /for sysadmins/:
- unit files are rigidly structured data and this is a Good Thing. Configuration is not code. Unit files are also very easy to write.
- systemd understands cgroups and manages them properly. Every single critique I've seen of systemd is from someone who either doesn't understand cgroups or doesn't realize why they are so useful and important. Cgroups manage jailing, process isolation, resource control (CPU shares, memory limits, block io shares, etc), privilege separation and dropping, etc., and this is all configurable by systemd using simple and declarative unit file directives. Like it or not, we are moving into a containerized world.
- machinectl is already useful for managing VMs and will only become more so as it matures.
- people shit on the systemd journal, but, I'm sorry -- you're wrong about this too. Structured binary logging is simply more efficient for this problem, both on the write and on the query side. The journal also can integrate with the systemd journals running in any VMs or containers running on your system, so you can get a unified and metadata-tagged logging stream of everything happening on the host
- socket activation enables zero-downtime restart/deployment for network services, any incoming connections to the port will be added to the listen queue by systemd before the target system comes back online
I've managed Unix and Linux systems for over 10 years. I prefer systemd based systems where possible. Hit me up offline (email is in the resume linked in my profile) for technical reasons if you care.
Unit files are simple to write, and make incorporating new custom processes even simpler (does it log to stdout? It'll log to journald and then syslog perfectly fine).
Process management is built in and sophisticated (restarts, timeouts, conditions, limits etc.).
The dependency system is amazing for ensuring your system fails sensibly: I can link mountpoints to processes, and easily guarantee that a mail daemon only runs when its network mount is ready, and then make it also shutdown if that mount goes away for any reason. This is huge for self-documenting infrastructure - you can ensure that even manual intervention in the system won't leave it in a weird state, and the why is all right there.
After 20+ years working with unix, I don't mind systemd. Every init system I've used has its quirks and they all get the job done.
Overall I'll hire the sysadmin who knows the most init systems still found in common linux and unix systems and is the most ready to adapt to whichever trend wins domination. Evolve or die.
As a sys admin type, I think it's fine. I prefer it over sysV init, upstart, and launchd. There are other options, but I prefer to stick with the more commonly used stuff.
It took a while to get used to, but it does what I need very well (dependency resolution, process supervision, etc).
"Ok" with sysvinit, sort-of "like" systemd and "hate" supervisor.
Btw: almost nobody considers the point about supervisor. A lot of clients' systems I manage are sysvinit or upstart systems, which use the init system to start supervisord and then use supervisor to (poorly) manage some services.
So even if I get why someone might like sysvinit better than systemd, I have no idea how someone can like sysvinit+supervisor better than systemd.
1 http://www.gobolinux.org/?page=at_a_glance