Hacker News new | past | comments | ask | show | jobs | submit login
Spotify migrate 5000 servers from Debian to Ubuntu (raphaelhertzog.com)
90 points by lmedinas on July 16, 2014 | hide | past | favorite | 38 comments



Ubuntu LTS is a joke, really. Any major issue found in any core package is usually forwarded "upstream" (which is Debian).

Very few people in Ubuntu have, apparently, the knowledge to debug issues which go beyond simple repackaging of an updated copy. Even issues related strictly to Ubuntu (such as mount dependencies with upstart with complicated setups with multipath) do not get fixed if the issue is not resolved in newer releases and/or got already debugged in Debian.

This is not what "support" in "long term support" is. Predictable release cycle in these terms is not worth much.

For desktop systems it might be different, but then again in our lab (~50 users) most of our Ubuntu's users shifted away to Mint over the course of 3 years. Only one is still using Ubuntu.

But of course I'm biased, I'm a Debian developer myself.


> Any major issue found in any core package is usually forwarded "upstream" (which is Debian).

Of 4571 bugs sent to Debian from Ubuntu (and tagged as such), 3905 came with patches [1].

> Very few people in Ubuntu have, apparently, the knowledge to debug issues which go beyond simple repackaging of an updated copy.

The data says otherwise.

If the bug originates upstream, then it's considered just good manners to forward the bug report upstream, rather than maintain the patch just in Ubuntu. Otherwise upstreams scream loudly that Ubuntu doesn't play well with the community. You can't have it both ways. Do you want bug reports for bugs your packages, or don't you? Do you not want free QA from Ubuntu users?

As for supplying fixes, that depends on priorities, and that applies to everyone (regardless of distribution). Better to forward the bug even if it isn't a priority to fix. But where the fix is a priority for Ubuntu users, a patch is generally sent to Debian, as the data shows.

[1]: http://udd.debian.org/cgi-bin/ubuntu_usertag.cgi


Shifting away from desktop Ubuntu has probably nothing to do with release cycles, and everything to do with Unity. Kubuntu, Lubuntu and Xubuntu got very popular when Unity debuted :)


It also has somewhat to do with release cycles.

If the package you're having problems with is an old release that is not maintained anymore upstream, and/or ubuntu shifted to another package, you can kiss your "support" goodbye in most cases.

Now I'm a bit too pessimistic, because in Debian such cases do also exist, though it's much more rare due to the way the stable release is managed.

And I have to say that for true long-term support, with support meaning=capability of debugging issues in-house for all the core stack they release (most importantly, the kernel), RedHat is probably the only real choice you have actually.


> for true long-term support, with support meaning=capability of debugging issues in-house for all the core stack they release

Your definition of support doesn't match that used by enterprises like Spotify, i.e. the ones with real money on the line. They don't care whether the debugging is in-house or elsewhere; they just want the problem solved, and they want someone they can call who will go solve it. That involves paying someone directly (Canonical or other support companies), indirectly (paying your own employees to learn and manage things), or both. I'm guessing the ecosystem thing is in play here -- it's just easier to find Ubuntu-competent people/firms than Debian-competent ones.

That said, the work done by the Debian team is very valuable (thank you), which it seems ought at some level be reflected economically. I have no familiarity with Debian's financial situation but the hosting and compute resources can't come for free. A reasonable business model might be for the Debian team, or a commercial branch or spinoff thereof, or even just some motivated entrepreneurs, to serve as a (potentially non-profit) service provider for the likes of Canonical. Seems like a natural evolution to me, but as I said I'm not familiar with Debian's economics.

> If the package you're having problems with is an old release that is not maintained anymore upstream

Wouldn't Debian suffer the same problem here, e.g. if some piece of software's maintainer gets hit by a bus? Or does Debian make the commitment to take "ownership", if necessary, of all the software it bundles? The latter seems logistically challenging but I am prepared to be (even more) impressed.


>> for true long-term support, with support meaning=capability of debugging issues in-house for all the core stack they release

> Your definition of support doesn't match that used by enterprises like Spotify,

My sentence was exactly saying that: if you truly want your problem fixed, and the problems include debugging the kernel, RHEL is the way to go. There are companies that offer professional Debian support, but not at the same level as RH, sadly.

> > If the package you're having problems with is an old release that is not maintained anymore upstream

> Wouldn't Debian suffer the same problem here, e.g. if some piece of software's maintainer gets hit by a bus?

I also said that. Debian suffers from the same problem, though if the package is part of the debian "base" system in a stable release, it will be fixed, no matter what. Note that base packages with release critical bugs do not enter stable releases: either the bug is fixed or the package is dropped before hitting the stable release. Once is released, it will be supported. Most of the maintainers for the packages in the base system are also developers.

This is also one of the main reasons Debian releases are "costly" in terms of manpower, and that's why usually releases slip off by months due to release-critical bugs that stop the release process.

I personally don't follow stable releases, as I'm a developer. I often move between testing and unstable, which are basically rolling releases with different degrees of stability. But the work of the maintainers and the base system is vastly underrated, and unfortunately criticized for being a slow process.


> For desktop systems it might be different, but then again in our lab (~50 users) most of our Ubuntu's users shifted away to Mint over the course of 3 years. Only one is still using Ubuntu.

That is due to the awful UI decisions Ubuntu made.

Then again I run Ubuntu Gnome instead of Mint.

> This is not what "support" in "long term support" is. Predictable release cycle in these terms is not worth much.

That isn't related to the OP's argument. The OP is basically saying that predictable support cycles of 5 years is something Debian should do in the interests of keeping big players in their ecosystem instead of them drifting into Ubuntu's orbit. The quality of Ubuntu's LTS is irrelevant if people switch purely because of marketing.

https://www.debian.org/News/2014/20140424

It was done for Debian 6 and I'm pretty sure the OP is trying to encourage Debian to continue that on Debian 7 & 8. However, with no guarantees, people will switch to Ubuntu since Ubuntu's quality is "good enough".


Sure, I understood the logic behind that. And having a more tight release cycle will also intrinsically help Debian, since maintaining backports/patching a system for a longer time is also more demanding.

My main problem with these sentiments though is that Ubuntu still has marketing for some reason. This means that it's actually more likely that you can find your $commercial package pre-built for Ubuntu, while there's no official support for Debian. Obsolete releases for commercial software are actually helpful (less support required). Not to mention that in most cases there's zero effort in making the package exactly the same....


> My main problem with these sentiments though is that Ubuntu still has marketing for some reason.

Yes, it tried to make Linux more "mainstream" and mostly succeeded. That is what happens when you have a ton of money you can throw at a problem.

http://arstechnica.com/business/2014/01/ubuntu-maker-boosted...

It is still losing double digit millions a year. Debian can't match that kind of loss.

I don't think Debian even has $20 million in a year in terms of donations? {not counting labor)


At my company, we don't even consider Ubuntu as a viable server platform. It's RHEL/CentOS for just about everything.

As you mentioned, the LTS Ubuntu versions are really not on the same level as the stable RHEL releases and support that is provided.

Yes, Ubuntu is very popular... in both the Desktop and Server arena (mostly catalyzed by EC2 making Ubuntu the default Linux image, even though EC2 is built out of RHEL/CentOS boxes). However, most "big metal" servers are RHEL/CentOS or something else.

In any event (and distro-fany-boyism aside ;-) ... one has to question the change from Debian Stable to Ubuntu LTS. The only thing I can think of, is they wanted a more "proper" company to call up for support... even though Debian are the guys Ubuntu calls up when they need support.

At the end of the day, my 2 cents would be to get a distro that focuses on being a server distro first and forthright, not a distro that is a jack-of-all-trades.


It's weird also, given that spotify were pushing systemd[0] hard, shuttleworth[1] conceded - but the version of LTS Ubuntu 14.04 [2] that spotify switched to is on Upstart.

[0] http://www.muktware.com/2014/01/spotify-uses-debian-endorses...

[1] http://www.markshuttleworth.com/archives/1316

[2] https://lists.debian.org/debian-publicity/2014/06/msg00014.h...


> ...even though Debian are the guys Ubuntu calls up when they need support.

Nonsense. Citation needed, please. Many key Ubuntu developers are also Debian developers, so I'm not sure what calling up Debian guys even means.


One reason they switched is because of Docker issues on Debian. Source is here (@4:46) https://www.youtube.com/watch?v=pts6F00GFuU#t=284. Whole video is worth a watch though.


RHEL 7 (and now CentOS 7) has native Docker support built in... seems that would have been a good choice of distro if they were looking for good, long term Docker support.


Indeed. We've been running RHEL kernels with a Debian userland for a looong time.


Not to mention that Debian consistently supports "in-place" upgrades when moving between releases, minimizing admin time doing the upgrade, and actual server downtime. Whereas Ubuntu... I tried using it on a few servers, their complete lack of in-place support meant the servers had to be rebuilt from scratch on a near-yearly basis if we wanted the latest release. Just not feasible.


> their complete lack of in-place support

How so? In what way does Ubuntu not support this?

Not to mention: if you're not at least using configuration management by now, you're doing it wrong. Rebuilding a server should be a completely automatic process.


Typed too hastily, meant "in-place upgrades".


In fairness, RHEL/CentOS did not support in-place upgrades between major releases until RHEL/CentOS 7 (just released). Prior to 7, it was recommend to always do a fresh install.


...which Ubuntu supports.


My reading of LTS has always been that they will continue to provide the updates from upstream, whereas a release that has past end of life will recieve no updates.


We're still running Debian Squeeze in production at one of the places I work. It had one advantage recently: The SSL package was so old that the heartbleed vulnerability hadn't yet been introduced, lol!


I have a client who was in the same position. They actually still have a few boxes on Etch... and amazingly, one Sarge box:

$ cat /etc/issue

Debian GNU/Linux 3.1 \n \

They're all running a very old hacked-up version of torque that no one remaining at the company can do anything with. At least they're walled off from everything else, I suppose?


One thing I don't understand is why companies don't pay Canonical for support? Is it such a bad thing to pay for Linux support?

I wonder if companies that have chosen Ubuntu LTS think they are supporting Canonical and its developers just because they are running Ubuntu. This article doesn't mention Canonical at all.

Support Debian developers is a great thing but Canonical probably spends tons of money and I rarely see companies saying "I chose Canonical and Ubuntu because...".

DISCLAIMER: I work for a company that provides Linux support services

EDIT: Typos


> One thing I don't understand is why companies don't pay Canonical for support? Is it such a bad thing to pay for Linux support?

I've never needed paid professional linux support. Or windows support. Or mac support. Or android support or any of that.

I think the issue is...once you develop enough in-house Linux knowledge you don't really derive a benefit from paying for support.

Tbh, the only kind of "linux support" I'd really want is overnight sysadmin support so I don't get calls at 3am. I suspect that isn't the kind of support you have in mind.


All software has bugs, and professional end users often hit edge case bugs that affect few others.

Do you push fixes for these back into your distribution (or upstream and cherry-pick into the distribution, etc), or do you maintain your own private repositories containing your modifications?

If you do the latter, then you have extra maintenance costs. If you pay for support from a distribution vendor, then you save that cost. Even if you have the expertise to do everything yourself, it still costs you time (and therefore money). Working with a distribution vendor amortizes that time between all of their customers, saving everyone money.


> All software has bugs, and professional end users often hit edge case bugs that affect few others.

I'm kind of amused by the implication that I'm not a professional end user.

> Do you push fixes for these back into your distribution (or upstream and cherry-pick into the distribution, etc), or do you maintain your own private repositories containing your modifications?

Or maybe, y'know, a third option where one picks software that is highly reliable for your use case.

You are suggesting a very false choice. If you have unique needs that frequently cause you to encounter edge case issues, you need someone in house who can fix those on your timetable and not a vendor's.


Nope. Not at all.

Backporting upstream patches by request, enhancing upstream, help debugging difficult, multi-level issues and so on. I also took it for granted but seeing what I see today, I understand.


> Backporting upstream patches by request, enhancing upstream, help debugging difficult, multi-level issues and so on. I also took it for granted but seeing what I see today, I understand.

I guess to me if you don't have the expertise in-house as a tech company, I think you have issues because you are outsourcing core competencies [the ability compile & maintain linux related software, the ability to debug complex issues that are likely part of generating your competitive advantage in someway].


There are silent corruption bugs in storage drivers for example. Having that kind of competency in-house is too expensive for most companies.(Not Spotify in this case, but problem could be much harder to fix)

http://labs.spotify.com/2013/01/28/finding-the-right-needle-...


That is the description of a hardware problem that the vendor they bought the hardware from was unable to resolve due to an unsupported OS.

That is where picking the right hardware vendor comes in. I wouldn't ask my OS vendor to support my hardware choices unless they specifically certified it. Neither Debian or Ubuntu certified that raid card...which puts you back to square 1 again, rendering their support moot.


As someone who admins a bunch of Debian powered servers, it would've easier for me if the Debian LTS project had been announced last year instead of earlier this year.

The 1 year gap between the release of the new version and the end of support for the old version just wasn't long enough. Sadly I'm also switching servers to Ubuntu because of it - along with a push towards using Docker containers powered by distros with long term support.


My team switched from Debian to Ubuntu a few years back for precisely for this reason: a predictable update schedule. Dealing with support isn't really my concern. I'm able and willing to workaround whatever edge-case issues arise, but when I deploy an OS on a box I need to know how long I can plan on the core OS elements being updatable. I need to be able to schedule mass OS upgrades months in advance. Ubuntu provides that capability, Debian has not. Though it sounds like Debian has woken up to this issue and is finally addressing it.


TIL Spotify has 5000 servers ... somehow I expected they'd have more.


They have more. A guess would be about the double.


I see. Did you used to work there?

Why were the other servers not upgraded?


I'd love too, but no. But I have a close friend that works in their SRE team. I also meet lots of Spotify people at meetups and such as I live in Stockholm.

My guess would be that they will not migrate their data-related machines (Postgres, Cassandra etc) as the reason for going to Ubuntu is partially for Docker and using Docker in their testing and staging environments (to begin with).

The other one being that they don't have anyone maintaining an patched inhouse kernel for debian anymore (or they don't want to do that).


Here are some actual #s, a few months old: https://www.youtube.com/watch?v=pts6F00GFuU#t=41




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: