I miss the Eudyptula Challenge [0], I never got around to complementing it. It was a series of 20 challenges to introduce you to Linux kernel development, and it was done entirely over email. I think it's been shut down for a few years now, I'm not sure when or if it'll come back at this point.
Aw man. That sounds pretty cool. I guess there isn't anything comparable out there? I'm interested in contributing to the Linux kernel, but it's just so hard to get into it.
The Linux Foundation recently released a free course as well titled "A Beginner's Guide to Linux Kernel Development" - haven't taken it yet, but another resource for the community:
I have looked at this one, it is useful if you do know about Linux internals already. It just introduces you to the development process (how to commit your code and etc.)
Started getting into kernel stuff for work. Does anyone have a good idea what kind of market demand there is for kernel devs or similar skill sets? Being new to this I absolutely love it, but have no idea.
The resources shared by everyone here are so helpful! I’ve been muddling through by reading the docs, source code, and various messages on the mailing list and... it’s been a little painful.
As long time kernel/OS dev, right now is not the best time for kernel jobs. Majority of jobs are in user space (particularly Node/JS/React or aws/Kafka/Hadoop type) or at least the interviews are geared towards them. Most companies which used to maintain their own kernel have either outsourced it or use cloud where minimal kernel change is needed. Even if a company has kernel/OS team, majority of their work is to debug, patch, and port. Usually even they don't want to touch kernel much since, it is super critical component of product, debugging is really hard, getting company specific patch upstream is hard, etc.
Obviously, RedHat, SuSe Canonical, Microsoft, Apple, AMD, Intel, firmware and embedded companies will always need kernel people. However, the demand has really gone down and it is no more considered "elite" team within companies. Jobs are still there but obviously teams lean towards hiring experts and veterans of kernel instead of new comers. Moreover, once you get over simple char driver and few lines of patch type stuff, kernel development is extremely complex and hard. You wont get much support either as very likely your problem will be unique to your company's hardware. So, my take - learn and do kernel stuff on side but avoid getting into it full-time unless you are really passionate about it.
Security product companies like mine (Capsule8), Crowdstrike, Sysdig, etc. will always be looking for software developers with a background in Linux (or Windows) development.
We're hiring. :) Email me (address in bio) if you have experience doing kernel development, reverse engineering, debuggers, emulators, compilers, etc. We do cool stuff.
Consistent kernel hackers get hired fast. Focus on a particular subsystem, participate in the community, and contribute regularly - you'll have opportunities in no time.
what would be a typical salary range for 1 year kernel dev + 4-5 years in related experience? let's say BayArea/NY/Austin? I really did like learning about kernel dev before. Might be time to learn again..
That's less than somebody with 0 years experience in FAANG will make. I wouldn't expect kernel developers to get paid much differently than non-kernel developers at a similar level.
It's really just down to location, then. They vary so much, and I don't have a lot of experience in the locations you mentioned. You can expect high-end software engineering salaries for good kernel hackers.
If you're doing reliability engineering for any type of service running on Linux servers, being well-versed in the kernel is a great power that separates you from others.
IBM and Redhat (although I guess nowadays they're pretty related) seem to do a lot of kernel work. There are probably even more that I cant named sleep deprived, but you can throw in something like "linux kernel developer" in a job search board to get an idea
The direct demand is low; the Linux kernel is open source so framing yourself as a 'kernel developer' probably is not a very good way to be an ideal candidate for a job opening.
On the other hand, most developers and even some non-developers recognize that kernel development is hard. If you have a name in the kernel development world this will certainly help your reputation. Likewise, if you have a good knowledge of Linux this will certainly help you for certain job openings (kernel development is probably a great background for most low-level job entries).
Someone here probably knows this, so I figured I'd ask: I submitted a small patch to the kernel a while back and I made a typo in it and it broke the tests, and then I couldn't figure out how to get the test bot to pick up my new patch and by that point I assume anyone who would possibly have reviewed my patch stopped bothering. Is there something I should do to get it looked at again?
If your original patch contained a mistake, you just send the new version with the v2 prefix to the mailing list. You can do this with git format-patch -v2 and then send-email will pick it up automatically.
You should note the change in the individual commit or the cover letter.
I had Al Viro CC'd on the emails when I sent it to linux-fsdevel. Are you saying I should send it to him again, directly? (The patch, for reference: https://github.com/saagarjha/linux/commit/4867a403decc364c8b.... I figured I should also take this opportunity to check to make sure there's nothing wrong with it that might cause it to be ignored.)
The idea may be to avoid shifting a 1 into the sign bit, which is undefined behavior.
The values in question are being convertd back to loff_t anyway, but that's an implementation-defined conversion rather than undefined behavior.
So that is to say, doing the shift in the corresponding unsigned type and then converting to the original signed type is better defined at the language level than just doing the shift in the signed type.
I think this is just academic, because shifting a 1 into the sign bit behaves the same way on all platforms targeted by Linux using the only compiler that you can use.
Well, or maybe not. The one problem there is new changes in GCC which assume that all sorts of things that used to work (and were thus de facto implementation-defined) are no longer happening in programs (because they are undefined behavior, so who would do that?) Thus even if the target hardware has stable two's complement behavior on a left shift that alters the sign of a value, and that has worked fine for years, an updated GCC can suddenly decide that it's okay optimize based on the idea that the program doesn't do such a thing.
On a different note, the proposed change is jarringly ugly because u64 is size specific, whereas loff_t isn't. A long long isn't defined as 64 bits; it is at least 64 bits. On a system that, say, makes long long 128 bits, the proposed code change is not correct. To make this change with the proper elegance and precision, what you need is for loff_t to have a sister unsigned type that pairs with it: a uloff_t.
> Well, or maybe not. The one problem there is new changes in GCC which assume that all sorts of things that used to work (and were thus de facto implementation-defined) are no longer happening in programs (because they are undefined behavior, so who would do that?) Thus even if the target hardware has stable two's complement behavior on a left shift that alters the sign of a value, and that has worked fine for years, an updated GCC can suddenly decide that it's okay optimize based on the idea that the program doesn't do such a thing.
Note that Linux compiles with -fwrapv or one of the similar flags that defines overflow to wrap, so this isn’t an issue. I guess this trivializes this patch more, but I figured that they might want the patch anyways in case they ever move off of the flag. (I couldn’t find a uloff_t, so I copied what other code seemed to be doing in this case.)
I’m not at my computer, so I can’t double-check this, but IIRC the issue was that ((long long)0xffffffff << 32) is undefined because it shifts past 2^63-1, which causes an overflow of the long long. (I know the undefined behavior sanitizer flagged some equivalent code in a Linux reimplementation I was working on, which is how I noticed it in the kernel.)
Awesome! I can't wait to dig through this, I've been wanting to contribute to the linux kernel for over 2 years but no one on the linux mailing lists seemed interested in helping a newbie (so much for kernel newbies). Thanks for putting this together.
Or pick a random SBC from Aliexpress or something you have catching dust at home, and try to improve the mainline Linux support for it. That will get you opportunities to learn more deeply the various kernel subystems, too. Especially around driver and arch code. That's what I did anyway.
I pretty much agree with this. For example, the BSD Handbook and documentation is the point of call for a newbie entering into the community and the codebase is much more readable than the Linux kernel.
Last time I checked, the kernel-newbies site doesn't seem to be beginner friendly and aside from following the IRC channel, the mailing list is still the way to do code reviews there, which is quite frankly pre-historic and very unfriendly for beginners sending patches at best.
At least for FreeBSD, they uses Phabricator for code reviews, HaikuOS uses Gerrit and SerenityOS uses GitHub and they seem to be the other OSes that have both good documentation about their kernels and their userland APIs. Perfect places to start learning about operating systems and kernels in general.
Prehistoric? It is actually how the Git workflow was designed to work, and it is a standard method that does not require third-party services nor tools. Just your mail agent.
Sending patches is as easy as sending one in GitHub or any other web-based system. The problem is that you (and many others) have never done it, and so anything different is harder.
You should consider that if learning a handful of CLI commands is such a problem for you, perhaps it is not the system that is being unfriendly to you, but that you are unfriendly to learning anything else that is not your way.
> Sending patches is as easy as sending one in GitHub or any other web-based system.
Its still prehistoric in projects like the Linux Kernel, from a beginners point of view compared to GitHub, Gerrit or GitLab, which most of the active contributors are already working at large companies who's work requires contributing to it, thus already have invested in time and money learning the contribution process. To look at the LKML and do code reviews via back and forth emails looks simple for a veteran Linux developer at a large company but very arcane for a student sending a patch to the kernel.
> The problem is that you (and many others) have never done it, and so anything different is harder.
Your assumption is quite funny here as I have done both and I can tell you that most of the beginners particularly students are introduced into open-source via GitHub or even GitLab these days and its used as a starting point into contributing to a OS project. IIRC, ReactOS and FreeBSD has retained more student contributors from Google Summer of Code than say GNU/Hurd, by both improving the contribution process for beginners. I still wouldn't recommend beginners to learn about OSes in general by contributing to the Linux kernel due to the above reasons. Project making contributions more accessible for beginners isn't a bad thing, its actually how they attract potential long term contributors to stay on, rather than to make things simple only for veteran developers.
> You should consider that if learning a handful of CLI commands is such a problem for you, perhaps it is not the system that is being unfriendly to you, but that you are unfriendly to learning anything else that is not your way.
There's my point, it is still beginner unfriendly. Beginners these days would start with Github or Gitlab with a GUI to send PRs to a different OS project and wouldn't bother learning tons of CLI commands to send a patch in a email + code reviews. It is therefore favours veteran developers at large companies who stay on being long term contributors than the old days of random Linux enthusiasts doing this, which I'm very surprised you don't see the contribution process prehistoric from a beginners eyes. Perhaps something needs to change...
Take GitHub, and boil it down, and it's a mish mash of version control, diff, and patch, all plastered over with a shiny HTTP veneer.
Diff/patch, git, and email alone are the basic tools you need. Even before VCS was a really big thing, that is how software changes were propagated around.
There is nothing wrong with sticking with the basics. In fact, I flat out refuse to tell newbies that they should host everything on GitHub, or GitLab. It's becoming a twisted new form of "social media-esque" hype.
There will always be those who don't like using the mailing lists for back and forth reviews and the same goes for GitHub (I use both anyway and have a self-hosted community GitLab and cgit). This isn't about me, its about attracting potential contributors to the Linux kernel in the long term.
If I were tasked to attract as many beginners/students to go this path, It won't be to start with the Linux kernel or the LKML mailing lists (that will put them off). It would ultimately be other beginner-friendly similar projects like SerenityOS, HaikuOS or some of the BSDs even, etc so they can get some experience. Then their mental model is used to the Git flow so they can transition to using bare Git commands and can go into a technical discussion involving anything about kernel internals.
Since Linux doesn't need to worry about attracting new outsider contributors due to commercial contributors doing most of the work, little needs to change. Which is why from the very start I much rather recommend beginners to start with other OSes and transition later to Linux internals.
Well, if you want to be more likely to get reaction from any one of 10 or so odd people the typical patch series goes to directly (sometimes simply because they may have contributed some code to some file you're also modifying now) + hundreds more on a bunch of subsys and more general mailing lists, forcing them to register on some random web service of your choice and then monitor it for further communication and replies and patch series revisions is not the easiest proposition.
Everyone knows how to reply to an e-mail. It's very low effort way to find interested people to review your patch/and get some reaction from maintainers.
That's how it is now. That's not to say, current system is not somewhat problematic to some maintainers/subsystems, but that's not really your businesss as a contributor.
I'm not sure why you think I have a problem with mailing lists or not (I regularly use them myself), when the point of this discussion is about starting points for interested folks in Linux kernel or OS development in general. I just made a simple comparison on how different projects do code reviews from an outsiders perspective.
The clear misinterpretation happens when I said 'beginners eyes' to assume that I'm some sort of 'beginner', when in fact as a maintainer I keep hearing the opinion of students and newcomers making this comparison and they use their preferred way to contribute (Github being mentioned often) when joining a project. Therefore, you giving a very naïve assumption that I somehow wanted to force everyone to '...register on some random web service' when I clearly said that beginners should start with a similar open-source project using similar tools like Gerrit, GitHub before trying something harder.
Yes, I have confidence that everyone knows how to reply to an email. But in comparison to tools like Gerrit or Github, I won't expect many non-commercial contributors to stay for long if the review process was via mailing lists, unless they are paid to work daily on the project, which is why I recommended beginners other OS projects that have a similar review process before going onto looking at the Linux kernel.
You assume contributing via mailing lists is hard, which it is not. In the project the size of a linux kernel it would be failry similar in complexity, and probably way harder to use anything else due to distributed nature of the project. (which I think is unreasonable to think will change)
I contributed various things to the Linux kernel over the years across many subsystems. Each time I only had to prepare a list of recipients and send the patch series to all of them. Often times the patch series involved multiple subsystems.
In the world where one maintainer is using github, the other his private gitlab instance, one is using gerrit, and the third gitea or whatever, and some holdout still accepts only mailing list contributions,... contributing for people like me who don't get paid for most of their work would be so convoluted, that I would not.
While learning how to send a patch series via e-mail, while it takes some time to learn, allows me to contribute to any part of the Linux kernel via a single workflow, that is easy to replicate.
> You assume contributing via mailing lists is hard, which it is not.
Well, some OS projects optimise for attracting newcomers whereas some do not and clearly the Linux kernel project doesn't, hence the higher barriers for entry and its even reflected in the comments in this HN thread.
If the goal was to bring in commercial contributors into the Linux Kernel then this is not a problem. But, the Linux kernel isn't in any contributor shortage nor it is hoping to attract non-commercial contributors because they don't need to. For any other less popular open-source OS projects using mailing lists, why do you think they either moved or now do code reviews, contributions on GitLab, Phabricator, Gerrit via a web interface in the first place?
> In the world where one maintainer is using github, the other his private gitlab instance, one is using gerrit, and the third gitea or whatever...
Well guess what? Gerrit, GitHub and GitLab all support send patches via email. The problem is not about preference, its about the starting point for beginners unfamiliar with OS projects and its certainly not the LKML and the Linux kernel. I instead recommended easier OSes (SerenityOS, Haiku, BSDs) to start with by using similar tools (Gerrit, Gitlab, Phabricator) for code reviews before looking at LKML and Linux.
Since the discussion is for attracting beginners into becoming long term contributors to open-source OSes, the starting point is not Linux or LKML. They first should look at easier OSes on GitHub/Lab/Tea or Gerrit to focus on contributing patches without trying or thinking about patch formats or top-posting, rather than starting in the deep-end, frustrating them with do's and don'ts for sending a patch and risking to putting them off altogether.
If you cannot by arsed to learn a few new development tools or workflows, I am sorry but this field is likely to burn you out very soon, be it Linux kernel development, the next shiny GitHub replacement or the new edition of HTML/CSS/whatever.
And let's not start about learning the myriad of tools you gotta learn to do in professional Linux kernel development (or any kind of specialized development for that matter).
If you want to educate your students on being one more cog in the machine and make them complacent on learning the bare minimum, that's on you. I for sure wasn't taught like that.
This isn't about me or what workflow I can learn. It's really about the entry point for beginners contributing to operating systems in general and making it accessible for them. Not just Linux.
On that matter, the barrier for being a long term Linux contributor is still very high and this is from the perspective of outsiders in general. So unless you're working at a company that does professional Linux development with the hardware and software resources available to you, it is actually the outsiders that will be definitely burned out and won't bother with contributing anyway and will go to another OS kernel project to start from.
Unlike you, I don't push beginners into the deep end for them to later drown. I get them started on an easier yet similar path where they use similar tools which eventually they later become potential Linux contributors at a company working on the kernel. But the starting path is certainly not Linux.
And why should beginners be contributing to the bloody kernel? Why should kernel devs (paid by companies) spend resources on making it easy for beginners?
Here is the thing: kernel development in the most widely used production one is not a trivial thing.
It is like pretending that architecture students should be helped to go work on the next biggest skyscraper as their first project and spend the time of actual architects to make it easy for them.
If you want people to get into kernel development, there are kernels wrote specifically for teaching purposes.
Just read all my posts again and then you will realise that you constructed a strawman argument and addressed a point I never said. No-one directly said that beginners should be contributing to the Linux kernel when I actually said that beginners should get experience from joining other similar but easier open-source OS projects before joining a company specialising in Linux development to be long term Linux contributors.
Technically, anyone can contribute to the Linux kernel due to its open-source nature given that their patches are reviewed by the maintainers and it is then accepted.
> Why should kernel devs (paid by companies) spend resources on making it easy for beginners?
Not only I didn't say that, but you do realise that there are companies that exist who both contribute to the Linux kernel and offer training for a one time fee as an introduction into kernel development? Given the beginner meets the experience pre-requisites: Bootlin, BayLibre and several others have professional kernel contributors that do this.
> If you want people to get into kernel development, there are kernels wrote specifically for teaching purposes.
Read all my posts again. I go beyond just 'kernels' and named other complete OSes to get OS development and kernel experience from before joining a company specialising in Linux development. I said in the last post that '...the starting path is certainly not Linux.', it seems you certainly didn't read that.
Every professional Linux kernel developer started as a beginner somewhere and certainly wasn't born an expert. Some beginners were university students who either joined Linux companies by participating in programs like Google Summer of Code or took a paid Linux kernel training course by a company specialising in kernel development. You will notice that your last comment is in agreement with my whole argument.
Great, thanks for this contribution. So helpful this just came at the moment when I got my hands on The Linux Programming interface book, hopefully this will all help myself to contribute, thanks guys
BPF has some pretty good docs out there if you just poke around the BCC GitHub repo (libbpf-tools/ is a good one) you’ll find a lot of them. I was able to start building programs using libbpf, but a lot of the “understood” info about the kernel is what’s been slowing me down, so I’m psyched to go through all this material and learn.
[0] http://eudyptula-challenge.org/