Maybe it's a good time to say a little bit about this project.
I started it because I was unhappy with the existing documentation. Most resources were mainly for 32-bit and there was hardly any documentation about Rust OS development. One day I decided to do a clean rewrite of my experimental rust kernel and had to look up all the required information again (since I had forgotten the details). So I decided to create the missing documentation myself and write some tutorials about booting a Rust kernel.
I wrote the first posts over the summer and it was kind of fun. I never wrote a blog post before and English is not my native language, so it was pretty challenging. I spent a lot of time rewriting and improving them until I was more or less satisfied. Eventually I had the guts to post it in the rust subreddit. I never expected that it would be useful for many people (who would want to write an OS in Rust?), but it totally took off. Someone posted it on hacker news and it even was the top post for a short time. I was completely stunned :D.
I really love writing these posts. It takes much time but it's a great way to really understand the code since you need to explain every detail. It also helps to make the code as simple and straightforward as possible.
I still don't know where this will lead. But I will keep on writing as long as people want to read it :). For the short term the plan is to explore interrupts and to add keyboard support. After that, I'm open to new ideas :).
Thank you for writing this series. I love learning the high-level OS concepts alongside real, working code. It's like proof that what you're talking about actually works!
Doing it in a fresh, up-and-coming language like Rust is icing on the cake. Keep it up!
It looks like there are a lot of fast moving projects on this now[0], namely Redox[1], and a lot of research and study demo codes. Looking forward to seeing how these efforts come together.
Seeing this for the first time today and it looks great! You explain things in depth and also focus on the tools that make it work (not only the pure Rust code). Keep it up, learning Rust is on my todo list and this seems a great project to use it.
Thank you for doing this! I find this kind of stuff very entertaining. I also hope to try writing my own kernel some day, and this blog series will be a huge help when doing so
!
What exactly makes Rust attractive for low level stuff like this?
Just wondering why Rust/D get lumped into OS and low level development in general when it seems more syntactic than anything else. Do you have to manually manage memory in Rust?
Yes, you need to manually manage memory. But you can't do it wrong in Rust¹. It's an error at compile time. No use after free, no double free, and no dangling pointers are possible. The same applies to iterator invalidation and data races. Coupled with sane defaults for things such as array bounds checking and value initialization, it results in a great language to write a safe kernel.
¹: Well, you can mess up in `unsafe` blocks. It's basically an “I'm smarter than the compiler” annotation. You can use it when Rust can't guarantee that something is memory safe. For example writing to the VGA text buffer at the special address `0xb8000` is totally safe, but Rust can't know that. We try to minimize the number of `unsafe` and to double check each use, but of course you can make mistakes and accidentally violate memory safety. But even then you just need to look at `unsafe` blocks to find the bug, which is still a big improvement over languages such as C.
> But even then you just need to look at `unsafe` blocks to find the bug, which is still a big improvement over languages such as C.
This is often said about Rust, but isn't exactly true. If you'd like this to be true in your own code, you have to have be discipline and put all operations which may violate invariants required by unsafe code inside unsafe blocks, even if the compiler considers that code to be safe.
Imagine you're implementing a type like Vec<T> that has a (raw) pointer 'buf: *mut T' to some slab of memory it owns, and another field 'length: usize'. A safe method of Vec<T> could make a mistake while mutating the length field and violate memory safety in a completely correct 'unsafe' block that assumes '[buf, buf + length)' is a valid range of memory.
Isn't this still a useful feature? You still at least have a better idea of where to look compared to C. With the feature of `unsafe` blocks in Rust you have points of entry to look at, and can be relatively sure that you only need to follow the several methods that are used in the `unsafe` block at most. In C you can happily have `some_function_1()` which causes the error you're trying to debug happen but is never called anywhere near where the error gets thrown.
Tracking a semantic error in a call that's three calls away from where your unsafe block lives is much easier and structured than going through everything everywhere for any error from anything.
(I haven't done much work at all in Rust and avoid `unsafe` - and `mut` and `.unwrap`, for that matter - like the plague when I do use it, so I'm very open to being wrong in my interpretation.)
EDIT: I also think a reasonable comparison would be Haskell's `do` blocks. It's guaranteed that anything with side effects will live in a `do` block, but of course you can still have semantic errors. Again, you know exactly the "functions of interest" to look at if this is the case - there's no chance of `someFunc1` to be doing anything funny if it's never called by anything in the `do` block.
Because the Rust type system is memory-safe by default. It aims to have similar performance characteristics as C, but support higher level abstractions without compromising safety.
If you don't understand why that is critically needed, I'd like to introduce you to /almost every RCE, buffer overflow, and stack smashing bug/ that has plagued us for the past 20 years.
These are problems limited to C, C++, and assembly language. Even C programs can be kept reasonably secure by following the MISRA-C standard.
Very secure and reliable operating systems have been written in existing languages such as ESPOL (MCP) and PL1 (MULTICS). There's also Zetalisp (Genera) and Oberon (Oberon), though they might not appeal if you dislike garbage collection. You might also consider Ada, designed with type and memory safety in mind and used in defence and avionics.
> Even C programs can be kept reasonably secure by following the MISRA-C standard. You might also consider Ada, designed with type and memory safety in mind and used in defence and avionics.
The problem with these older systems is that they don't support heap allocation while remaining memory-safe. (A quick Google search seems to confirm that MISRA-C is in this category.) Or, if they do support heap allocation, they use a GC to manage it. Both of these restrictions are often considered undesirable for a kernel, except in specific embedded situations in which you can get away with statically allocating all your variables. Rust's lifetime system can ensure that you use malloc/free correctly, so it doesn't have to wall dynamic memory allocation off into unsafe code.
Rust uses RAII and ownership/borrowing for safe, performant memory management (no GC). At the same time, it doesn't automatically use memory. So, stuff goes on the stack unless I explicitly put it inside a heap-based thing (e.g. a box or a vector). These things clean up after themselves and can be used cleanly.
At the same time, since you don't have to use the heap, and low level code isn't too hard to write in Rust, you can easily write non-allocating OS-level code if you wish. And with the appropriate (unsafely implemented, but verifiable) zero-cost abstractions, you can still have safe code at this level.
Rust was specifically designed as a language for doing low-level stuff. Or at least, to fit squarely in C's domain.
Building an OS rather than applications still has some rough edges, but we're working on them. One of the reasons that I'm playing with OS stuff is to make sure we have a core team member to advocate for those features, as they are fairly niche. Most people are interacting with their OS, rather than writing one, so it's easy to put them off.
I think Steve's goal is it to make OS development more accessible to newcomers. Even people from dynamic languages who never heard of stack/heap should be able to follow it. It's a really great project that dispels the myth that OS development is too hard for the average programmer.
AFAIK it will roughly follow the first posts of Writing an OS in Rust, but at a much slower pace and with more extensive explanations. After the Set Up Rust post, it will diverge from the series and explore keyboard input instead of memory management.
Steve is a dang awesome writer (he wrote most of the Rust book) and intermezzOS looks pretty great already. I think it will bring new people with new ideas into OSdev and thus I'm really excited about it.
I started it because I was unhappy with the existing documentation. Most resources were mainly for 32-bit and there was hardly any documentation about Rust OS development. One day I decided to do a clean rewrite of my experimental rust kernel and had to look up all the required information again (since I had forgotten the details). So I decided to create the missing documentation myself and write some tutorials about booting a Rust kernel.
I wrote the first posts over the summer and it was kind of fun. I never wrote a blog post before and English is not my native language, so it was pretty challenging. I spent a lot of time rewriting and improving them until I was more or less satisfied. Eventually I had the guts to post it in the rust subreddit. I never expected that it would be useful for many people (who would want to write an OS in Rust?), but it totally took off. Someone posted it on hacker news and it even was the top post for a short time. I was completely stunned :D.
I really love writing these posts. It takes much time but it's a great way to really understand the code since you need to explain every detail. It also helps to make the code as simple and straightforward as possible.
I still don't know where this will lead. But I will keep on writing as long as people want to read it :). For the short term the plan is to explore interrupts and to add keyboard support. After that, I'm open to new ideas :).