I just started porting an application using corrode on a C library at work. This was more of a POC, to show the team. Based on that, I think this book is missing a section on incremental conversion.
You don't want to convert entire applications at once, you want to do it a source file or module at a time. This means incorporating the Rust toolchain into the build of the C/C++ project Makefile, or etc., system.
It is not difficult, and will increase confidence in the final product. This can even be used to test incremental conversions in production, rather than having any doubt about the entire conversion.
This is an awesome start for a great manual on the area. I'm very inspired by the experience I had in doing this, and hope that we do more.
> I just started porting an application using corrode on a C library at work.
Can you tell us a bit about your experience with Corrode? The Readme itself states that it's not expected to work on "real" code, and there are open issues like this: https://github.com/jameysharp/corrode/issues/52 titled "Arrays are broken!" in which Corrode's developer wrote half a year ago: "The current hack gets arrays translated correctly in function arguments, but gets them wrong everywhere else."
In your experience, does it work better than this makes it seem?
(To be somewhat fair, the issue does have detailed discussion, and partial fixes seem to have been merged, but it's still open.)
Most of the issues I ran into were not exactly with the corrode tool, but with compiler options, i.e. passing the DARWIN_C_LEVEL flag for macOS (my main dev machine). Actually, the library I hacked on was only built for Linux, I spent a lot of my time converting the C to multi-platform, not because it was necessary, but because I didn't feel like running a VM.
What I found is that corrode produces mostly compilable code. I had played around a bit with FFI from C to Rust before, so the task of integrating a Rust library into a C codebase was already familiar to me. I would love to see a guide for doing that in corrode though, like an example project, for others. If I have time I might do that. One really cool thing, is that the Rust compiler detected some undefined behavior in our C code, which I filed a bug for to our team (it was an uninitialized stack variable, passed as a pointer to a function; in practice it would never actually rely on the undefined behavior since all the fields of the struct were set in that function, but it's still poor form).
There are some things which you will need to massage, but I found in general that it wasn't much. I had a problem with null function pointer creation that I need to go back and look at.
I think my biggest issue, is that every conversion produces unsafe code in Rust. It's left up to the developer to figure out how to convert it to safe code. I think this is ok, but unsafe is still dark arts for me in Rust (meaning that I don't completely grok the best practices right now). I need to go back and reread the Rustonomicon: https://doc.rust-lang.org/nightly/nomicon/ , as there may be more details there than the last time I checked it out. I would love it if corrode could try and produce more safe blocks of code, but I also understand why it's this way.
Anyway, I think it's an awesome tool, works 95% of the time, and you can probably easily come up with some standard techniques for going back to fix up the commonly incorrect cases.
I'll reiterate this: integrate the Rust code into your C build right at the beginning (I used Cargo + staticlib), I even made this optional in my setup, so that you can flip between the two implementations easily for testing. I set up a make recipe which converts the C src to Rust with corrode (to capture all the CFLAGS needed), and that will only run as long as the .rs file doesn't exist. I found this to be very powerful for verification. I'd highly recommend corrode at this point, but don't be misled to believe that you won't need to make changes to the produced code.
Edit: I should mention that to get all this setup, with all the C library fixes in macOS included and the Makefile integration, it took me 6 hours to get setup, with about 10-15 minutes spent actually making the produced .rs file compile.
>This means incorporating the Rust toolchain into the build of the C/C++ project Makefile, or etc., system.
One warning I can give you based on recent HN posts:
Embrace Rust fully. Many expect Rust to be a "safe C++" or something of the sort, and try to port not only the code but their way of thinking into rust... which leads to pain and frustration.
You seem to have lots of experience so maybe you already know that, but I'm pointing it out just in case ;)
Well, I've been working with Rust now for going on 2 years. And I know C and have very outdated C++ experience.
I'd say C best practices probably translate better to Rust than C++, but with solid systems knowledge, as long as you don't fight the language, and instead try and embrace it, I think it's fine.
I think a lot of people want to bring old patterns to Rust, some translate, some don't, but fundamentally the memory system is basically the same.
The guide really isn't written yet. After the introductory material, it's mostly a list of 'todo' items. [1]
The "corrode" translator apparently just compiles C to unmaintainable, unsafe Rust. Raw pointers in C become unsafe raw pointers in Rust. That's probably not too useful. See [2]. Now you have heavy technical debt. Also, losing all the comments during translation is cruel to later maintenance programmers.
Good inter-language translation for programming languages is very hard. Usually, you lose the idioms of both the source language and the target language. There are few, if any, good systems for this.
To do this well, you need verification-like analysis of the source program. You need to know if a pointer can ever be null. You need to know how big a passed array is. You need to know if something has single ownership. If an analyzer could look at a C program and answer "yes", "no", or "don't know" to those questions, with a reasonably low percentage of "don't knows", then you could do a translation to proper Rust.
You could brute-force some of this by running the Rust compiler from the translator. Start out with everything being a single-owner reference, and compile. Anything that generates a borrow checker error gets turned into a reference counted item on the next try.
It's too bad there aren't more Corrode examples on line. But from what's available, it just passes C raw pointers through function calls as unsafe Rust raw pointers. The interfaces generated are no better than they were in C. You can't rewrite one side or the other and get safety, because the interface itself is unsafe. That's no good.
This isn't a syntax problem. It's a semantic problem.
The interface is FFI, it has to be unsafe, because you can't guarantee anything about the C code.
The rust code though, can be converted to pure Rust. A strategy I use and most other FFI code I've seen, is to have a shim layer, which is unsafe, that calls through to safe code. And vise versa for calling out to C. The function definitions don't need to be unsafe, just the direct calls to C and conversions to C types.
> This isn't a syntax problem. It's a semantic problem.
I never said it was.
You can still make trivial semantic transformations on code by hand or by using a tool (corrode could do it itself, too) and make it safe. As bluejekyll said it's a bit tricky since you can't guarantee anything about the calls until you port those too.
I find that most of the arguments given against C++ (such as memory leaks or dangling pointers) can be avoided by using Modern C++ features, like never allocating memory with new but using std::make_unique or avoiding null pointers with not_null from GSL or std::optional.
Those things help but it's still possible to return a pointer to an inner part of an allocation(say an element in a vector) and then have the vector go out of scope.
Rust is much more robust about catching those cases and verifying that you don't have objects outliving their references.
> Those things help but it's still possible to return a pointer to an inner part of an allocation(say an element in a vector) and then have the vector go out of scope.
That's something that the programmer needs to specifically ask for, and I'm not aware of any C++ compiler which doesn't throw a bunch of warnings regarding an object's scope and lifetime.
That means your point boils down to arguing that if a programmer really wants to, he can still shoot himself in the proverbial foot by writing C++ code, but in order for that to happen he needs to have a death wish and intentionally avoid all warnings and safe practices to begin with.
True, found similar bugs in my own C++ code, and I read and fix all warnings.
However, I never spent more than 10 minutes fixing that.
Debug version of MS C runtime fills freed heap memory with a magic number 0xFEEEFEEE. You’ll get exception in runtime, and you’ll immediately have the idea what’s wrong with the code.
Yeah, this is a trivial example. It becomes trickier when you start dealing with things across library boundaries where you might not have source code.
I've seen 5 minute fixes and things that took weeks to track down(yay for non deterministic repros).
Yeah but we're talking about lifetime bugs here. In the case of Rust they are encoded in the function signature so you don't have to rely on convention or documentation.
In most cases you have the source code you’re building. Combined with some stuff in C runtime, this pinpoints the C++ problem nearly as accurately as Rust’s compiler would.
Most commercial libraries are written in C or C++. The reasons include performance, smaller runtime, and ease of use (all OS kernels are written in C, so the majority of languages already feature decent C interop). Even if you code Rust, you won’t have that compile-time checks inside proprietary third-party libraries.
So if I can summarize: If you don't use Rust you won't see the benefits of Rust? ;)
FWIW it's getting pretty common in the Rust ecosystem to wrap 3rd party libs with proper lifetime annotations with opaque types when possible. Things like OpenGL, JNI bindings and whatnot can still benefit from lifetime annotations the compiler provides.
Rust proponents often overestimate practical impart of those C++ safely problems fixed by Rust.
At the same time, they often underestimate or ignore Rust’s problems caused by the same safety thing. The main of which is, it’s hard to implement some widely used data structures in Rust. Here’s my old thread where I discuss the topic in details:
All the exploitable UAFs in shipping browser engines for years and years are a strong counterexample to the idea that UAF isn't a problem in modern C++.
Out of curiosity - do the codebases for modern browser engines routinely get subjected to static analysis tools and runtime tools like asan (Address sanitizer) / valgrind?
Are there categories of UAF that would not be caught with valgrind / asan? Or is it the case that the test suites do not have sufficient coverage to exhibit the issues?
Requires manual annotation and isn't nearly as systematic and comprehensive as the Rust compiler, but I've found threading bugs with this in existing code that I was about to modify.
Let me share a sanitized C++ memory corruption story.
We were testing our application the night before a big demo for our client. Ours was a C++ OpenGL application with several "phases" the user could enter from a "main menu" type screen. Our demo explored these phases in a pre-planned order. All went well, until somebody decided to try doing the demo in a slightly different order. Once the tester visited a certain phase of the application, the thing crashed and burned. The stack trace was littered with some incomprehensible container-implementation-detail junk. Debuggers weren't pointing at an obvious sign, other than that something was dereferencing a garbage pointer.
We noticed that if we went back to the original canned order of operations (this was a demo, after all), the application didn't crash. But this still made us nervous. We found that we could sometimes swap a few phases early in the application's run with no ill effect.
Here's part of the class responsible for some of the higher layers of the stack trace, from one of the lower level libraries we build all of our applications on (this is important because, for hysterical raisins, we can't source-step debug those libraries in production). Can you spot the bug?
#include <map>
#include <string>
#include <vector>
struct Message;
class MessageCentral {
public:
void HandleMessages() {
// messages were collected somewhere and organized by their queue.
const std::vector<std::string> queues_with_new_messages;
for (const std::string &queue : queues_with_new_messages) {
const std::vector<Message> msgs; // = messages_for_queue(queue);
const auto &handlers = handlers_[queue];
for (auto &h : handlers) {
for (const auto &msg : msgs) {
h(msg);
}
}
}
}
void RegisterMessageHandler(const std::string &queue_name, std::function<void(Message)> handler) {
handlers_[queue_name].push_back(handler);
}
private:
std::map<std::string, std::vector<std::function<void(Message)>>> handlers_;
};
(Note: this is probably way easier to figure out with all of the other code needed to implement MessageCentral taken out! It took us three engineers working for about 2.5-3 hours each, with lots and lots of gdb and printf debugging, to nail this one down. This particular crash would have been impossible in Rust, because the compiler would have rejected the Rust equivalent of the above C++ code.)
(oh, by the way, we couldn't use runtime memory sanitizer tools like ASan or valgrind because the application linked to CEF. In retrospect, we could have dropped in a fake CEF that didn't interfere with ASan or valgrind, but that would have required build resources we didn't have available in the frenzied night before a big demo.)
I assume one of your handlers invokes RegisterMessageHandler invalidates the iterators to handlers_[some_queue_name]. As such, the following line exhibits UB:
for (auto &h : handlers) {
In practice, the exact behavior is likely to depend on if the vector resized it's underlying storage or not.
EDIT: I'll also note that using a debug build of the SC++L with iterator debugging enabled should give you a useful diagnostic here - given a modern SC++L implementation. Whether or not your application runs well enough, to actually get to the point of being able to reproduce the bug with iterator debugging enabled, is another matter entirely of course.
EDIT x2: I'll further note I'm cheating here by having encountered and fixed this same bug on a different codebase.
It only works if the whole company buys into it, including only linking against libraries whose authors also adopt those practices.
What I have learned since learning it in 1993, using it including teaching the language to first year students, is that any safety feature that isn't imposed by the language instead outsourced to best practices or external tools tends to be ignored by the majority.
What you are saying is probably true but it's also sad. That's why we end up with languages without preprocessor or multiple inheritance because some people misuse these extremely useful features.
Is there a compiler option in C++ to only allow this safe variant of the language to be used? I find this argument to be a false comparison to the features of Rust, especially on dataraces, etc. I've never seen a 100% clean, modern C++ codebase. If you know of one, could you point us to it?
Also, this is just me, given experience writing code in C++ and Rust, I find Rust to be a more ergonomic language (YMMV).
> And what do you do when clang isn't an option, due to customer, target hardware, OS or even language extensions?
Then you cannot use Rust and must settle for lack of safety. (A profoundly silly question -- if modern C++ is not an option for whatever reason, then Rust is doubly so.)
You made a bad argument, hide your ego in your pocket and admit it instead of writing such a nonsense. Both clang and Rust compiler are based on LLVM and they both can target only those platforms that LLVM target. If you can't use clang today then you can't use Rust also. We are not talking about "hypothetical" situations or alternative worlds because in those situations i can say whatever I want about clang also...
You didn't even reply to what I wrote? We are talking about Rust and Clang compilers and platforms they target and this was the context. It started when you tried to belittle Clang because it shows warnings and then you tried to make argument about hypothetical Rust compiler that can target OS that Clang can't, again trying to show Rust > C++. Even if you can't show superiority of Rust over C++ you are inventing hypothetical compilers that you think will work as your argument. If anyone will read your comments then he can see you are obviously biased and your ego just magnifies that effect. Think about it, objectively, get some air, there is world besides HN also.
A good bit of "legacy" C++ features are used in a number of applications that articles lists as "mission critical" perfectly well, too. Some of the arguments along memory safety, to my eyes, appear to be bordering on the hysterical. There are many legitimate criticisms of C++ ("legacy" or "modern"), including some related to memory management and safety. That hasn't impeded its use (its safe use) in a number of applications.
As a long-time user of C++, I find Rust to be more appealing in many ways. I don't think "C++ is dangerous" is a good approach to advertising Rust to the C++ community.
I care about safety and recognize that I can achieve it in C++. These claims that it's not possible, prohibitively difficult and so on are basically unsupported assertions. They're exactly the kind of argument that people who would like to see Rust gain traction shouldn't be making, because they impugn the credibility of the person making them.
The over emphasis on memory safety makes for a very weak pitch. I only spend a tiny fraction of my time on memory related issues. Most bugs are simply not memory related.
If you're writing Firefox or SSL, a memory mistake may be a vulnerability and a serious issue. In games, hpc, hft, etc, a memory mistake is almost always just another bug.
Would be great instead to see more examples of how e.g. destructive move in That allows generating better assembly.
In games a memory mistake is yet another way of loosing sales of a game, create bots to win network matches, work around in-game sales of items, ...
In HPC a memory mistake is yet another way of producing corrupt results for writing that paper, that after all is wrong.
In HFT a memory mistake is yet another way of producing corrupt values, leading to the automated trading bot selling items way lower than they should have been, loosing millions in the process.
It is yet another, exactly. There are many possible mistakes, memory is just one. One that consumes a very small fraction of my time.
I'm not a game developer so I can't judge your estimation of bug severity there but it seems highly speculative to say the least.
On the other hand your comments vis-a-vis HPC and HFT show that you have very little domain knowledge. Memory corruption is far more likely to crash a program, or to cause results that are wildly wrong. Wildly wrong is not so bad; any scientist worth his salt will question the results and see there are issues.
Your comments on HFT are even worse. Exchanges will not accept "selling items way lower", placing orders too high or too low produces rejects. And even if the order is accepted, it simply crosses the book at the best prices on the book. If you want to write about how you really lose millions, read about Knight Capital. Nothing to do with memory.
In both HPC and HFT, subtly wrong is a much bigger problem than wildly wrong. And memory is much less likely to produce subtly wrong unless someone is deliberately exploiting it. Again, it's not a matter of this never happening. It's just that it's rare in comparison to all of the other ways in which you can make mistakes, and particularly subtle mistakes.
This really seems to be one of the problems with Rust. Most people involved seem to be very firmly in the classical SV-esque tech world, working on browsers or core infrastructural components for the internet, or web servers, or what not. Chrome bugs and heartbleed are the examples du jour.
The problem is that this is not the main area of use for C++. The dominant industries for C++ are finance, video games, and HPC, along with some more demanding programs on mobile, or (rarely) on desktop. People like you who lack domain knowledge about the main areas in which C++ are used, stomping their foot and insisting that memory problems are central and that Rust is valuable, does not really achieve anything.
> Would be great instead to see more examples of how e.g. destructive move in That allows generating better assembly.
One example isn't about the destructive aspect: a move in Rust is always a memcpy. Since there's no move constructors, there's no way to run arbitrary code, so it's always a memcpy. The optimizer can also then elide this copy if it isn't actually needed.
Yes that's what I was referring to, the terminology I used has been used before in C++. Rust s moves are considered destructive in the sense that the old object is no longer functional, which of course is fine because it can't be used.
I think the difference is that in C++ the programmer has to be careful, whereas in Rust the compiler makes sure that only safe stuff happens (outside of unsafe blocks, that is).
Absolutely. But you still CAN make these mistakes, so people will. No one claims that it is literally impossible to write safe C++, just really damn hard and that you'll probably fail given a sufficiently large codebase. Rust makes it impossible for some set of errors.
Agreed. C++ is far safer than C and just as safe as rust. If the rust crowd was really serious about convincing C and C++ devs to move to rust, then they ought to have made the syntax identical (or at least much closer) to C than it currently is.
That's the primary reason Java took off. If you know C, you can write working C++ and Java programs straight away. For rust, we have to stop and relearn how to apply the logic (thus slow/no adoption).
And, I'm really sick of the constant HN sales pitches for rust. Make the transition easy and rust will stand on its own/sell itself (or not).
Being easy to use for people who don't want to learn new things is not a good driver for language design unless the primary motivation for the language's development is education. If you want to use C or C++, use them. Rust is its own thing and it will either succeed or fail to break into the mainstream but I can tell you that if it was just C or C++ with lifetime stuff bolted on then I would never have bothered learning it.
Criticizing it because C/C++ programmers can't just drop into it and start productively writing production-quality code is very short-sighted, IMO, and it's amazing how much it happens around here.
> I think the 2017's focus on productivity is very important.
If the yardstick of “productivity” is how fast you can write incorrect programs, I'm not terribly interested in “productivity”. I'm sick of being a defensive user, having to refrain from probing how systems work in undocumented and likely unforeseen scenarios, lest I completely break them in unrecoverable ways.
I'd rather see an axiomatic semantics for unsafe Rust, so that I can formally prove that my unsafe Rust code doesn't break safe Rust's integrity guarantees.
> So now that even them have got the AOT memo, and Go and Swift also exist
Support for running in an unmanaged environment is a red herring. If you want to manage resources correctly, you have to care about ownership and lifetimes, regardless of whether you are doing systems programming (user programs manage resources too), need AOT compilation (JIT-compiled programs manage resources too) or need a GC-less environment (GC doesn't work for resources that need to be reclaimed eager and deterministically).
> I imagine Rust's image should be better than a language from experts for experts.
I have no idea. I'm not a member of any community (not a “Rustacean”, not a “Haskeller”, not anything), and I have no personal interest (financial or emotional) in any specific technology becoming popular. I only care about technical facts. Are facts for experts only?
> GC doesn't work for resources that need to be reclaimed eager and deterministically
Which is why usually there are other language or runtime features for deterministicaly managing resources besides just the GC.
The question is how to keep the way of Rust memory management, while e.g. making it easier to write cyclic graphs, or reducing even further the cases where one has to explicity type lifetimes.
> Which is why usually there are other language or runtime features for deterministicaly managing resources besides just the GC.
Other than what Rust (and its research ancestors) uses, I haven't seen any alternative that isn't fundamentally broken in some way or another.
> making it easier to write cyclic graphs
I'd do it the same way I do it in any other language: initially create a DAG, then mutate it to create the cycles. This needs reference cells or something like that. Rust can do it already.
> reducing even further the cases where one has to explicity type lifetimes.
I don't see any way to do that in the vicinity of where Rust lies in the design space. What Rust has settled for (explicit lifetimes in type definitions, elision at use site) seems perfectly sensible to me. Of course, if a better alternative is found, that's a great thing.
I have no idea why you got downvoted. The only questionable thing in your comment is the use of 1 and 2, instead of 0 and 1. Downvoting for this reason seems petty. :-p
I'd really like to see some guides for using rust incrementally in a large C or C++ codebase.
* What are some good ways to deal with a lot of C macros and C/C++ header files in general.
* What is a good build strategy for integrating with a big, say, cmake build system?
* How to choose APIs, other than always using the lowest common denominator (C) for everything? Put another way, how do you choose which parts to write in rust first so that you don't end up with rust code that looks like it's translated from C?
* How to work with other memory allocation strategies, like a region allocator (like postgres memory contexts)?
(0) RefCell can't be used in a multithreaded context. You have to use an actual Mutex.
(1) ConcurrentModificationException isn't reliable. It's thrown on a best-effort basis.
In any case, in a high-level language, just memory-safety is a very cheap guarantee. You also want to protect the internal invariants of your data structures - especially the concurrent ones!
This looks interesting, though I can't find much info about inheritance and code reuse there, which should be a major topic I suppose? Who is the author of this book?
Porting an inheritance model to Rust is ... interesting. You can emulate it with some macros, but usually you should redesign the idioms used based on the exact use.
Yes, I'm not talking about porting things 1:1. I'm talking about explaining OOP approach that Rust suggests, and may be examples of how this methodology can be applied to solve issues like code reuse, that C++ at times solves with inheritance and etc.
Nothing. Unless you think rust will become the next c++, and the world will move to it en masse, in which case people still won't rewrite most of the old libraries to it.
At best people can hope that somebody comes up with a neat import system of c++ namespaces, a rock solid debugger and a fluid ide.
Rust will be like Go. Both pushed by corporations, and probably follow the fate of cobol, if history is to be asked.
Perhaps in adoption, but Rust has a huge feature over Go, which is that it can be integrated into existing codebases through FFI conventions, such that you can slowly incorporate Rust in safety critical areas (as an initial starting point).
If anything it was good for the overall good of our ecosystem. People coded it, used it, built with it. We all are in a better place knowledge and experience wise. New languages generally are good.
The book appears to be promising, but it is still far from being complete and doesn't touch the main points of interest, such as inheritance and problems arising from any form of template specialization.
Perhaps the author could get the book up to a presentable point before advertising it.
There was a rant earlier this week in the same vain: Instead of presenting something pre-mature check back later once its done.
I think it is still wothwhile to present something you have instead of waiting once its done. Also the understanding "once its done" varies from the use case. This in your opinion unfinished book might be of use to others.
It is also good to have something written and share it with others to get feedback (as you did) at an early stage.
I bet most people just read the first page of this guide and thought "yeah, I should upvote that, I'm gonna read it later" and don't realize that the article is full of gaping holes and TODOs which really devalues the entire work. I started reading it seriously and as the TODOs came in masses, I was disappointed and put it away.
You don't want to convert entire applications at once, you want to do it a source file or module at a time. This means incorporating the Rust toolchain into the build of the C/C++ project Makefile, or etc., system.
It is not difficult, and will increase confidence in the final product. This can even be used to test incremental conversions in production, rather than having any doubt about the entire conversion.
This is an awesome start for a great manual on the area. I'm very inspired by the experience I had in doing this, and hope that we do more.