Hacker News new | past | comments | ask | show | jobs | submit login
You can’t “turn off the borrow checker” in Rust (steveklabnik.com)
152 points by ngaut on Sept 15, 2018 | hide | past | favorite | 76 comments



I think there's mental mismatch between groups of people who talk about "turning off the borrow checker". The borrow checker is a tool to validate references. Sometimes people using references in Rust might feel like the borrow checker makes using references too cumbersome in a certain situation, so they switch to using a tool other than references (Rc, indexes into a Vec, etc). But this isn't bypassing the borrow checker; it's bypassing references themselves. The same phenomenon happens in C++; if references start to be a pain, you might switch to using something else (shared_ptr, indexes into a vector, etc.).

When this happens in C++, we don't call this "bypassing the borrow checker". You don't need a borrow checker to know that references aren't always the right tool for a given job. It's the same in Rust.


Yup. Part of why I wrote this post is for exactly this reason. This phrase is used colloquially, but I think it misleads a lot of people on how Rust actually works.


I thought this was going to be a response to Jonathan Blow's video about how doing your own memory management is effectively turning off the borrow checker: https://www.youtube.com/watch?v=4t1K66dMhWk

The takeaway being that the borrow checker doesn't magically prevent the use-after-free class of bugs. Although you will never experience a segmentation fault in safe Rust, the bug is still there and your program keeps running in an invalid state. The symptoms are changed, but no less dangerous.

To make the problem even more obvious, think of allocating a large array to be used as a heap and handing out indices to implement your own malloc. You have bounds checking to prevent indexing outside the bounds of the heap, but it doesn't really help when the elements have logically different lifetimes and occupy different parts of the array. I don't think this is a contrived example either. A less obvious version of this can easily creep into large or complex systems, as evidenced by the Entity Component System in Rust example.


So, I had been thinking about this post for a while, and Blow's video caused some more discussion that made me post it. But it's not a direct response, I still haven't watched the video, and so I don't know what he actually said. If I wanted it to be a response, I would have linked to it.

> The symptoms are changed, but no less dangerous.

I would take issue with this sentiment. There's a world of difference between "logic error and/or panic" and "undefined behavior".

Yes, Rust doesn't fix all bugs. But it's still an improvement here.


>There's a world of difference between "logic error and/or panic" and "undefined behavior".

I recently started using Rust daily as a break from $dayjob because I've never really liked C++. I took the time to watch Blow's full rant because I think we made an interesting point, that would take issue with your retort.

The naive version of West's "memory allocator" (without the generational index), in the context of her game, would have also had undefined behavior (in the game world). The naive system still defeats the borrow checker, but you can still end up in a situation where you try to deference something that no longer exists, and worse still since the object that lives there was the same type as before, your state corruption is even more silent. This necessitates the need for a generational index, however West only knew to use a generational index because she is an experienced game dev, not because the borrow checker told her to.

For Blow (who hasn't written rust), believes the borrow checker (and/or language) should prevent these kinds of logical bugs in his game code and bypassing it in this way effectively turns it off ("entity safety"), while the Rust borrow checker only really guarantees memory safety.

His final argument is then, since the borrow checker doesn't provide "entity safety", it impedes on games development because Blow (and any other modern game developer) would have been smart enough to start with a proper ECS system anyways and the borrow checker wouldn't have bought him anything. This final argument is where I disagree with Blow, but 1.) I don't think any programmer is smart enough 100% of the time however I will concede he comes from a different world (Game Dev) which has stricter deadlines than most other industries so he may be more sensitive to tools like the Borrow checker. 2.) Something I've noticed with Go as well is when the language developers tend to say something like "you can't use this toy because you will shoot yourself", it becomes a personal attack on developer ego rather than a nuanced trade off on system stability.


> would have also had undefined behavior (in the game world).

"Undefined behavior" is a term of art in programming languages. She would have a logic error, but not UB. See my comment over here: https://news.ycombinator.com/item?id=17995007


> His final argument is then, since the borrow checker doesn't provide "entity safety", it impedes on games development because Blow (and any other modern game developer) would have been smart enough to start with a proper ECS system anyways and the borrow checker wouldn't have bought him anything.

But even in C++ gamedev using a proper ECS, one is still using plenty of plain-old references all over the place for things unrelated to the world state, no? If so, then saying "the borrow checker doesn't help manage world state" doesn't imply that the borrow checker doesn't benefit the codebase in other places, especially considering that use of an ECS circumvents the places where we have determined that references (and hence the borrow checker) are already poorly suited to model.


Thanks for the response. I wasn't trying to speculate on your intentions for publishing the blog post.

> There's a world of difference between "logic error and/or panic" and "undefined behavior".

Is is really so different for the programmer who wrote the bug?

If you have undefined behavior, the language implementation can do whatever it wants. It won't actively work against you, but the implementer is given permission to ignore what would happen if you violate their assumptions.

With a logic error in custom memory management, the program execution will still be following well-defined rules but the invariants assumed by the programmer will no longer hold. The resulting behavior appears effectively undefined to the programmer, because the point of invariants is to ignore what would happen when they are broken.

Defensive coding with panics/asserts will definitely help catch some of these mistakes during development.

> Yes, Rust doesn't fix all bugs. But it's still an improvement here.

I applaud your efforts with Rust, it's great to see someone actually trying to improve the state of programming languages.


> Is is really so different for the programmer who wrote the bug?

Yes, absolutely. The symptoms may be similar (though the logic error will still never lead to memory corruption or segfaults), but debugging is much easier when you can still rely on the language's invariants, if not your own.


It’s all good, it’s a totally reasonable thing, which was also brought up in all the other threads :)

> it won’t actively work against you

I guess it depends on what you mean by “active.” Consider the Option<NonNull<T>> case. We can do the null check in safe Rust. We know the check is done. Now consider the case with UB: https://blogs.msdn.microsoft.com/oldnewthing/20140627-00/?p=...

These kinds of things can cause lots of subtle issues. The rust code won’t.


> The resulting behavior appears effectively undefined to the programmer, because the point of invariants is to ignore what would happen when they are broken.

I still think there are big differences here, especially when we think about these things as security issues.

If you write a program that's supposed to draw some pixels to the screen, and you have a logic bug, you program is going to draw the Wrong Pixels (https://xkcd.com/722). But your program isn't going to install a rootkit on your machine, or mine bitcoins, or send all your spreadsheets to the bad guys. If you never call the `mine_bitcoins()` function anywhere in your program, there's no way a logic bug can make you somehow call that function.

Not so with undefined behavior. An attacker who exploits a buffer overrun in your program can make you do anything. This almost sounds paranoid, but as soon as your code is taking input from the internet, this is really truly the problem you have. This sort of problem is why projects like Chrome spend millions of dollars building security sandboxes for their C++ code, and researchers still keep coming up with ways to break it.


If you segfault, then the error is clear. You're crashing, because (as the debugger or valgrind will show you) you tried to dereference this memory after you already freed it. You can then figure out why you freed it this early and change that. If you're in an undefined (according to your application's internal logic) state, it can be much harder to track down why it's acting erratically.


Use-after-free and other memory errors won't necessarily segfault anywhere near the source of the problem. It could also limp along, corrupting memory in weird places, until something totally unrelated segfaults instead.

ECS might let you continue with an outdated index, but that problem is contained.


Logic bugs with undefined system state are much easier to debug than UB, that is why people use memory-safe programming language.


Citation needed.

There is great tooling to pin down incorrect memory accesses when you are using the system allocator (valgrind, clang sanitizers). You're truly on your own if you access logically repurposed memory within a persistent system allocation.


> If you segfault, then the error is clear. You're crashing,

if you segfault. UB means anything can happen. Sometimes that's a segfault. Sometimes it means worse things.


To point; if you free memory, and reuse it (without nulling the reference), you likely won't segfault.

Simple example (compiled on OSX)

  #include <stdlib.h>
  #include <stdio.h>

  struct X {
      int x;
  };

  int main() {
      struct X *a = (struct X *) malloc(sizeof(struct X));
      a->x = 4;
      printf("%d\n", a->x);
      free(a);  // a is no longer a valid reference
      struct X *b = (struct X *) malloc(sizeof(struct X));
      printf("%d\n", b->x); // b is probably reusing the memory used by a
      a->x = 5; // updating a probably updates b
      printf("%d\n", b->x);
  }
If you compile this without optimization (clang test.c) you'll probably get

  4
  4
  5
'Probably' because this is both relies on both undefined behavior (which is partially why turning on -O2 changes the result), and the way malloc is implemented.

Fortunately, in a simple case like this, compiling your application with '-fsanitize=address' will give a very nice error in this case. :)


Just for fun, on Windows, I get

  > cl.exe foo.c
  > foo
  4
  10372440
  5
  > cl.exe -O3 foo.c
  > foo
  4
  9025424
  9025424
I've seen stuff like this work on OS X, and segfault on Linux. Yay UB!


No, but valgrind will tell you.


Not necessarily. It can tell you if it is triggered by your tests; but it won't tell you if it isn't. So if you run your test suite under valgrind, and you don't trigger the problem, valgrind won't tell you that there is a potential issue for certain inputs. Which, in this case, will result in silent corruption of the heap.

So, trivially adding argc to main,

    if (argc > 5) {
       free(a); // a is no longer a valid reference
    }
    // valgrind won't catch this issue
    b->x = 2; // valgrind complained about us dereferencing b before intializing
    a->x = 5; // valgrind won't complain about this if argc <= 5
results in a program that valgrind won't catch. Valgrind is great; but your users won't be running it when they use your program.

Now sure, you combine valgrind with other tools like afl (https://en.wikipedia.org/wiki/American_fuzzy_lop_(fuzzer)) or KLEE (https://klee.github.io/), and insist that your test suites have full coverage (however, code coverage isn't the same as input space coverage), but the point is, you're stuck doing runtime analysis (and need to know that you need to do that analysis) to make sure you did this right. Baking this type of error checking into the type system itself is valuable.

Given that large projects like Google's Chrome keep hitting these issues, it seems reasonable to say that they aren't strictly trivial to solve. :)

https://www.cvedetails.com/cve/CVE-2017-5036/

https://vuldb.com/?id.100280


> The symptoms are changed, but no less dangerous.

The symptoms of a use-after-free-style logic error are less dangerous in Rust, because it's much harder to get RCE.


What I'd like to understand better is what idioms are emerging as best practices thanks to the (good!) pressure that Rust puts on us.

For memory management, some of the major algorithms available are global allocation, stack allocation, malloc/free, reference counting, and tracing garbage collection. (The Entity Component System model is doing your own allocation so it's a subtype of either malloc/free or ref counting.)

Stack allocation and global allocation with borrow checking seem very attractive in terms of safety, but you have to know more in advance about how the memory is going to be used because growing the allocation while you are deeper in the stack is not possible. Are there any idioms or algorithms which are particularly friendly to this model? For instance, traversing a preliminary object graph to establish max allocation requirements before allocating a large block on the stack?


> What I'd like to understand better is what idioms are emerging as best practices thanks to the (good!) pressure that Rust puts on us.

We all would. They're still emerging! I think the generational index idea is one of them, though.


> The symptoms are changed, but no less dangerous.

Potentially even more dangerous. Having the program segfault immediately is much preferable to, say, Heartbleed.


Any situation where C would segfault immediately, Rust would also segfault immediately. The only difference is that Rust references are an alternative to C pointers that provide stronger safety guarantees at the cost of restricting what is possible (in other words, the same tradeoff that all static analyses make, including every type system). You can also use C-style pointers in Rust, but you can get away with just using references the vast majority of the time, which gives you better memory safety guarantees at no runtime cost since references are just pointers at runtime (and they're often automatically marked as restrict pointers at that). You're not going to get Heartbleed in Rust by using references.


> Any situation where C would segfault immediately, Rust would also segfault immediately.

For equivalent code, yes.

The borrow checker can however compel you to write code that would not immediately panic in Rust, when the C code you would have written has at least a chance of segfaulting immediately or being detected with valgrind.


The example that kicked off this discussion, replacing references or pointers with array indices, is presumably one example you're thinking of?

The problem with using that example to make this argument is that it is the equivalent code. Jonathan Blow's argument was not that the borrow checker forces people to use a worse pattern - he uses basically the same pattern in C - but that the borrow checker doesn't help you with that pattern, and so the borrow checker is extra friction.

Now, Catherine West's argument in the keynote video was that the borrow checker pushes people toward this better design, albeit without helping solve the use-after-"free" aspect, while Jonathan Blow says he would just start there as an experienced game developer, so he doesn't need to be pushed.

There's also the fact that the borrow checker isn't completely gone, and still applies to any references you put in the array or temporarily take to the array or that don't involve the array at all; as well as the fact that Rust also helps with things like parallelism, which definitely applies to games and doesn't need to be "bypassed."


I'm afraid I don't follow. Can you give me an example of the borrow checker compelling one to write such a piece of code? I admit I can't think of an example of Rust code that would fail to panic when the equivalent C code would tend to segfault.


The issue isn't the equivalent of Rust code in C, it's C code without an equivalent in Rust.

Rust should prevent the typical UAF bug in C, but if the programmer unwisely insists on using the same pattern in which they fail to track object lifetimes correctly, they can still implement it by using array indexes instead of pointers. Which can also be done in C (as was the case in Heartbleed), but it isn't as common, and is potentially worse than the usual C UAF because it produces a consistent successful out of bounds access rather than at least the possibility for a segfault.

It's possible to prevent someone from doing something bad and have that cause them to do something worse. "Nothing is foolproof to a sufficiently talented fool" etc. Safer tools are not a replacement for competence.


> The issue isn't the equivalent of Rust code in C, it's C code without an equivalent in Rust.

I'm afraid I still don't follow; Rust also has C-style pointers, they just require one to use the `unsafe` keyword to dereference. I can't think of any C code that can't be expressed in Rust in this way.

> Rust should prevent the typical UAF bug in C, but if the programmer unwisely insists on using the same pattern in which they fail to track object lifetimes correctly, they can still implement it by using array indexes instead of pointers.

I don't see how this shows that Rust is any more dangerous than C; a C programmer is no less capable of mistracking object lifetimes. Furthermore, array access are checked by default in Rust; even in the degenerate case of using the `get_unchecked` method (which requires the `unsafe` keyword as well), that's no more dangerous than every C array access.

> Safer tools are not a replacement for competence.

Why can't we as an industry have both safe tools and competent programmers? These are not mutually exclusive.


> I'm afraid I still don't follow; Rust also has C-style pointers, they just require one to use the `unsafe` keyword to dereference. I can't think of any C code that can't be expressed in Rust in this way.

What I mean is without an equivalent in Rust that doesn't require unsafe, which people are correctly admonished to avoid using whenever possible.

> I don't see how this shows that Rust is any more dangerous than C; a C programmer is no less capable of mistracking object lifetimes. Furthermore, array access are checked by default in Rust; even in the degenerate case of using the `get_unchecked` method (which requires the `unsafe` keyword as well), that's no more dangerous than every C array access.

The equivalent things are equivalent. The issue is that if you call one dangerous thing unsafe but not another thing that is as or more dangerous, the naive programmer assumes that not having to use unsafe means whatever they're doing is safe.

And it's little help to bounds check the array because that wasn't the problem. The data the programmer was expecting to be at that index isn't there anymore but the array itself still is and there is now some other data there.

> Why can't we as an industry have both safe tools and competent programmers? These are not mutually exclusive.

It's not that we can't have both, it's that we should have both.

Whenever we get better tools, people are tempted to hire worse programmers, because better programmers are more expensive and worse programmers with better software should be as good as better programmers, right?

This is obviously at odds with the ideal of using the new tools to create better software, and doesn't actually work, because a smoke detector can't save you if you don't know what to do when it warns of low battery.


If only we could guarantee immediate segfaults.


That's literally what Rust compiler errors are doing. :P


> If only we could guarantee immediate segfaults.

It should be possible to create a malloc implementation that does that by making the minimum allocation size a page and then not reusing virtual addresses for new allocations. Then once an allocation is freed, any access to it is permanently a segfault.

That may not be practical on existing architectures with 48-bit virtual addressing though, since you could plausibly exhaust the address space. The full 64 bits might be sufficient for most things at least.

You could also get most of the benefit by not reusing virtual addresses until you run out.


It's not, because that relies on actually doing the de-reference. Thanks to UB, that may never actually happen, the code may get removed entirely.


If the code is removed entirely then what memory is being improperly accessed?


It's impossible to tell, as that can cause other issues. See the link I posted about time travel elsewhere in the thread.


The day will come when the compilers that do things like that will be righteously categorized as malware.


The day will come when language specifications that allow compilers to do things like that will be righteously categorized as archaic.


I think the primary argument for Rust advocates here is that what you describe is only possible by using explicitly "unsafe" code (i.e. code using the "unsafe" block that is required).

The weakness of this argument is exposed in your comment also: in libraries "unsafe" is often hidden behind an abstraction. To take a simple example, consider the standard Rust Vec: plenty of "unsafe" code resides in this object, but you'll rarely if ever see "unsafe" wrap typical vector operations ("unsafe { myvec[0] }"). Partial mitigation is that this does reduce the surface area where one might have to look if odd behaviors start to appear in a Rust program.

Overall it's still a net benefit.


> ("unsafe { myvec[0] }")

Part of the point in my post is that this does not remove the bounds check. This unsafe block does nothing.

This actually might be a better example than the one I picked...


And just as in the OP, the compiler makes it clear that this unsafe block does nothing:

    warning: unnecessary `unsafe` block
     --> src/main.rs:4:3
      |
    4 |   unsafe { myvec[0]; }
      |   ^^^^^^ unnecessary `unsafe` block
      |


I might be in the minority here (and I don't do game development, which is the topic of this thread), but I tend to start by using `vec.get(i)` (which returns an option) and then only switch to the index notation if I end up checking the length immediately beforehand and want to do an early return or something to avoid extra indentation in the code where I use the accessed element.


I think modern C++ compilers are smart enough to eliminate the bounds check if they can prove that it never fails.


Yup, Rust will too.


I mainly do it this way to reduce the chance of my code panicking, not specifically for any performance reasons.


So it’s the difference between crash early (C) and don’t crash but run wrong (Rust), inherent by design in the main selling point of Rust (borrow checker)?


Why is everyone assuming C is "crash early" as opposed to "undefined behaviour will crash if you're lucky and cause RCE in the worst case, or any weird thing in between". The selling point of the borrow checker (at least one of them) is that it doesn't allow undefined behaviour unless you explicitly enable unsafe operations. If there was a way to detect UB and predictably crash in a performant way, you could probably implement that in Rust as well. In fact, that's often what is done with generational indexes and similar.


[flagged]


None of those are related to anything in the post you're replying to. What point are you trying to make?


Broadly speaking, C is the king of running wrong without crashing, so no.

The borrow checker isn't really making anything worse, it's just failing to save you from this particular problem. In C you could still write the exact same bug, or you could even write it with actual pointers. I gather a lot of browser exploits start that way these days.


No. Most modern C++ ECS take the same approach as the Rust ECS that is being discussed.

Also, since I started using generational indexes, I have never had a bug caused by using stale data, so at least for me it hasn't been a real issue.


I'd say that rust crashes early in a strict superset of when C crashes early.

The case where rust fails to crash early is you have are using indexes into an array as a pseudo pointer, but you've "freed" the object you wanted to use and put another (of the exact same type) in it's place. If you do the same with malloc/free in C you also aren't going to crash early. But in C you can also fail to crash early because of things like "and now an object of a different type is in it's place".


I believe the example here is where you bypass the borrow checker by keeping your own pool of memory and then allocating objects on that pool, using a custom heap implementation with manual memory management (or with a custom garbage collector). You can do that in either C or Rust (or any other language for that matter).


I don't see what there is about C that forces things to crash early. It's more like the difference between full of bugs that no one sees (C), and get the problem with your approach highlighted by the type system (Rust).


As a non-coder (physicist) writing Rust, the thing that really stuck me was that the time between successful compile and flawless operation was significantly shorter than the C and Python I write. Furthermore, this difference when writing anything threaded is simply breathtaking. Im my experience there are simply fewer corner cases that the Rust compiler lets through.


I think that makes you a coder :)


> This means that we can combine it with Option<T>, and the option will use the null case for None:

This sounds interesting, can anyone elaborate on this?


It's a really cool optimization, and it doesn't just apply to Option. Here's a StackOverflow thread: https://stackoverflow.com/questions/46557608/what-is-the-nul...


One of the design goals of Option is to be a library-level replacement for the language-level null value found in languages like C. Furthermore, one of the design goals for Rust itself is to have its abstractions be zero-overhead. With a naive implementation of Option, these goals would be in opposition.

To illustrate why these goals would be in opposition, look at a trivial example of a tagged union (enum) in Rust:

  enum Foo {
    Bar(i32),
    Qux(u32)
  }
At runtime, any value of type Foo will either be in the Bar state, or it will be in the Qux state. Both Bar and Qux hold types that are 32 bits in size, and since only one of those states can be active at a time, we know that Foo only needs 32 bits of storage to satisfy both these states. But additionally, it needs extra storage to store the runtime information telling us which state it's currently in. The smallest "extra" amount of storage that can be added to a type is 8 bits, so we would expect every value of type Foo to be 40 bits in size at runtime. In fact it's larger, due to alignment and padding, so Foo will be 64 bits in size at runtime.

Let's bring it back around to Option, which is an enum that looks like this:

  enum Option<T> {
    Some(T),
    None
  }
A null pointer in C will be the same size as a non-null pointer in C: 64 bits, assuming a 64-bit platform. A Rust reference would also be 64 bits, and these cannot be null. If we were to try to use Option on a reference to "opt-in" to nullability, what size would that "nullable reference" be assuming a naive implementation of Option? Well, firstly we'd need storage for the value of the reference itself (64 bits), and then, as per above, we'd need our "extra" storage to tell us at runtime whether our Option is a Some or a None. And again, because of alignment and padding, this would theoretically result in a type that is 128 bits in size in total, which is real shame since in theory distinguishing between two states only takes a single bit of storage. Overall this would be a performance regression from C, where nullability does not impose any space overhead.

Fortunately, Rust's implementation is not naive. Remember: Rust references cannot be null. That means that the Rust compiler knows that any type that is a reference will not contain a value that is all zeroes at runtime. Rust leverages this knowledge for optimization: for any Option containing a reference, only a single pointer-sized piece of memory is needed, and the None case will be represented by a value of all zeroes. This means that the Option is now a zero-overhead abstraction for this use case, because Option<&Foo> will be the same size as &Foo.

And this smart logic isn't hardcoded for the Option enum. Any enum, written by anyone, can automatically benefit from the ability to "hide" the enum tag in such "uninhabited" values. The OP's example of NonNull<T> is, like references, an example of of a type that has an uninhabited value that permits this optimization. Others include the NonZeroU8 type and its friends, where Option<NonZeroU8> will be the same size as a standard u8, though which give up the ability to represent zero (in the future this may be extended to allow arbitrary user-defined types which can make whatever values they want act as uninhabited for the purposes of enum size optimization, but it will take some work to get there).


Is it possible for Rust to store this in 64 bits?

  enum Boxed<T>
    Number(double),
    Some(T)
  }
ie, a type which can either be a double or a pointer by encoding the pointer as invalid (NaN) floating point numbers? Many JS engines use this trick.


An interesting question! Let's start this nerd-snipe by thinking about easier cases first.

1. Could Option<f64> be optimized to be 64 bits in size? This would imply that the implementation can guarantee that a particular one of the zillions of possible NaN values will never be generated as a result of a legal operation. I want to say that this is probably true in practice, though it will be implementation-dependent; I don't know how LLVM handles this, but I suspect that it only uses one or a handful of the possible NaN values to represent NaN in practice. So while it's probably feasible, it would make part of Rust's ABI dependent on implementation details of the backend (consider: is it likely that every possible backend reserves, and will always reserve, one NaN value that will never be used?).

2. Could the following enum be 64 bits in size?

  enum NanBox {
    Number(f64),
    Pointer(u32)
  }
A 64-bit float should have 2^53-1 possible values for NaN, which means that theoretically that should give us plenty of space to hide an entire 32-bit integer in the significand. This still presents the same worries as the prior point, except 4.2 billion times more severe. :) Furthermore, now you might have a new problem: consider that an Option<&Foo> that is Some is already a valid pointer, with no bit twiddling required. Additionally, above in point 1, a theoretical 64-bit Option<f64> is already a valid f64 in the Some case, with no bit-twiddling required. But here in our NanBox case, while Number is always a valid f64, the bit pattern of Pointer is not automatically a valid u32! At best, we'd have to mask out the irrelevant upper 32 bits, but now we're assuming something very specific about what bit patterns LLVM will never use for NaN. And in the worst case, we might have to do a nontrivial amount of extra work at runtime to make the Pointer look like a valid value. It might still work, for all I know, but it's another thing that requires careful thought.

In the end it boils down to what sort of guarantees LLVM wants to provide about its FP code generation (and maybe it's even architecture-dependent?), and whether Rust wants to risk irrevocably making its ABI nonportable. In contrast, there's no risk in assuming that references have an uninhabited value at 0x0, because Rust is in full control of that.



That would require Rust to convert all NaNs to a single "canonical" NaN. I don't think it does that, in fact you don't want Rust to do that, since it would make it impossible to do these NaN-boxing tricks by hand.

On the other hand, a CanonicalNaN<T> could be created in the future to allow these tricks, similar to the NonNull<T>/NonZeroU8/etc mentioned above.

(As an interesting aside, for the RISC-V ISA, all floating-point operations generate a single canonical NaN: "Except when otherwise stated, if the result of a floating-point operation is NaN, it is the canonical NaN. The canonical NaN has a positive sign and all significand bits clear except the MSB, a.k.a. the quiet bit. For single-precision floating-point, this corresponds to the pattern 0x7fc00000." So this scheme would work even better on RISC-V, since after any floating-point operation, it's guaranteed that any NaN is the canonical NaN.)


What I do instead (given the excellent sibling replies about how it's not that simple) is store the value in a `u64` and then provide a method to decode it into a non-layout-optimized enum. Since that enum is not stored anywhere but the stack, it can be optimized out if the decoding is inlined: https://github.com/rpjohnst/dejavu/blob/a768ce515b3a110542aa...


No, because NaN is a valid floating point number in rust.


There really should be a ”normal” f32 and f64 type with guarantees for non-NaN, similar to the nonzero integers. More importantly than size, these floats would be totally ordered unlike the partially ordered regular ieee floats.

Edit: turns out these exist in various forms e.g “noisy float”


"in IEEE-754 there are 2^53-2 different bit patterns that all represent NaN"

From https://developer.mozilla.org/en-US/docs/Mozilla/Projects/Sp..., which describes how Mozilla implements nan-boxing.


Thank you for this fantastic explanatipn! It is much appreciated


My TLDR/alternate title: "unsafe Rust retains most safety benefits of Rust (including the borrow checker)"


Note that you can drop down to unsafe C-style code in Rust. https://doc.rust-lang.org/stable/nomicon/

Anyone who claims Rust is simple should ensure they thoroughly understand that book.

Another way to "turn off" the borrow checker is to write a scripting language that compiles to Rust which automatically annotates all variables with the longest lifetime possible, and spits out mutable references depending on whether you actually mutate anything.

There's also RefCell, which lets you defer borrow checking till runtime. It's handy for pretending like your references are immutable.


I don't think that anyone's claiming that Rust is objectively a simple language. Simpler than some other languages, certainly, but it's a medium-sized language at best.

Furthermore, unsafe code gives new users a firm boundary of complexity that can be ignored. New to Rust? Don't use the unsafe keyword. Using the unsafe keyword? Read the nomicon first. It's quite useful for onboarding to know that all the C-style UB shenanigans are behind a gate that can be ignored until you're comfortable with the rest of the language.


Also note that none of those things turn off the borrow checker. Which is the point of the article.


not that steve isn't correct but Rc and Arc effectively (in exchange for runtime overhead) "turn off" the borrow checker. i'm sure i'll get yelled but it's just this week i had the borrow checker yelling at me for something and i realized that the appropriate thing to do was use Arc (yes of course i'm not advocating for ref counting instead of being more precise).


Rc and Arc are about (shared) ownership not borrowing. If they're turning off anything, it's ownership. Borrowing works the same way as ever.

In fact that's an advantage of Rust here: because of the borrow checker you can safely get a reference to an Rc's contents and hand it off to something without having to alter the refcount, so you get significantly less refcount traffic than in other languages where such a thing is unsafe (or not under your control). That's especially important for Arc.

It also provides for nice safe optimisations like "move out of this Rc if I'm the only owner" (Rc/Arc::try_unwrap).


I think you are confusing Rc/Arc with RefCell, since they are often used together. In order to provide interior mutability, the borrow_mut method of RefCell takes &self rather than &mut self. So, it is possible to obtain two mutable references. The following compiles:

    let c = RefCell::new(42usize);
    
    let mut mut_ref1 = c.borrow_mut();
    let mut mut_ref2 = c.borrow_mut();
    
    *mut_ref1 += 1;
    *mut_ref2 += 2;
RefCell uses UnsafeCell to subvert the borrows checker. However, instead RefCell implements run-time borrow checks. So, the code above will panic while attempting to do the second mutable borrow. It is a trade-off. You lose catching errors during compilation, but you can have interior mutability.

This is one of the goals of unsafe, which UnsafeCell requires. Sometimes, you need to do something that safe Rust does not permit, but that you can prove to be safe (such as RefCell).

That said (and it depends a bit on your domain), I have used Rust for probably 1.5 years without using Arc, Rc, and RefCell. The first time I needed these types when was writing a Gtk+ application and later in some multi-threading programs.

These three types are very easy to overuse when you are coming from other languages. IMO it is better to avoid them and try to work with the borrows checker until you have sufficient experience with Rust to identify when interior mutability and reference counting are really necessary.


One of my major points is that "avoiding the borrow checker" and "turning off the borrow checker" are two distinct ideas.

When dealing with unsafe code, it's very important to know exactly what's going on. Thinking that it turns off checks leads to you misunderstanding what's going on, and that's dangerous.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: