An interesting question! Let's start this nerd-snipe by thinking about easier cases first.
1. Could Option<f64> be optimized to be 64 bits in size? This would imply that the implementation can guarantee that a particular one of the zillions of possible NaN values will never be generated as a result of a legal operation. I want to say that this is probably true in practice, though it will be implementation-dependent; I don't know how LLVM handles this, but I suspect that it only uses one or a handful of the possible NaN values to represent NaN in practice. So while it's probably feasible, it would make part of Rust's ABI dependent on implementation details of the backend (consider: is it likely that every possible backend reserves, and will always reserve, one NaN value that will never be used?).
2. Could the following enum be 64 bits in size?
enum NanBox {
Number(f64),
Pointer(u32)
}
A 64-bit float should have 2^53-1 possible values for NaN, which means that theoretically that should give us plenty of space to hide an entire 32-bit integer in the significand. This still presents the same worries as the prior point, except 4.2 billion times more severe. :) Furthermore, now you might have a new problem: consider that an Option<&Foo> that is Some is already a valid pointer, with no bit twiddling required. Additionally, above in point 1, a theoretical 64-bit Option<f64> is already a valid f64 in the Some case, with no bit-twiddling required. But here in our NanBox case, while Number is always a valid f64, the bit pattern of Pointer is not automatically a valid u32! At best, we'd have to mask out the irrelevant upper 32 bits, but now we're assuming something very specific about what bit patterns LLVM will never use for NaN. And in the worst case, we might have to do a nontrivial amount of extra work at runtime to make the Pointer look like a valid value. It might still work, for all I know, but it's another thing that requires careful thought.
In the end it boils down to what sort of guarantees LLVM wants to provide about its FP code generation (and maybe it's even architecture-dependent?), and whether Rust wants to risk irrevocably making its ABI nonportable. In contrast, there's no risk in assuming that references have an uninhabited value at 0x0, because Rust is in full control of that.
1. Could Option<f64> be optimized to be 64 bits in size? This would imply that the implementation can guarantee that a particular one of the zillions of possible NaN values will never be generated as a result of a legal operation. I want to say that this is probably true in practice, though it will be implementation-dependent; I don't know how LLVM handles this, but I suspect that it only uses one or a handful of the possible NaN values to represent NaN in practice. So while it's probably feasible, it would make part of Rust's ABI dependent on implementation details of the backend (consider: is it likely that every possible backend reserves, and will always reserve, one NaN value that will never be used?).
2. Could the following enum be 64 bits in size?
A 64-bit float should have 2^53-1 possible values for NaN, which means that theoretically that should give us plenty of space to hide an entire 32-bit integer in the significand. This still presents the same worries as the prior point, except 4.2 billion times more severe. :) Furthermore, now you might have a new problem: consider that an Option<&Foo> that is Some is already a valid pointer, with no bit twiddling required. Additionally, above in point 1, a theoretical 64-bit Option<f64> is already a valid f64 in the Some case, with no bit-twiddling required. But here in our NanBox case, while Number is always a valid f64, the bit pattern of Pointer is not automatically a valid u32! At best, we'd have to mask out the irrelevant upper 32 bits, but now we're assuming something very specific about what bit patterns LLVM will never use for NaN. And in the worst case, we might have to do a nontrivial amount of extra work at runtime to make the Pointer look like a valid value. It might still work, for all I know, but it's another thing that requires careful thought.In the end it boils down to what sort of guarantees LLVM wants to provide about its FP code generation (and maybe it's even architecture-dependent?), and whether Rust wants to risk irrevocably making its ABI nonportable. In contrast, there's no risk in assuming that references have an uninhabited value at 0x0, because Rust is in full control of that.