Really, the only constraint for the semantic of unsafe blocks is that programs that does not contain unsafe blocks cannot have undefined behavior, everything else is by choice.
For example, in theory you can make pointer dereferencing safe (!), and make every operation that might create a invalid pointer unsafe. Rust chose to do this the other way around, probably out of usability reasons.
Even if a pointer was guaranteed correct at the time of creation, it can't be known safe to dereference in the future unless you put a lifetime on it, and then it's just a reference.
For example, in theory you can make pointer dereferencing safe (!), and make every operation that might create a invalid pointer unsafe. Rust chose to do this the other way around, probably out of usability reasons.