Yes, they are necessarily global pointers. If a self reference is an offset, what happens when you take its address (and pass that somewhere else that doesn't know where it came from)?
Now every "reference to a reference" (and similar things) can dynamically be a global pointer or a relative one, with no way to tell how to use it.
Yes, it's safe to cast a reference to a raw pointer and a raw pointer to an address.
It might have been nicer in some ways if the language had restricted that a bit more up front, but it would probably have taken some design work to do that while still supporting a lot of the low-level stuff people need to do.
An internal reference could only be cast to a relative memory offset. It’s not a global memory address in disguise like a normal reference. So the same casting rules wouldn't apply.
Are all references necessarily (global) pointers? Couldn't self references be offsets?