It's unclear to me whether you understood when you are being given potentially untrustworthy information by the server.
Some of the other designs you described try to have users verify the ephemeral end-to-end encryption keys. If Signal did that then obviously each call, text conversation, or whatever has new keys, trust doesn't continue from one to another. But Signal's long-term identity key relates everything together. The Safety Number is about ensuring you really have Bob's long-term identity key (and Bob has yours) rather than about this particular call, conversation, etc.
Some of the other designs you described try to have users verify the ephemeral end-to-end encryption keys. If Signal did that then obviously each call, text conversation, or whatever has new keys, trust doesn't continue from one to another. But Signal's long-term identity key relates everything together. The Safety Number is about ensuring you really have Bob's long-term identity key (and Bob has yours) rather than about this particular call, conversation, etc.