Yes, because your computer doesn't know anything about the destination besides its address. So it's not possible for your computer to verify anything else about the host.
Well yes, that's obviously true, which is why it's (idealistically speaking) important to verify the routing mechanism.
Of course most of this is mitigated by end-to-end crypto but given that we see all too frequently how fallible that can be, this topic remains of interest. I mean if crypto fails and leaks your private key (a la heartbleed) and it falls into the hands of an attacker who can hijack some BGP routes then that attacker is potentially in a very powerful position. We've seen BGP hijacking by spammers needing clean IPs in the past, so this isn't a totally implausible situation.
Edit: I guess you could use IPSec. It's not perfect, but it's pretty cool that we could have end-to-end crypto at the IP layer. https://en.wikipedia.org/wiki/IPsec#Security_architecture