Yup, precisely. In Matrix, rooms are not hosted by any single server - instead, they're replicated over all participating servers, much like git. But you need to check the provenance of the original messages to stop spoofing, hence needing to go check signatures the first time you interact with messages from a given server. Once you're joined, things should be fast however.
Basically it sounds like you can download the messages relatively quickly from any old server, but the first-time process of verifying takes awhile. Maybe there's a way to show that visually, so you can get the messages up on screen earlier, even if the content isn't trusted yet?
It also seems like there might be gains from prioritizing connections with servers whose members chatted more recently.
