Its room based. Each room is synced between all servers used by participating members. But if you have a room with only people from 1 server, that metadata stays on that server.
Is it possible to configure this so that the metadata remains on the server by restricting access to only those users who are not members of any other servers?
It's not about being a member of a room on another server, but the homeserver of the account itself, of all the members of the room, must be local to that same server as the one that created the room in the first place.
I don't think there's a setting for "only allow people from X server", but you can make the room private and only invite people from that particular server.