The confusion seems to stem from two kinds of privacy goals here: Metadata privacy towards third parties (i.e. who is calling who; Signal explicitly does not provide this) and reciprocal location privacy of two calling parties (i.e. I don't know where I am called from and vice versa, only who I am talking to).
Signal's conscious choice is to interpret a user adding another as a contact as an implicit signal to mark them trustworthy enough to forfeit the second kind of privacy in exchange for better voice quality (latency and bandwidth) as well as to lighten the strain on their resources.
Signal's conscious choice is to interpret a user adding another as a contact as an implicit signal to mark them trustworthy enough to forfeit the second kind of privacy in exchange for better voice quality (latency and bandwidth) as well as to lighten the strain on their resources.