As I understand it, the purpose of using telephone numbers is that it can use the contact list stored on your phone without having to store contact lists on their servers.
Contact lists can be useful information for an attacker. If Signal doesn't have it, it can't be taken from them.
There's nothing inherent in phone numbers here. Both iOS and Android also allows you to add e-mail addresses (and other identifiers) to your local contacts. I'm yet to hear an argument as to why e-mail addresses or other identifiers can't be used in addition to phone numbers, or why it would be a complicating factor.
Contact lists can be useful information for an attacker. If Signal doesn't have it, it can't be taken from them.