Contact permission is only required to associate a name with a number. All you would need to do is type in a phone number and Signal would have worked.
I agree that it would be nice to have another root artifact for using signal (screen name, email, etc). The one frustrating piece about Signal is not being able to use it across mobile devices.
However it needs to be considered Signal was originally securing texts via TextSecure. It didn't, fundamentally, come up as a Slack or Telegram competitor. So that design decision may make it feel "behind" in comparison. However, I'll again state that I don't trust the others. There's no way you could force me to use FB Messenger or Google (product here). Most complaints here are fundamentally still UI/UX, not privacy. And useable privacy was the original intent. Signal has that in spades.
We didn't have each others phone numbers in our contacts.
Manually adding by phone number didn't work.
At this point we just gave up.
I use prepaids because I travel a lot so my mobile numbers are just throwaways.
I'd rather not give _all_ of my contacts to an app anyway.
So there really should be a way to manage contacts inside the Signal client and add people via screen names or email addresses.