Signal's use of SGX was to increase users' trust in Signal - OpenWhisperSystems couldn't log your contacts even if they wanted to. Its up to potential users to decide if they think OpenWhisperSystems will surreptitiously perform an attack on their own secure enclave to secretly log your contact information.
