That really depends on your threat model. Every effort to reduce your attack surface increases the effort needed for your adveraries. Your "won't help very much" is only true for the trusting trust problems in the paper, i.e. if you need to survive state sponsored attacks
Trusting Trust actually says that vetted source isn't enough! Fortunately it's been defeated (https://dwheeler.com/trusting-trust/), such that vetted source and multiple compiler executables that are unlikely to be compromised in the same way can be enough if you use them carefully (which, to my knowledge, we don't).
http://users.ece.cmu.edu/~ganger/712.fall02/papers/p761-thom...