No. You can evaluate compiler speed, memory use, quality of error output, generated code quality, ie most of what anyone cares about without knowing what's in it.
On what code? Any code you pick is an arbitrary benchmark. There are infinite number of context free grammars you could feed the compiler and for any of them in might make some unknown optimization X that is superior to all other compilers. The number of programs that you can write are infinite therefore you cannot say for certain that is better or worse for all programs, or in other words, objectively say it is terrible.
The idea that you need to test with infinite inputs to be "objective" is preposterous. You can in fact test with one input and the result is still objective, and you can conclude that it is objectively terrible or not on that one input. In reality, a suite of real world tests within your application domain provides actual useful information -- those aren't arbitrary benchmarks if that's the actual code you care about.
No. All optimizations have tradeoffs, if you don't know what it's doing then you cannot know the tradeoffs made. Empirical black-box testing works to a degree, past which it only gives false assurance, as it would in this case.
But you can know what's doing. Just have it output the .s/.asm file
You can also look at the flags and the different optimizations you can turn on and off.
MSVC had some issues with C++ compatibility (a long time ago) but overall it is not bad. A compiler that can compile a whole operating system probably has most of its issues ironed out.
> But you can know what's doing. Just have it output the .s/.asm file
That's still black box testing. There ~may be~ almost certainly is hidden state that could mean that the optimization you expect to be applied is not (or an optimization is misapplied) under certain circumstances, and you can't/won't discover this in advance due to those conditions.
> A compiler that can compile a whole operating system probably has most of its issues ironed out.
History has shown this to be mostly wrong, given how many critical bugs have been discovered in GCC since it was capable of compiling linux
