D3D is not IPC. It is mostly implemented in the process' address space. When you call a D3D function you are not magically dumping all your primitives into the kernel-side driver (which would be slow), you are calling into nvd3dum.dll which _may_ eventually call into the kernel. All of this happens in the same process and therefore the GPU driver can do whatever the hell it wants to it.
Yes but this derouted the engine.dll (or equivalent) code in userspace (not sure if that's the right term here). I agree that they probably whitelist or even just don't check the graphics or rendering related functions, since tons of mods allow you to do that without any widespread ban wave.
But when it's hooking to the core engine code, I think that's a different story. Correct me if I'm wrong though.
There is no way to make this distinction purely from userspace, you cannot "sandbox" a library you are loading into your process. Therefore they just keep a whitelist of graphic drivers that are allowed to load (and do whatever they want).
Anything else would just not work. You can't prevent the drivers from doing whatever they want or you would quickly hit a myriad crashes. You cannot rely on what the OS calls a "graphics driver" because then someone would implement their own drivers (wrapping NVs). Your only resort is to allow the well-known drivers and prevent everything else.
The same reasoning applies to all operating systems (e.g. OpenGL ICDs). It is not windows exclusive.
And btw, mods which change rendering related functions are some of the most classic cheating devices and therefore are of critical importance to block in the first place.
I really doubt that; not because they haven't done that in the past, because they have; but because with the frequency of updates of game clients these days you really cannot get away with that as easily. On the other hand, hooking the lower-level Win32 functions is extremely common, I am yet to see a GPU driver that doesn't implement some level of idle/activity detection based on hooking the message loop, for example.
My bet is they have just decided to put whatever this new thing is in a separate module which they they forgot to send to Valve for whitelisting.
There's a world of difference between hooking a message loop and hooking a Win32 function:
You can hook a message loop using regular Win32 APIs, while hooking a Win32 API requires you to modify code in the process. Specifically you need to inject a trampoline function into the process memory and then overwrite the first two bytes of the Win32 function you'd like to hook with a jump instruction to your trampoline.
The former is a lot easier to allow while the latter is much more likely to open the flood gates for cheats.
