At that point you've frozen a working image of your dependencies along with a bunch of other stuff, but you still don't have a clear picture of what those dependencies are and how to work with them.
Your build works in a vm of RHEL 5 with packages x and y installed, ok, now you need to add a new build step that depends on something that's only available for RHEL 7. So you run the VM generate script (if you can even find it a year or two later) and the build doesn't work. Now you get the same fun debugging experience. Standardising which OS and package installs every developer uses (whether with a VM or just with something like Puppet) means you get to roll the dice less often, but it's still a dice roll whenever you want to change something.
And running all your builds in VMs has a cost in terms of performance but more importantly in terms of comprehensibility. If my build is just Maven I can run it in my IDE, then I can push one button differently and run it in my IDE with a debugger attached. If the build is running slowly and I'm not sure why, I can push one other button and now I'm running it with a profiler. Some of that is just an argument for having your build tooling written in the same language you're using for development (if I was working in e.g. Python, I'd be more likely to have a Python IDE and a Python profiler to hand, so would want to use a Python build tool), but even if you're used to working in Bash and C I'm not sure the VM advocates have good answers to how you debug something running in a VM as yet.
why not on copies of known-good machines? vm boxes with the same definitions would alleviate almost all of that 'roll the dice' aspect, no?