I would say the difference between 2 seconds and 2 minutes is significantly more noticeable than the step from 20 minutes to 2 minutes.
Apart from the fact that the factor is bigger (60 vs 10) it matters that if you go under 10s your workflow changes, as you don't have to work in "async"-mode anymore "oh, while this compiles I check this stuff in the documentation". For me who is not the best multi-tasker this is a real benefit.
I think it depends highly on what kind of developer you are and what your workflow is like. For some developers, frequent compiling is part of the development process. For others, it's something you do at the end (or after large chunks of work) to test/verify.
If you're one of those "Work on the next line, keep changing it until it compiles, then keep changing it until it runs" developers, then yea you'll want fast compiles.
If your workflow is such that you take your time, do your work once, and compile/run/test as the last step before you're ready to commit, then it doesn't really matter if your project takes 20 minutes to compile. You're only doing it once or twice a day.
When working on a new project I sometimes spend weeks coding without attempting to compile once (it usually takes a couple of day then to sake down all typos and thinkos); my boss is horrified by this but it works for me. In this scenario I care little about compiletime.
The problem is when I'm fixing bugs or adding small features to an existing codebase, especially when tests need to be added or modified. The compiletime turnarounds does kill my productivity in these cases.
Apart from the fact that the factor is bigger (60 vs 10) it matters that if you go under 10s your workflow changes, as you don't have to work in "async"-mode anymore "oh, while this compiles I check this stuff in the documentation". For me who is not the best multi-tasker this is a real benefit.