It's impossible to please everyone with rotary controls. If I read you correctly, you're asking for them to be controlled by actual rotary movement of the mouse. Many people dislike that (and I'm one of them) because it's hard to make circular motions smoothly. Meanwhile, the rest of the people dislike linearly-mapped rotary controls because the motion doesn't match the visual form of the control.
I would have suggested making them respond linearly, but to both x and y motion. That is, the effective position change is the maximum of the mouse x and y axis displacements from the click position.
My view is that rotary motion is not useful in an on-screen GUI (unless you have a MS Surface Dial perhaps?); rotary-looking controls only exist to save space and provide a friendly appearance. The way they actually behave should be whatever is most efficient to drive, regardless of their shape.
You've misread me--I'm suggesting to use the angle formed between vertical and the mouse's current position, and then wherever it moves, to add to the current position of the rotary knob.
This doesn't mean that you end up with having to carefully dial the knob, but it does let you click and drag the mouse in such a way that the rotary dial at least rotates in the direction you'd expect. The current behavior sometimes turns CW, sometimes CCW, seemingly at random--if we decide to skeuomorph, we might as well skeumorph correctly.
You could achieve both your core requirements from the final two paragraphs by having, effectively, a slider control underneath the rotary dial. The slider is the actual control — behaving as you describe in para 2 — and the rotary dial is merely for pleasing visual feedback. The GUI would have to emphasise the roles of those two separate parts somehow.
I would have suggested making them respond linearly, but to both x and y motion. That is, the effective position change is the maximum of the mouse x and y axis displacements from the click position.
My view is that rotary motion is not useful in an on-screen GUI (unless you have a MS Surface Dial perhaps?); rotary-looking controls only exist to save space and provide a friendly appearance. The way they actually behave should be whatever is most efficient to drive, regardless of their shape.