I used to use text consoles for work and X only for the browser for many years. I switched only to terminals in X when I hacked up a rxvt enough to be very close to the text mode console (including the font and pretty closely matching GPM for cut-paste). It is generally responsive "enough", though for years I keep blabbing on about figuring out some way to check the latency.
rxvt versions newer than 2.7.1 and, more recently, rxvt-unicode seem to have some other issues that make them really slow (particularly non-bitmap font rendering), but rxvt-unicode supports mixing bitmap and other (Terminus) fonts which seems to be a reasonable solution.
rxvt versions newer than 2.7.1 and, more recently, rxvt-unicode seem to have some other issues that make them really slow (particularly non-bitmap font rendering), but rxvt-unicode supports mixing bitmap and other (Terminus) fonts which seems to be a reasonable solution.