I would really like to use just the linux virtual terminal. Unfortunately it has no unicode support and there is an unsolved issue with amd GPUs that causes the switching from a vt to X.org to take an unbearable amount of time.
Is it a GPU issue? I have an NVidia GPU and I still wouldn't regularly hit Ctrl-Alt-F1/F7 to switch tasks.
Estimating, it takes at least 500 - 1500 ms. I have no idea what's happening here... if it's an X thing, a driver thing, etc.
I don't recall it being fast on any machine I've used recently. It's at least 100x slower than it should be to be usable -- it should be around 5 to 15 ms, or even less.
I have to look into Kmscon which seems promising.