Historically this was more tractable because driver code was much simpler and there was a lot of cloning/emulation of "legacy" hardware interfaces (a lot of this is still present on PC, despite vendors' desire to get rid of it). Typically either the hardware capabilities were fundamentally narrower in scope than modern hardware or the hardware had its own controller (sometimes with more raw power than the host CPU...) that implemented an abstract interface. A bare-metal VGA driver for a single mode is no more than a few dozen lines of code, whereas a modern GPU driver stack literally has an optimizing compiler built into it.
