You must be young! I did run graphical Linux 1997 as my main driver. The distribution was a black hat something, you used a rewrite tool to create a boot floppy which would boot the CD ROM later.
It came with at least three DEs, olvwm and xfce and whatnot.
Driver were never of an issue for me maybe just luck.
I've been using Linux almost exclusively since 1994.
Lack of drivers was definitely a problem but it made me spend a lot of time researching whatever I bought first. Lots of reading Linux HOWTOs and Usenet forums about whether a particular card was supported or not.
I had to install from 30 floppy disks because my brand new IDE CD-ROM wasn't supported for another 6 months. Same with my new Diamond Stealth 64 graphics card. I had to wait 6 months to be able to run X11
Those both came with my new Pentium 90 PC. After that I bought hardware specifically for Linux like my BusLogic 948 SCSI card