DOS programs could run in protected mode by using a DOS extender. There were also quasi-standard interfaces that enabled programs so written to interoperate with environments that might use protected mode for their own operation and run DOS programs in VM86 mode. This support could also be extended to TSR's (background services) or device drivers.
Of course this was also problematic in that it could make some DOS software incompatible with earlier 8086-based hardware.
Of course this was also problematic in that it could make some DOS software incompatible with earlier 8086-based hardware.