The limiting factor for old consoles is largely RAM. The NES only had 2 KiB of RAM, making it pretty infeasible to even run anything CP/M-like. Its 6502 CPU also doesn’t have things like privilege levels/memory protection.
Moving up to the Sega Genesis with its 16-bit 68000 that many old UNIX systems were built around: it still only has 64 KiB. But, hey, that’s a lot better than 2 KiB, and it has memory protection. But the 68000 can’t do virtual memory (unassisted) and I don’t believe the Genesis has a timer interrupt, which seems like it’d complicate multitasking (I guess you could run the scheduler when the graphics system raises the vertical retrace interrupt?) but IMO it’s about the minimum level that could run something like what we tend to think of today when we talk about an operating system.
It's true the NES has 2 KiB built-in RAM, but a cartridge can provide its own RAM. Some of the later cartridges (MMC5 for example) supported up to 128 KiB of extended RAM. There's not really any limit to how far you could push that with a custom mapper.
Moving up to the Sega Genesis with its 16-bit 68000 that many old UNIX systems were built around: it still only has 64 KiB. But, hey, that’s a lot better than 2 KiB, and it has memory protection. But the 68000 can’t do virtual memory (unassisted) and I don’t believe the Genesis has a timer interrupt, which seems like it’d complicate multitasking (I guess you could run the scheduler when the graphics system raises the vertical retrace interrupt?) but IMO it’s about the minimum level that could run something like what we tend to think of today when we talk about an operating system.