Use poetry/Debian. Standard Python tooling is acceptable in 2024; using some busted 3-plus-year-old "supported" environment with a package manager with a really busted constraint solver, which doesn't even come bundled with compilers, is unnecessary.
I've installed plenty of compilers using conda - how long ago was it that you last tried it? even ROS is available in conda now, instead of requiring specific obsolete versions of Ubuntu.