The problem I experienced multiple times in 50-200 KLOC projects is not the time needed to import the modules, but the memory consumption caused by the imports. Moving some imports from top-level module statements to inner functions' code could improve the memory consumption several times, e.g. from 250MB per process to 80MB per process.
One tool I used was https://github.com/mnmelo/lazy_import but I'm not sure it's updated for Python 3.7/3.8.