While it was still on RC builds I tested it on a few projects I usually run and I got between 0 and 15% faster wrt 3.10, without any regression. But sure, every project is different and it looks like you've bumped into some with your project.
If it's not too big, you might as well just leave things as they are and wait for 3.12 to see what changes then. Apparently the changes will be bigger, so it could be be much better than 3.11 (without being much worse, hopefully!).
https://ltworf.github.io/typedload/performance.html