Throw your nVidia card in the bin and be free. After many years with nVidia I removed the card and I've been happily using Intel hardware for a year now.
Because God forbid you actually have to use the card you paid for, oh no, we can't have that. No one should work with machine learning, edit video or play games, Intel HD should be enough for everybody /s
Lenovo X1E is a laptop... So removing the GPU will be a bit of a challenge ;)
I thought of turning the GPU off under Linux, but external displays (through the thunderbolt port) can only be used with the nVidia GPU. I need external displays for my work, so disabling the nVidia GPU is not an option for me.
Throw your Linux OS in the bin and be free. After many years with Linux I removed it and I've been happily using Apple/Windows hardware for a year now. /s