I don't actually think it's a bad choice... but that may say more about the state of what's available than about this particular chip.
The Cortex-M version of TrustZone is still fairly new, and these M33 devices are some of the first that implement it. You need a lot of care to use it correctly, but it has the potential to reduce the attack surface significantly. Crypto operations and key memory can live in the trusted world, while things like the USB stack can live in the non-trusted world.
If you really wanted a die shield, Maxim makes a line of "DeepCover" secure microcontrollers (Cortex-M3/M4, no TrustZone) that might fit the bill. They also have tamper pins for driving an external shield (e.g. https://www.edn.com/wp-content/uploads/media-1203638-p118fig...). You could do something like that and then fill the void with epoxy. External shields can be somewhat useful if your device stores keys in battery backed RAM (e.g. ATMs, POS terminals), because the shield remains active even when the device is off (if the shield is ever de-energized, the keys are wiped). USB security tokens typically store keys in (encrypted) flash, and don't have a battery, so you can take all the time you want grinding off the shield while the token is off, and then just short the right pins together before you power it up again.
ARM has a Cortex-M35P design that has TrustZone as well as some more advanced physical security features, but as far as I'm aware, nobody is selling one yet.
Downsides across the board:
Both ARM and silicon manufacturers are cagey about releasing any information about their security products. Most require NDAs before they'll even tell you what's in these chips, let alone how to use them, or how they work. I've worked with a few, and most of them have had some pretty scary bugs. They're worried that if they released the chip errata publicly, nobody would buy their chips. That's probably fair, but it also leaves you (understandably) less than confident that anyone has ever implemented a hardware crypto accelerator correctly.
Trusted execution contexts and memory protection seems like a good idea in theory, but I'm worried about all the complexity we're adding to these little chips. To lock the thing down, you have to pour over a thousand page datasheet and disable all of the debug interfaces you find, and enable all of the protection features that you find. If you're being thorough, you might write some little test programs to confirm that you at least can't use the easy methods to access things that should be protected. But at the end of the day you're crossing your fingers and hoping that there isn't a giant gaping hole off to the side that you forgot about. Reference software for these platforms is often crap.
¯\_(ツ)_/¯
I do have high hopes for this chip though... Apparently Oxide is also using it and has been sharing notes with SoloKeys (https://twitter.com/kc8apf/status/1360415931940302850). I think they'll get there eventually, but I think it'll be a while before we can be confident that it works correctly and that nothing obvious was missed.
