Yes, if you install a passive hole in nearly every computer made, well, having those computers phone home is silly. You call the target machine, when you want something.
Packet wouldn't have to crash the machine, just start up a more active trojan.
Hardware level trojan is relatively hard to develop.
I mean, it depends on what do you mean as a trojan. I think it is entirely possible to create a hardware trojan that sends some packets somewhere. It'd make a neat DDoS. :-)
Less likely is a classic trojan that gains control over your machine. Do not forget there are many layers between the userland and the hardware (eg, drivers, the network stack etc.).
It is a completely different story if evil Bob created both the hardware and the driver...
It would be used only once. Like the nuclear weapon.