You're correct in that, from the HW side, the "crown jewels" of Cray is the network ASIC's. Everything else is pretty much commodity.
AFAICS, the main difference between the latest Cray network (slinghshot) and IB are:
- Adaptive non-minimal routing, needed to get good performance on their more scalable network topology. IB has basically static ECMP, which works good for fat trees, which scales as O(log N). But for really large machines topologies like Dragonfly which Cray is using become interesting.
- QoS features to isolate jobs from each other.
- Some kind of compatibility with Ethernet. Not sure exactly how this works, but apparently when the NIC communicates with other compute nodes in the cluster it can use the high-performance mode, but if it communicates with some other system, such as Lustre servers, or management systems, or maybe even, say, pulling a container image from an external system, it looks and behaves like an ethernet NIC with the normal Linux TCP/IP stack running on top.
AFAICS, the main difference between the latest Cray network (slinghshot) and IB are:
- Adaptive non-minimal routing, needed to get good performance on their more scalable network topology. IB has basically static ECMP, which works good for fat trees, which scales as O(log N). But for really large machines topologies like Dragonfly which Cray is using become interesting.
- QoS features to isolate jobs from each other.
- Some kind of compatibility with Ethernet. Not sure exactly how this works, but apparently when the NIC communicates with other compute nodes in the cluster it can use the high-performance mode, but if it communicates with some other system, such as Lustre servers, or management systems, or maybe even, say, pulling a container image from an external system, it looks and behaves like an ethernet NIC with the normal Linux TCP/IP stack running on top.