On: Serving AI from the Basement – 192GB of VRAM Setup

[Tepix]

So, how do you connect the 8th card if you have 7 PCIe 4.0 x16 slots available?

[manav → Tepix]

PCIe bifurcation - so splitting one of the x16 slots into two x8 or similar.

[metadat → manav]

Worth mentioning - this also cuts the available bandwidth to each card by 50%.

[angoragoats → metadat]

While you're technically correct, assuming you're using PCIe 4.0 or higher, the performance difference between x8 and x16 is practically zero.

[metadat → angoragoats]

Is this because the graphics cards are not using all available PCIe bandwidth? Or why?

[angoragoats → metadat]

Yes. In fact, even running a video card in a 4x slot (again, assuming PCIe 4.0) results in only a modest (5-20%, depending on what you’re doing) drop in speeds.

[Tepix → angoragoats]

Even for training?

[angoragoats → Tepix]

I haven’t done a ton of training but everything I’ve heard and read indicates that PCIe 4.0 8x provides enough bandwidth for just about any application. You might see a negligible drop in performance, but no more than a few percent.