On: GLM-4-9B: open-source model with superior performa...

[norwalkbear]

Isnt 3-70b so good, reddit llamaers are saying people should buy hardware to run it?

Llama-3-8b was garbage for me but damn 70b is good enough

[reaperman → norwalkbear]

The unquantized llama 70B requires 142GB of VRAM. Some of the quantized versions are quite decent but they do tend to get overquantized below around 26.5GB of VRAM (~3 bits per weight).

So you’d at minimum be looking at dual 3090 with NVLink for about $4000 or so. Or for the highest performing non-quantized model, you’d be spending about $40,000 for two A100’s.

[norwalkbear → reaperman]

So a MacBook m series is a decent buy

[Manabu-eo → reaperman]

No need for NVLink just for inference, not even with tensor parallelism. And you can get used 3090 much cheaper than that.

[wkat4242 → Manabu-eo]

True, you can buy 2x 4090 FE brand new for $4000