On: Mistral releases ‘unmoderated’ chatbot via torrent

[thelastparadise]

Could it run on a 4x 3090 24GB rig?

These can be built for about $4500 or less all-in.

Inference FLOPs will be roughly equivalent to ~1.8X A100 perf.

[mutex_man → thelastparadise]

You could run it on a single high end GPU. I can run llama2's models ,(except 70b) on my 4080.

[7moritz7 → thelastparadise]

This can run on 1x 2060S 8 GB

[ComputerGuru → 7moritz7]

With what degree of quantization?

[7moritz7 → ComputerGuru]

None, just the default weights using ollama. It's fast too. 13b is where things get slow

[chriscappuccio → thelastparadise]

does a 4x 3090 rig need nvswitch