On: Llama 3.2: Revolutionizing edge AI and vision with...

[regularfry]

Qwen2.5 has a 32B release, and quantised at q5_k_m it *just about" completely fills a 4090.

It's a good model, too.

[kristianp → regularfry]

Do you also need space for context on the card to get decent speed though?

[regularfry → kristianp]

Depends how much you need. Dropping to q4_k_m gives you 3GB back if that makes the difference.