On: Why host your own LLM?

[Aerroon]

I don't know if he actually said this, but power limits on gpus do work. I've been limiting my gpu to about 60% max power during long AI workloads and I lost about 10% of performance. A 40% reduction in power usage seems worthwhile.

It probably depends on your gpu though.

Undervolting would probably be even better.

[robotnikman → Aerroon]

same here with the 3090 I use. Power limited to 60% and I barely notice a difference when experimenting around with AI or playing games. Really good to use in a SFF PC

[bavell → robotnikman]

What OS/software are you using to throttle the cards? I'm interested in doing this on Arch

[zten → bavell]

    ~ > sudo nvidia-smi -pl 300
    Power limit for GPU 00000000:09:00.0 was set to 300.00 W from 450.00 W.

[Aerroon → bavell]

You got the Arch solution. For anyone else on windows: you can just use MSI Afterburner (regardless what you have an MSI card or not).