On: Why host your own LLM?

[Bostonian]

ChatGPT is powerful, but it gives you different answers to the same question from one session to the next. And research found that overall performance can vary over time, sometimes for the worse. So you may host your own LLM for reproducibility.

I have not tried public LLMs myself. Do they give reproducible results?

[TacticalCoder → Bostonian]

> I have not tried public LLMs myself. Do they give reproducible results?

Public LLMs I don't know but images generated using StableDiffusion are, of course, totally deterministic.

There really is no reason a LLM cannot be deterministic and if it isn't: fix it (even if this comes at a tiny performance cost).

[mcbuilder → TacticalCoder]

If you fix the random number seed virtually all LLMs should be deterministic. However, just 1 token difference in the input could produce a very different output, depending on the sampler, model, etc. So, LLMs can be deterministic, but in practice they are pure alchemy.

[varispeed → Bostonian]

> it gives you different answers to the same question

Sometimes answer is wrong and then right.

If it is deterministic, then what if gets "stuck" on wrong answer?

[embwbam → Bostonian]

They do that on purpose. The API gives you a setting you can change to 0 or 1 for maximum creativity or maximum reproducibility

[sp332 → embwbam]

That doesn't work for GPT-4 though. https://news.ycombinator.com/item?id=37006224

[Makhini → embwbam]

Except it's never reproducible. It's a bug probably.