In my experience it still sometimes makes up schema or outright outputs mansplaining-like plaintext instead of JSON even if I give it correct schema. Happened once about 15~20 attempts, 5% is still too high to be considered reliable :(
I've tuned temperature, added a logit_bias to heavily prefer `{` token, this helped with the plain English v.s. JSON issue, but didn't help with hallucination. I guess I really need API access to {role: "tool"}.
I've tuned temperature, added a logit_bias to heavily prefer `{` token, this helped with the plain English v.s. JSON issue, but didn't help with hallucination. I guess I really need API access to {role: "tool"}.