On: ChatGPT-4 significantly increased performance of b...

[matheusmoreira]

Isn't ChatGPT getting progressively better scores on medical and law exams? It will probably pass the USMLE and the bar one day. If it doesn't already.

It's gonna be interesting.

[coffeefirst → matheusmoreira]

Yes, but we should expect that, the answers are in its training data.

The problem is passing tests are an okay proxy for competence in humans, but if you think of LLMs as a giant library search engine, the thing it is competent at is identifying and regurgitating compiled phrases from its records.

Which is awesome. It can't be a doctor.

[benreesman → coffeefirst]

Relatedly, a lot of the cap R fanfic crowd points at stuff pretty clearly lifted out of the Metamorphosis of Prime Intellect or whatever.

And it’s dope that it can do that!

But let’s keep our heads about what it is.

[Jeff_Brown → matheusmoreira]

Yes and that's amazing -- but law exams resemble programming exams. In the wild, both labors require you to keep a mountain of project-specific context in your head, something that tests like the LSAT cannot evaluate.