It's one of the most amazing and surprising things I've seen in the last 12 months in machine learning (I follow and work in the field).
It surprises me a lot more than the excellent performance of GPT-3 on text generation for example. GPT-3 is amazing but looking at GPT-1 -> GPT-2 -> GPT-3 it isn't surprising. Counting on the other hand is something I wouldn't have expected from a summarizer.
But to me that isn't as surprising. Not claiming I would have thought of it, but if you have a very large multi-dimensional space (such as GPT-3) then giving it some examples of something pushes it into that general area of the space.
Generalizing concepts isn't a new thing - one could argue that word2vec from 2014 did that pretty well. GPT-3's "concepts" are vastly more complex than the single word (or maybe 2 word) concepts in Word2Vec though.
I mean in that sense, GPT probably just extracts low-n counts as separate concepts.
I'd love to see an architecture that can keep a separate short-term memory to allow it to count with multiple digits and follow algorithms. On the other hand, given what we've seen from GPT, at that point I would actually worry about it becoming a general intelligence...
I agree it probably doesn't "understand" math, but it has learned that number words can substitute for each other in a sentence (three ships/four ships/five ships) which isn't surprising.
But it has somehow learned to link that word with the correct length of the sequence of names, which is astonishing. I can't think of obvious "cheats" that make this work.
The best I can think of is that is has learned to count commas when they are separated by words.
It surprises me a lot more than the excellent performance of GPT-3 on text generation for example. GPT-3 is amazing but looking at GPT-1 -> GPT-2 -> GPT-3 it isn't surprising. Counting on the other hand is something I wouldn't have expected from a summarizer.