The deepemoji one is fooled by "was my flight delayed? no.". I feel for the computer when it meets that one "do I speak in questions?" person. chuckles.
On a more serious note, Hinton and other alluded to the need to restructure NLP studies as to focus more on the nature of recursion within language, which is basically what Chomsky has been saying for decades. It's interesting to see whether they converge.
I'm wondering if these tasks have a form of bias that decreases the performance. If the model sees only positive examples and no negatives then it is biased on the positive paths of decisions. The moment where one changes the path to be incorrect, the model can't recover from the mistake because there weren't any negative examples during pretraining. There's many words that never follow some words but the model never sees that.
On a more serious note, Hinton and other alluded to the need to restructure NLP studies as to focus more on the nature of recursion within language, which is basically what Chomsky has been saying for decades. It's interesting to see whether they converge.