"We have to learn the bitter lesson that building in how we think we think does not work in the long run. The bitter lesson is based on the historical observations that 1) AI researchers have often tried to build knowledge into their agents, 2) this always helps in the short term, and is personally satisfying to the researcher, but 3) in the long run it plateaus and even inhibits further progress, and 4) breakthrough progress eventually arrives by an opposing approach based on scaling computation by search and learning.
...
The second general point to be learned from the bitter lesson is that the actual contents of minds are tremendously, irredeemably complex; we should stop trying to find simple ways to think about the contents of minds, such as simple ways to think about space, objects, multiple agents, or symmetries. All these are part of the arbitrary, intrinsically-complex, outside world. They are not what should be built in, as their complexity is endless; instead we should build in only the meta-methods that can find and capture this arbitrary complexity. Essential to these methods is that they can find good approximations, but the search for them should be by our methods, not by us. We want AI agents that can discover like we can, not which contain what we have discovered."
I'm a little bit tired of the Bitter Lesson copypasta.
It doesn't seem to me like point 1 to 3 have been proven to be true about ConvNets. Sure, scaling convnets further help, I don't think anyone would argue that training a big model on more data would not at least see some improvement.
I also think there are irrefutable lower bounds of computation that it would be important to consider when building these meta-methods. If your problem requires N steps of computation for a problem of N size, but your method can only always perform at most K steps of computation, no amount of data or compute you throw at it will result in the right solution.
I don't think the parent comment is irreconcilable with the Bitter Lesson. I don't think transformers are a path to AGI, but that doesn't mean we need to go back to symbolic AI. We need to iterate on the meta-methods we use to search for and encapsulate the complexity of intelligence.
I think the bitter lesson has a lot more to do with the physical limits of von neumann computation and the structure of computing as we currently understand it than the direction that AI research should go.
That the emergent deep learning truthers are on the ascent should not be an excuse to completely discount knowledge based approaches. We're currently grappling with a number of limitations in the latest generation of ML models, particularly surrounding the cost of computation required to train and run them, and the fact that they have absolutely no way to verify the quality or correctness of their output. Clearly there are drawbacks to relying on a system we barely understand to produce our programs for us.
...
The second general point to be learned from the bitter lesson is that the actual contents of minds are tremendously, irredeemably complex; we should stop trying to find simple ways to think about the contents of minds, such as simple ways to think about space, objects, multiple agents, or symmetries. All these are part of the arbitrary, intrinsically-complex, outside world. They are not what should be built in, as their complexity is endless; instead we should build in only the meta-methods that can find and capture this arbitrary complexity. Essential to these methods is that they can find good approximations, but the search for them should be by our methods, not by us. We want AI agents that can discover like we can, not which contain what we have discovered."
http://www.incompleteideas.net/IncIdeas/BitterLesson.html