There are probably ly a hundred. Many people have asked for research into this for I don't know 20 years. But the hype doesn't die down and modern trends have mostly involved doing more GD with bigger models...
A colleague and myself experimented with some alternatives and passed notes once in a while... For certain modelling problems you can save literal gpu days by going against the grain and get better results.
Most ML papers I used to read fell into a couple of categories...
1. Baseless empirical result that probably was p hacked.
2. Mostly untested result.
3. A rediscovery of something know for decades without attribution to the source they probably read it from.
4. Incomplete description of problem, resolution, or probably fraud/incompetency.
5. Useless in general. Unverifiable, etc.
6. Review article.
7. A derivation with way more omissions/flaws than say an engineering paper.
I'm somewhat seasoned on optimization methods personally, but yea it seems once people go ML they tend to um stop studying the fundamental literature that ML came from. "Online masters program learn AI in 12 weeks from nothing!". Oh okay so calculus won't be included in that... Or statistics... Or... Yep it's going to be scikit learn notebooks
...
It does feel like academic outsiders hacking/taking on the brand of serious academic research to give themselves authority.
> Baseless empirical result that probably was p hacked
This to me seems like the biggest regression in science. It's all heresy which is very hard to re-produce or learn general lessons from. It feels like disparate social science methodologies are being used to study math.
Nobody is going to look back and benefit from these papers. I often bring up to ML folks limitations proven in the book Perceptrons and wonder how their models differ. I have never gotten a response.
I once tried talking about how there is probably a fundamental limit to how deep of a graph traversal chat bots can do to a colleague. And he started blankly at me and said "it'll work". As if chat bots can now solve NP complete problems with ease because "AI"...
I'm so glad I left the field the level of snake oil relative to sustenance is pretty hard to stomach.
Maybe we are wrong about complexity theory. I know people who have dedicated good chunks of their lives to studying it. One things for sure, if we are wrong about it, and have no basis for studying any of its exceptions, it's hard for me to accept hot takes like this as worth considering. The general "we can do ...insert currently impossible thing... Because AI!" Gets very old. Once had a boss request that light travel faster then it does- literally... "no" wasn't an acceptable answer. Anyway...
I float between a few technical fields. Some in natural science, computer science, data science hybrid roles, data bases/engineering, etc. Not a jack of all trades, nor a master of none. What I do have mastered isn't something people hire for, so basically I am an averagely smart person who will take any job and figure it out to pay the bills.
At home though I play with all of the areas of creation I can get my hands on. I guess I am just in the field of discovering new things and making things.
A colleague and myself experimented with some alternatives and passed notes once in a while... For certain modelling problems you can save literal gpu days by going against the grain and get better results.
Oh well...