I think this field is suffering from some confusion of terminology. In my mind there are three subfields that are crystallizing that each have different goals and thus different methods.
The first one is Data Science. More and more businesses store their data electronically. Data Scientists aim to analyze this data to derive insights from it. Machine Learning is one of the tools in their tool belt, however often they prefer models that are understandable and not a black box. Sometimes they prefer statistics because it tells you if your insights are significant.
The second one is Machine Learning Engineering. ML Engineers are Software Engineers that use Machine Learning to build products. They might work on spam detection, recommendation engines or news feeds. They care about building products that scale and are reliable. They will run A/B tests to see how metrics are impacted. They might use Deep Learning, but they will weight the pros and cons against other methods.
Then there are AI Researchers. Their goal is to push the boundaries of what computers can do. They might work on letting computers recognize images, understand speech and translate languages. Their method of choice is often Deep Learning because it has unlocked a lot of new applications.
I feel like this post is essentially someone from the first group criticizing the last group, saying their methods are not applicable to him. That is expected.
> I feel like this post is essentially someone from the first group criticizing the last group, saying their methods are not applicable to him.
Probably more accurate to say that it's the first group criticizing others in the first group who try to act like people in the third group. Data scientists who use deep learning for everything, when a more interpretable model would do just as well.
Sounds very familiar in pattern to those people who were part of the 'big data' hype who would attempt to use Hadoop for everything.. even work that could have been accomplished with an Excel sheet and an hour.
Looks like us Engineers have a common pattern in running towards the next shiny thing ;-)
Somewhere along the journey/career some of us wise up and learn when to say No, and use the right tools for the job, and ignore the "cool" factor.
And the criticism you note isn't one-directional in the field at large. I'm finding that ML/AI researchers deriding ML/Data engineers and "scientists" as not doing "real" ML or AI is becoming a thing, similar to how some computer scientists deride engineering as not doing real computing.
This and the above comment kind of says my thoughts on the matter. I think there is a very real emerging field that combines mathematics, traditional computer science, and AI into a big family resemblance of quantitative methods and research, but that we're still in the early stages and we're still sorting things out. I remember hearing someone on hacker news joke that when he got out of school people called him a statistician, then people started saying he was a data miner, and now he sells himself as a data scientist, but he hasn't actually done anything differently apart from just keeping up with the field. Sometimes it feels as if we're just playing with words and people get excited for advances and what it means for the future, but the reality is people just keep plodding along make incremental progress, finding new ways to leverage the tools they have, and continuing to publish papers. For all of the tech industry's worship of "disruptive innovation" and massive changes, the reality is that we've been developing and refining our quantitative abilities and figuring out automation since the industrial revolution. AI may become less exciting to people 5 years from now, but we will still continue doing what we've been doing.
If you mean hype in media and general public, I agree with you. Research is inherently risky and uncertain, often that is not conveyed correctly. Also research results are often oversold.
If you mean that big tech companies are overinvesting in Machine Learning then I have to disagree. It's not a coincidence that the companies that invested the most in Machine Learning (Google and Facebook) are companies that have a lot of data and already use Machine Learning in their products. It provides things like better feed ranking and better signals for search. These "invisible" improvements to existing products are often overlooked.
And the criticism you note isn't one-directional in the field at large.
Yes, like I mentioned I think there is confusion about terminology. Many ML Engineers (in my definition) call themselves Data Scientists. This leads to misunderstandings, when people don't recognize that others have different goals.
You mean, some tech companies understand the difference between Data Science, AI in general, ML and Deep Learning and try to utilize the best tools in hand to handle their data.
But what is driving this hype, as well as the Blockchain hype, is not engineer-driven companies like Google, but rather MBA-driven buzzword-friendly tech companies (imagine Balmer-era Microsoft), non-tech companies who want to share the cake, the media and finally the investors who are misled by the rest, but end up creating a capital-based feedback loop for them.
As far as the people driving the hype train right now are concerned, Machine Learning is the way you do AI, and Deep Learning is just a more powerful (ahem deeper) version of Machine Learning. This is what AlphaGo used to defeat Lee Sedol, so it's obviously superior, and we should use it to process all data, in the same way we should strive to store everything on a blockchain, which is clearly superior to hash tables and databases.
This is the first time I'm hearing that computer scientists deride engineering as not-real-computing. Any references? It doesn't check out based on my background. In fact anecdotally, I've heard the reverse. I've heard EE and stat algorithms folks criticize CS ML/CV researchers for using algorithms as black boxes compared to the rigorous standards of EE/Stat (aka reviewer standards in IEEE Transactions in Information Theory, IEEE Trans Sig Proc, majority of stat journals). Now obviously there are a vast number of CS researchers who are competent at stat-level rigor. But there are also a large number of ML researchers (applied ML?) in CS departments who use algorithms as black boxes.
The prof of my applied ML class this semester jokingly refers to ML research's approach to statistics as "punk stats", which I find to be pretty accurate.
I'm imagining someone tweeting at Robin, several years back:
"Good Prediction Markets expert says: Most firms that think they want advanced Prediction Markets really just need linear regression on cleaned-up data."
under our recent article " Artificial Intelligence Generates Christmas Song".
Basically, if there is no pixie dust that makes humans intelligent, and instead it is a matter of the architecture of the brain and the first few years of supervised sensory input, then neural net breakthroughs (which use a similar architecture/topology) have the potential at any moment in time to break through and match general human intelligence.
What I mean is that if someone sent back source code from 80 years from now, but we had to run it on a bunch of Amazon / Google servers in a server farm, we're pretty much guaranteed to have enough computing power to do so!
(This is a combination of the number of neurons, their number of connections, and their very slow speed.)
We have the hardware.
Now: we do not actually have the source code from 80 years from now that we can go ahead and run on those machines.
So, we're at like heavier-than-air travel right before the Wright Brothers flew at Kittie Hawk. Except we have like jet engines already -- just no way to design them into something that flies.
I think that AI is vastly underrated and I watch with incredible interest every single breakthrough.
For the second category above, ML built into products or engineering solutions, Alphago surprised me, because Go had an intractibly large possibility space, it's not in any sense subject to brute-forcing or exhaustive search.
Dragon's Natural dictation surprised me in that using its model its able to get basically perfect dictation. I've never worked as a transcriptionist, but a small search reveals it has basically annihilated the industry of medical transcription.
These are not the big, general breakthrough.
But the big, general breakthrough is right there, somewhere. The results researchers are coming up with are astounding, and they're doing it in many cases with neural nets, quite similar to the wiring of the human mind.
The hype is faaaaaaaaaaaaaaar less than warranted for the stage that we're at. At any moment someone can put together something that achieves higher-level intelligence and can be set loose upon the world corpus of culture.
True, there are no clear indications that this is about to happen (For example: people do not extract innate language algorithms from the human genome which encodes them), so we are not exactly taking many steps that we could be trying to. As far as I know we're not even genetically engineering people to see what different parts of DNA do -- which is obviously a very, very good thing, and who would allow anyone to bear to term a child made as an experiment to see what DNA does.
But despite not going from a human starting point, the results that we are achieving in many cases match and surpass human ability - while we do know that in many cases some of their architecture is similar. I feel quite strongly that we have more than enough hardware for general intelligence - and I see advances every day that could end up going past the ppint of no return on it.
--
EDIT: got a downvote, but I would prefer a discussion if you think I'm wrong.
I didn't downvote you, but the TL;DR of the article is that most ML demonstrations to date have been toys and there are no known real-world applications that would justify the "40% of jobs lost!" hype.
And you're trying to rebut this by referencing an AI-generated Christmas jingle. I think the author rests their case...
A lot of revolutionary technologies begin as toys. In the 9th century two brothers created a machine called, "the instrument which plays by itself." It uses a rotating cylinder with "teeth" that triggers keys which were encoded to play music. One can say it's a programmable music machine.
Fast forward to the 1700s and artisan engineers in France were creating similar devices that automatically played music. Toys for the entertainment of the aristocracy. Some insightful engineer noticed the use of pinned cylinders to recreate music, and realized the device might also be used to create textiles with a loom. You could program a machine to create textile patterns. While the initial implementations didn't work very well, eventually Joseph-Marie Jacquard created a version that revolutionized textile manufacturing in the early 1800s, relying on punch cards instead of expensive-to-create cylinders.
By the mid-1800s Charles Babbage became interested in Jacquard's work using punch cards to create complex textile patterns. "The major innovation was that the Analytical Engine was to be programmed using punched cards: the Engine was intended to use loops of Jacquard's punched cards to control a mechanical calculator, which could use as input the results of preceding computations." - https://en.wikipedia.org/wiki/Charles_Babbage#Analytical_Eng...
I think your story does more to reinforce the parent post's point. The chain of events you outline spans an entire millennium. Over that time, yes, a lot of technology developed, and eventually we got to computers, and there was even the movement that gives us the term Luddite. But it was still one long chain of gradual, evolutionary developments over the course of a very long expanse of time. At no point was there anything like the kind of sudden turmoil that futurists like to claim that deep learning will produce.
If the world evolved in linear terms, you would be right. However, the rate of scientific advancement has multiplied exponentially over the history of mankind: our scientific understanding has advanced more in the last 100 years than it has in the previous 10000 combined (nuclear physics, quantum physics, molecular biology, information technology, astronomy/cosmology - you name it.)
Remember, 10 years ago smartphones were in the realm of science fiction. Now, everyone has one in their pockets.
Today we can train a computer for a week to generate a cute little christmas jingle. How many years do we need to train a human child to do the same? AGI won't happen overnight, sure, but its effects of that research will be felt continuously along the way.
We never know, that is what the original parent comment was about. Regarding gradual progression, remember - we are now in a highly connected world, where things move at tremendous pace. And a lot more population have access to the latest information. Just like the invention of the wheel, or the invention of steam engines, or the computers, transistors etc, there could be a tipping point, which could happen today as I write or tomorrow or even after a decade. We can only guess. But looking at the speed of things, I would say the probability of it happening very near is very high and is worth the hype.
The '40% of jobs lost' figure is reachable without any Singularity-style event. That's what you can get just by applying modern automation techniques to low hanging fruit. Self-driving vehicles, automatic checkouts at supermarkets, fully automated fast food places, robotic warehouse storage/retrieval etc.
If you used to have 10 people working at a supermarket and you now have 6 with self-checkouts, that's your 40% jobs lost (for that supermarket) right there.
Self driving vehicles are not a low hanging fruit. We are far, far away from fully autonomous vehicles that don't need a human backup driver. This is despite massive investment by Google and others.
True, I shouldn't have grouped it in with self checkouts etc. When we do achieve reliable self driving capability it will be arguably the single most impressive engineering feat we've accomplished.
Tesla are describing their 'hardware 2' self driving platform, fitted to all of their new cars, as "full self-driving hardware", and they've posted several videos recently of fully autonomous journeys on public roads. They've said they plan to roll out an OTA update to enable full self driving next year.
Mercedes-Benz have a limited self-driving mode on their 2017 model.
Uber-owned Otto, a company focusing on self-driving technology for trucks, plans to start offering its services next year.
And of course Google is quietly continuing their self-driving project, although they've gone a bit quiet recently.
Now, it does look to me like we've still got a fair few corner cases to straighten out, but either all of these companies are badly wrong, or we'll be seeing self-driving vehicles on sale to the public in the next few years (even if not next year).
Obviously it is hard to know when that magic moment will happen that some kind of general AI is created that can learn in some sense similarly to how humans do. My every indication and astonishment at the results that are being produced strongly suggests "at any moment". The results are absolutely astonishing every day and we have vastly more than enough firepower.
You might also be interested in this separate thread where I dealt with questions of consciousness and pain. I again reference it here:
> Obviously it is hard to know when that magic moment will happen that some kind of general AI
> is created that can learn in some sense similarly to how humans do. My every indication and
> astonishment at the results that are being produced strongly suggests "at any moment".
As an IT guy with a basic but solid neuroscience education (which isn't even needed for what I'm about to say): Yep, you are on the hype train, and very deeply. I really like reasonable discussions, I have no idea what this is right here. We will see more amazing results, sure - but applications will be specialized narrow subjects. From creating "some kind of general AI is created that can learn in some sense similarly to how humans do" we are still very far away. Your statements remind me of 1960s "future" hype,a nuclear reactor in every car by the year 2000, stuff like that.
My hype is different, because in my estimation we already have the hardware. You write:
>As an IT guy with a basic but solid neuroscience education
-- could you go ahead and take a few minutes (maybe will take you 5-10) to read through my above-referenced links referencing my previous discussion and tell me whether I'm correct in your estimation on the bottom-up aspect - i.e. the amount of computation that human neural nets can likely be doing, and how it compares to server farms with fast interconnects today?
I'm not an expert in neuroscience so your feedback might be helpful there.
don't you think it would have a lot more bearing if you had 7 billion devices nonchallantly walking around cracking RSA every day using the same or less hardware? (but we couldn't reverse-engineer them, because they were obfuscated in biology)?
The fact that they weren't reverse-engineered (yet) would still have huge bearing on everything.
By 7 billion samples I mean the humans walking around. Your analogy with an RSA crack is fundamentally different beccause biology doesn't do it in 3 pounds of grey goo in seven billion different bodies already.
so you would have to come up with an analogy that uses something we cant use, to say, okay fine it exists and fine, we have the hardware to also do it, but the former doesn't have any bearing on us doing the latter.
How much computing hardware does it take to beat a human at chess or at go? (Less than you think, Deep Blue was hopelessly inefficient.) And how long did it take for those much simpler things to happen?
Interesting observation. If brains routinely cracked RSA, that could be evidence that P=NP.
Still it wouldn't help us find the P-time algorithm in question. We could say "it seems to exist", but that would not imply "we'll discover it any day now".
For brains numbers have a completely different meaning and internal representation than for computers. Brains don't "think in numbers". Doing the kind of math we invented is a major effort for the brain, it's not what it developed for, and it is very poorly equipped to do explicit numerical calculations (emphasis on "explicit"). So looking at brains to "crack RSA" seems like waiting for a hammer to be useful in driving screws.
> i.e. the amount of computation that human neural nets can likely be doing
We don't know nearly enough what they are really doing! We only know a few selected bits and pieces! You are basing your assumptions on nothing, so according to logic any conclusion is possible from a faulty premise. On which you promptly deliver spectacularly.
Also, the "computation" a brain does at one moment leaves out the time aspect: Lots of things lead to constant changes. The wiring changes all the time. The "computation" metaphor has little use for describing or understanding this major aspect of "brain". The more I learned about neuroscience the more unhappy I got with the computing metaphor that I had had going in (as a CS graduate, naturally, I think). The brain is so very, very different from my pre-neuroscience-courses notions.
How much neuroscience do you know? If the answer isn't at least an undergrad introductory course (the accompanying book is over a thousand pages), why do you get the idea you can make any predictions?
Read this to read about complexity in biology vs. engineering and what scientists in the field think how well we are dealing with it:
>if there is no pixie dust that makes humans intelligent [...] then neural net breakthroughs [...] have the potential [...]
You're assuming neural nets are the right model.
Like a 19th century person saying, "if there is no pixie dust ... then eventually Newtonian mechanics will explain these unexpected wobbles we see in the planets' orbits."
Yes. Furthermore, neural nets are just one small part of the solution to [edit] general intelligence. A truly scalable intelligence that learns on its own, and doesn't rely on "the right answers" through training data by human experts. Without this training data, neural nets can't do much ...
I kind of doubt if neural nets have to be the right model.
Any model that gives reasonably similar results to human thinking will be useful. Planes don't have feathers and don't eat bugs, but we say that they "fly" anyway, and they carry far more cargo / passengers (and faster) than birds ever could.
What do you mean "assuming" it's the right model - a known-good model is the model in human brains, which is definitely a series of neurons, no? what else would it be?
Here's one: neurons are cells, and cells are really complex and have all sorts of internal state on different scales (organelles, molecules, etc). Neural network models in computers are an extremely simplified abstraction by comparison. And that's still ignoring all the other things that may or may not be happening in the brain beyond the connections betweens neurons themselves.
I think you've fed majorly into the hype but to address your point there was a great comment on neurons (from someone who knows much more than I do) just a week ago:
https://news.ycombinator.com/item?id=13030766
I read your consciousness post and will reply here because that thread is already locked. I think the problem of consciousness comes from its definition. The word consciousness is too large, encloses too many aspects of cognition, religion and philosophy to be useful. It's a suitcase word, reifying various meanings into one name.
What we need instead of "consciousness" is a set of concepts:
- "perception" - means representing inputs in such a way as to be able to adapt to the environment to achieve goals (I place qualia here)
- "judgement" - the ability to select the best action for the current state
- "reward" - comes from time to time, and is used as a signal in behavior learning
These three concepts : perception, judgement and reward are simpler, clearer and less ambiguous. They have implementations in AI, not human level yet, but getting there. These concepts make the problem concrete instead of bringing up 2000 years of attempts to "get consciousness" based off armchair philosophising.
A more interesting question than "what is consciousness?" would be: what are the reward signals that train the human brain. We do Reinforcement Learning in the brain, but the reward signal is complex, made of multiple channels, which are evolutionarily optimized for our survival. It's the engine of drive that puts us in motion. We need to reverse engineer that in order to replicate human consciousness in silico.
Consciousness is a replicator, a system that is concerned with survival and maintaining balance in the face of perturbations and entropy. What it does is to make us go find food, make babies and protect from danger - essential actions without which the human species would disappear, and consciousness with it. Or, in other words, it maximizes its rewards over time, reward being having food, shelter, company, access to learning and a few more. But it's just limited number of reward types, and they are much simpler to understand than consciousness itself.
So consciousness is just that thing that maintains itself instead of becoming disorganized and dissipating out like most physical processes that lack a self stabilizing, self replicating dynamics.
Consciousness is a balancing system for the colony of cells that just want to pass their genes into the next generation.
>Consciousness is a balancing system for the colony of cells that just want to pass their genes into the next generation.
is there anything in that which requires that it be a physical biological substrate rather than a model? Can't computer models follow the same behavior as long as that's how their environment / system is set up? (More specifically, can't they also have brains made up of cells connected to each other, in a way closely analogous or even an exact simulation of chemical pathways?)
I think there is absolutely no need for the system to be biological. A computer model can work just as well. As long as it solves the same behavior problem, there is no difference between brain reinforcement learning and artificial reinforcement learning.
80 years is very conservative. Rich Sutton has a very well articulated argument for achieving human level hardware (using Moores law, and an estimate of the computational power of the brain derived from measurable computations in the eye (retina?).
But as you said, it's the algorithms that are by far the long pole. Current supervised learning is much akin to simple rote learning. This is the promise of reinforcement learning - and the ability to truly learn on your own through experience. That's scalable.
*that said ... I'm biased being one of Rich's students :(
I hate this. Remember they did it to Michael Faraday in 1813, deriding him as not a real scientist as he was about to play a part in advancing one of the greatest challenges of mankind (unification).
Not a bad way to put it. You could make his argument valid though by discussing the economic value of the problems solved by the first category of people vs. the third. Right now I agree with him that the first is bigger than the third. But I believe that balance is starting to tip and the potential value of the third will keep increasing relatively to the first to the point of dwarfing it (hence no bust).
I don't think that there is a "balance" to tip, it's just that the previously theoretical work of the latter group is starting to become practical to the former in a business setting. Neural networks have been a topic of research for many decades before consumer demand for graphical video games drove the technology to make them practical. Backpropagation was first described as a solution to efficiently training multi-layer neural networks in 1975 but only in the last decade have we developed the infrastructure for non-CS researchers to use complex neural networks using a sub-$1000 teraflop GPU and an iPython notebook.
Off the top of my head the closest analogy would be number theory, especially the study of prime numbers. Before information technology in general, number theory was esoteric and considered useless by many pure and applied mathematicians (I'm simplifying a bit for argument's sake) but all of that accumulated research proved massively useful once we started to communicate electronically. WWI and II cryptographers didn't become or absorb number theorists as a group, they just adapted the knowledge to their field under the umbrella of electronic warfare. I think this is happening with data scientists, who are starting to experiment with ML but it's still just another tool in their toolbox. The media hype train focuses on the flashy AI contests and muddles the terminology but the real work [1] is happening behind the scenes in data science.
[1] By "real work" I mean work that directly translates into value on a company balance sheet. The theoretical work is important in and of itself and has been happening for decades.
Good question.
Just one anecdotal data point here ... So take it for what it's worth.
I'm a grad student focusing on reinforcement learning but have a lot of interaction with many deep learning folk. I'd have to say that they seem mostly motivated to learn how to make deep learning even more powerful and how to apply it. Not so much solving what's going on inside the box.
We do understand in general, just not every neuron and input weight in particular, because they are too many for limited human working memory to hold at once.
An automated labyrinth could be solved by hand up to a size, but if it becomes larger, it would become "impossible to understand". We'd need to rely on computers to find the route for us.
It's not computer magic, just an ability to hold more data at once. We're limited to 7-8 things - try to remember a string of more digits and see if you can - it's hard, we just have that kind of limitation. So any computer algorithm that can't be broken down into 7-8 understandable parts is hard to grasp.
Of course we can make ML tools synthesize the preferred input of any neuron. That does help a little.
For many researchers, "understand" relates more to the underlying math. Things like "what kind of convergence properties and guarantees are there" and "formality, what can this model learn and not learn"?
I understand that most people working with deep learning wouldn't want this type of thinking to spread amongst the public, and I surely don't want it either.
But you have to be totally unaware of reality to think that DL is the definitive tool for AI.
Most impressive results in DL in the past 2 years happended like this:
>deepmind steals people from the top ML research teams in univerisites around the world
>these people are given an incredible amount of money to solve an incredibly complex task
>a 6000 layers deep network is run for 6 months on a GPU cluster the size of Texas
>Google drops in their marketing team
>media says Google solved the AI problem
>repeat every 6 months to keep the company hot and keep the people flow constant
>get accepted at every conference on earth because you're deepmind (seriously, have you seen the crap that they get to present at NIPS and ICML? The ddqn paper is literally a single line modification to another paper's algorithm, while we plebeians have to struggle like hell to get the originality points)
I'll be impressed when they solve Pacman on a Raspberry Pi, otherwise they are simply grownups playing with very expensive toys.
Deep learning is cool, I truly believe that, and I love working with neural networks, but anyone with a base knowledge of ML knows better than to praise it as the saviour of AI research.
Rant over, I'm gonna go check how my autoencoder is learning now ;)
> I'll be impressed when they solve Pacman on a Raspberry Pi
I think this is the thing people don't quite get when they buy into the hype. These systems are extremely inefficient. Requiring terrabytes if not petabytes of data and basically a powerplant next to a data center to power the whole thing.
The work is valuable and pushing the boundary on what the hardware can do is great but so far all these things lack any kind of explanatory power and suck up a lot of energy to power the black boxes. DARPA recently put out a research program for making systems more efficient and adding explanatory capabilities to them (http://www.darpa.mil/program/explainable-artificial-intellig...). Ultimately that is the direction these things must be headed if they are to provide real value for the masses. Relying on a clever black box only takes you so far and is not beneficial in the long run because as these systems become more integrated into the institutions that drive large scale decision making they'll need to be held accountable for those decisions.
It's not all hype. We are using deep learning to solve real issues in our research, things that might not get done as well due to the representational learning aspect and our relatively small manpower. It's a definite advance when it comes to improving state-of-the-art and will have impact across all sorts of things, particularly small groups who have problems but don't have hundreds of people and ten years to figure it out.
The powerplant criticism is more true of the training phase and much less true of using the resulting networks on a larger scale. The good thing about the "hype" is that more resources are being directed into the field so they'll be more efficient processing platforms (e.g., ASIC or FPGA) and better delineation of what's really needed and if there are possible shortcuts (e.g., ReLU).
The black box problem will prevent its use in some systems, but even in some areas of medicine, it will be fine because medical AI must be used in conjunction with the final decision-maker, much like how Watson is being positioned. A deep learning system that detects anomalies in patient imaging with very high precision will be useful even if it can't explain why it thought it was an anomaly. It's quality control for the radiologist, etc.
I think that's a good use case. Such use cases are great and I agree it's not all hype but when everyone and their grandma has an "AI" startup you have to wonder what exactly "AI" means at that point.
I can't speak of their other work, but as someone who plays go and follows go bots, the AlphaGo project was a phenomenal piece of work that shouldn't be discounted. Go is a very difficult game that requires great intuition and a deep understanding of the patterns.
Sure, maybe their timing was fortunate and it might have happened a year later anyways. But this is definitely not a guarantee, and they were still the ones to do it.
P.S. For anyone who is interested, another very strong bot is out in the wild now. It has been playing on the KGS go server the past week under the name Zen19L, and has a rank of 9d. Some great games have resulted as people challenge it.
So is image recognition, but I think the main contention is that solving Go required little additional cleverness to what has already existed. The ideas of self-play, Monte Carto methods, and neural networks are not new and not novel for the problem. Not that it's so trivial -- I'm sure it took a while to work out the exact architecture and details, but what of that work actually teaches you something?
The combination of the methods was certainly novel as far as I am aware. Also, it was not clear ahead of time that it would actually succeed in defeating the best humans. Demonstrating that it could be done is a grear achievement.
There is also the symbolic value. It was a "coming of age" event to a certain degree. I believe go was the last classic board game that researchers had been longing to conquer.
I haven't looked deeply into this, but my understanding is that image recognition is still somewhat subpart outside of well-curated datasets. Is that not the case anymore?
By that token, IBM's wins at chess and Jeopardy also deserve to be seen as "coming of age" events. And while the wins certainly showcased IBM's engineering prowess, I'm unconvinced Deep Blue or Watson moved the AI needle in a meaningful way. If they had, others would have followed in their technical footsteps. But AFAIK, no one has. I believe this lack of high tech repercussion will be true of AlphaGo too. Novel AI for game play just doesn't transfer outside the game.
You don't think Deep Blue was inspirational to folks who were perhaps deciding if they wanted to pursue AI research?
The same holds for AlphaGo and Watson. Also, my understanding is that they are more technically interesting than Deep Blue. Given how recent these projects are, their legacy is only beginning to unfold.
I agree with you that applying this tech to domains other than games is no easy challenge. But I would be very surprised, in the long run, if events like this are not documented as key steps along the way at some point in the future.
My problem is that what people seem to be saying now is that "computers are smart enough to solve Go now, thanks to Google".
1) Too much credit is given to Google for this result. What I see as already a huge brain drain on society is only going to get bigger.
2) People are going to expect that if "computers are smart enough to play Go", they're smart enough to do ____. What goes in the blank? Very little right now, but I guarantee you investors and the public have a lot of ideas and think it's around the corner.
Hopefully a lot of good things will come out of this wave of AI, but who knows what or when. I think the point is that there may be a panic and contraction before anything really awesome happens, and a lot of that is going to be because of "stunts" (for lack of a better word) like this being overhyped.
Agree generally.
Except being unimpressed unless performance is achieved on sub Google scale hardware.
Today's Google supermachine is tomorrow's raspberry pie. No need to artificially constrain our bounds. There is, after all, the inevitability of Moores law.
Moore's law has been dead for a while now. Most of the chip in your phone is powered off because otherwise it would burn up. Highly recommend watching this video: https://www.youtube.com/watch?v=_9mzmvhwMqw
Really thanks for posting this video. Very cool to see Sophie Wilson talking so elequently and with obvious authority on a subject I and many others deeply care about. Great!
True, but even without Moore's Law, using neuromorphic chips instead of general-purpose CPU/GPUs would likely be much more efficient. In the meantime it makes sense to use large server farms to emulate candidate neuromorphic architectures.
Interesting. I've yet to hear a Moores law is dead argument, so perhaps I should watch the video before commenting further. But the fact that most of the chip is turned off, doesn't falsify the fact that most of it still exists. Cooling it properly is a separate problem independent of computation no?
I'm not sure solving Pacman on a Raspberry Pi would be terribly worthwhile. Deepmind's research into getting their software to play Atari games was interesting because they used a general algorithm that just received the values of the screen pixels and game score and learned from there. They used the same software for each game - it wasn't tuned or set up differently and so was a form of general AI if a very sub human one. It was interesting research that it worked as well as it did and played some games very well but wasn't good as Pacman because it lacked the ability to plan ahead.
They are primarily a research company, not a marketing setup. The latest stuff the released on letting the networks do something like dreaming which reduced the time to learn by up to 10x seemed interesting and I look forward to seeing how they do with Starcraft and the hippocampus. A lot of this stuff is cool because it gives insights into the human mind and how the brain works rather than practical gadgets. https://www.extremetech.com/extreme/240163-googles-deepmind-...
When I was at Watson this is the first thing I told every customer: before you start with AI are you already doing the more mundane data science on your structured data? If not, you shouldn't go right away for the shiny object.
This said I still believe the article is mistaken in its evaluation of potential impact (and its fuzzy metaphore of pipes). Unstructured or semi-structured or dirty data is much more prevalent than cleaned structured data on which you can do simple regression to get insight.
Ultimately the class of problems solved by more advanced AI will be incommensurably bigger than the class of problems solved by simple machine learning. I could make a big laundry list but just start thinking of anything that involves images, sound, or text (ie most form of human communication).
And before you do mundane data science on your structured data, you should figure out if there is a better way to get cleaner raw data, more data, as well as more accurate data.
For example, I predict stereo vision algorithms will die out soon, including deep-learning-assisted stereo vision. It's useful for now but not something to build a business around. Better time-of-flight depth cameras will be here soon enough. It's just basic physics. I worked on one for my PhD research. You can get pretty clean depth data with some basic statistics and no AI algorithm wizardry. We're just waiting for someone to take it to a fab, build driver electronics, and commercialize it.
Stereo vision is obviously highly effective in biology as it has independently evolved a great many times. Time-of-flight may be poised for a renaissance, but it scales badly and is active, not passive. Stereo vision, and its big brother light fields, are far more general and are certainly not going to "die out".
> Stereo vision is obviously highly effective in biology as it has independently evolved a great many times
This argument sounds like a second cousin of the "chemicals are bad, but if it is natural it is good" argument. Just because it evolved in nature doesn't mean it's optimal, or it's the best system under massively different constraints. And who knows what evolution would have thrown up after a few more billion years.
I think the point he was making is that it is effective in different environmental challenges and constraints. Not that it's optimal. So it might still be useful for building into a robot that's supposed to explore a environment(think space and planets for example). It may get relegated to academia, if we find other optimal solutions for our robots doing our boring chores though.
Human vision is active rather than passive as well. We really wouldn't be able to get around the world if we relied on passive inference from sensory stimuli to model our environment.
Terrible for what purpose? Humans seem pretty good at throwing things to each other and catching them.
I'm very bad at coming up with a good numeric estimate of linear size. As a fencer, I could never tell you how many inches between me and my opponent, how long his or my arms are, how tall he is, etc. I could definitely tell you which parts of our bodies are within reach of each other's arm extension, fleche, lunge, etc.
Terrible for the purpose of proving a general-purpose stereo machine vision system is practical.
The distance at which a stereo vision system can capture precise depths depends on the distance between eyes, and the eyes' angular resolution. Human depth perception works well for things within about 10m, but when you get out to 20-40m humans get a lot less info from stereo vision.
When you get to that distance, humans seem to have a whole load of different tricks - shadows, rate of size change, recognising things of known size, perspective and so on. You can see a car and know how far it is even without stereo vision, because you know how big cars are, and how big lanes and road markings are. You can even see two red lights in the distance at night and work out whether they're the two corners of a car, or two motorbikes side-by-side and closer to you.
On the other hand, your basic general-purpose stereo machine vision system doesn't try to understand what it's looking at - you just identify 'landmarks' that can be matched in both images (high contrast features, corners etc) and measure the difference in angle from the two cameras. This is relatively simple and easy to understand!
For tasks that humans can do that involve depth perception of things more than ~40m away - flying a plane, for example, where most things are more than 40m away if you're doing it right! - nice simple stereo vision can't get the job done, because humans are actually using their other tricks.
Of course, despite this limitation stereo vision comes up a lot in nature - it's still a beneficial adaption, because most things in nature that will kill you do so from less than 10m away :)
> Of course, despite this limitation stereo vision comes up a lot in nature - it's still a beneficial adaption, because most things in nature that will kill you do so from less than 10m away :)
It's actually pretty rare for non-predatory animals to have good stereo vision. Most of them are optimized for a wide field of view instead, evolving eyes placed on either side of their head. Think rabbits, parrots, bison, trout, iguanas, etc.
It depends what you mean by "practical", though. If you mean "practical for use as a dense 3D reconstruction technique" then sure, it's pretty bad. If you mean "usable under an incredible range of lighting conditions and situations" then I'd say it's practical.
Edit: IMO binocular vision is probably more to do with redundancy than depth perception. If you damage or lose an eye, you can still operate at near full capacity. Losing vision 'in the wild' is a death sentence.
Right but in this case there is a trade-off: if you don't have the better data now how long before you will? Should you start now (and gain a time advantage at a cost) with what you have or wait for the better data (read sensor/process/technology/people etc).
My experience is that waiting for cleaner data is often like waiting for Godot and will often be a project killer (sometimes justifiably). This is a key issue at the moment in advanced ML: clean large training sets ideal for supervised training are elusive and the companies making real-world advances are pretty much all using available data (and semi-supervised techniques) rather than expensive made up training sets.
>Better time-of-flight depth cameras will be here soon enough. It's just basic physics. I worked on one for my PhD research. You can get pretty clean depth data with some basic statistics and no AI algorithm wizardry. We're just waiting for someone to take it to a fab, build driver electronics, and commercialize it.
You should talk to us at Leaflabs. Commercializing research-level tech in embedded electronics is what we do.
> For example, I predict stereo vision algorithms will die out soon, including deep-learning-assisted stereo vision
Maybe for regular "cameras", but parallax still gives the most accurate results for 3D reconstruction from satellite images, where it is a very lively area of research. The resolution, and surface coverage, of radar satellites is far worse than what you can obtain by stereo matching optical images taken by the powerful telescopes on satellites. I guess in other fields like microscopy it makes a lot of sense also. Not all imaging happens indoor on commodity cameras!
However as ma2rten eloquently describes, for the purpose of AI research it might be beneficial to bite through the harder AI required when only having access to stereo vision.
A lot of problems are not tackled on a fundamental level. Occlusion, context, proprioception, prediction, timing, attention, saliency, etc.
A simple rat has more intelligence than whatever is behind a dashcam, security cam or webcam.
You're ignoring form factor for optimized technology.
Yes, a quality TOF system would be great. However good luck convincing consumers to adopt hardware with lidar on it. The Tango is having enough trouble on it's own and it does pretty well for consumer systems with IR.
Besides that you can't do FTDT with laser systems AFAIK. You need something to capture unseen places, such as ultrasonics/HF - which I guess you could argue fall into TOF but I haven't seen that work done.
In the end my money (literally!) is on the opposite if your approach, namely building better RGB systems because there are already a trillion cameras deployed that we can extract from.
I think the hidden strife in your triplet is a land mine. That is, getting more data is at odds with cleaner data for most folks. Adding accuracy? I hope you remember the difference between precision and accuracy. :)
Any links to your papers? How much better have things gotten over e.g. Kinect v2, which was pretty bulky and power hungry, and had active cooling? I'd imagine it has gotten a lot better.
A light source built into the camera is modulated in time. The camera uses this to infer depth. When the modulation is synced up right, the camera can see the light wave "travel" as it illuminates close objects first and then farther ones over successive frames. The camera isn't actually fast enough to capture the light wave traveling, but the timing between the shutter and the light source is shifted by nanoseconds each frame to accomplish this.
They can have problems operating outside because it is hard to make a lightsource brighter than the sun.
That's an older form of ranging. It's easier to do than pulse ranging, but you have to outshine ambient light at the color being used full time. The Swiss Ranger [1] is a good example of such a system. It's indoor only and short range. With a pulse system, you only have to outshine ambient light for a nanosecond, which is quite possible in sunlight.
I've been expecting good, cheap non-scanning laser distance imagers for a decade. In 2003, I went down to Advanced Scientific Concepts in Santa Barbara and saw the first prototype, as a collection of parts on an optical bench. Today ASC makes good units [2], but they cost about $100K. DoD and Space-X buy them. There's one on the Dragon spacecraft, for docking. That technology isn't inherently expensive, but requires custom semiconductors produced with non-standard processes such as InGaAs. Those cost too much in small volumes. There's been progress in coming up with designs that can be made in standard CMOS fabs.[3] When that hits production, laser rangefinders will cost like CMOS cameras.
Hey, 3D vision system amateur here, but very interested to learn more!
Can anybody point me to some literature or reference materials about attempts to combine the inputs from multiple techniques simultaneously?
E.g. a device with stereo conventional cameras and infrared cameras & emitters which compares the resulting model from each input source/technique and actively re-adjusts final depth estimate?
Is "sensor fusion" the right jargon to use in this context?
Or, even crazier, a control system which actively jitters the camera's pose to gain more information for points in the depth map with lower confidence scores / conflicting estimates?
But maybe such a setup is overly complex and yields minimal gains in mixed indoor & outdoor scenarios?
We have a setup that combines ToF, structured light and multiple colour cameras to reconstruct hands from the elbow down. Short version: it's a massive pain in the ass. In fact the setup really only works because we have a preconceived motion model (particular hand gestures) and have carefully arranged the scene to avoid interference. I'm unaware of a general solution where you can just throw more cameras in and get better scene data.
One neat thing though you might want to look at: if all you have is structured light (ie Kinect v1) you can simply attach a vibrating motor to each emitter/receiver to avoid a lot of interference per [0]
No, that would also qualify as "AI algorithm wizardry" as dheera puts it. Instead it's just a hardware sensor that measures the time at which light arrives at each pixel. Knowing the speed of light you can calculate the depth of each pixel.
So the basic criticism here is than Hanson, in suggesting firms clean up their data and then apply simple analytics to it, is defining away the problem that ML solves?
Hanson is economist, not a computer scientist. He may not be right about exactly why the hype is unfounded. But he is not wrong about the fact that it is unfounded. There is no "awe-inspiring" rate of success with these projects, no massive revenue streams that are boiling up all over the landscape, which is absolutely what would have to be happening already to possibly justify predictions of "47% of US jobs in 20 years".
The giant piles of dirty data are that way because for thirty years no one has considered them worth cleaning up. How will they create such astounding amounts of unexpected value?
> The giant piles of dirty data are that way because for thirty years no one has considered them worth cleaning up. How will they create such astounding amounts of unexpected value?
It was left that way because we didn't have the tools to process it. Imagine the amount of unprocessed video data that we can now annotate pretty accurately. What's the value of that data now?
Most firms that think they want advanced AI/ML really just need linear regression
That's how AI always looks in the rearview mirror. Like a trivial part of today's furniture. Pointing a phone at a random person on the street and getting their identity is already in the realm of "just machine learning" and my phone recognizing faces is simply "that's how phones work, duh" ordinary. When I first started reading Hacker News a handful of years ago, one of the hot topics was computer vision at the level of industrial applications like assembly lines. Today, my face unlocks the phone in my pocket...and, statistically, yours does not. AI is just what we call the cutting edge.
Open the first edition of Artificial Intelligence: A Modern Approach and there's a fair bit of effort to apply linear regression selectively in order to be computationally feasible. That just linear regression is just linear regression these days because my laptop only has 1.6 teraflops of GPU and that's measley compared to what $20k would buy.
The way in which AI booms go bust is that after a few years everybody accepts that computers can beat humans at checkers. The next boom ends and everybody accepts that computers can beat humans at chess. After this one, it will be Go and when that happens computers will still be better at checkers and chess too.
> Open the first edition of Artificial Intelligence: A Modern Approach and there's a fair bit of effort to apply linear regression selectively in order to be computationally feasible.
Does the book mention linear regression at all? The term doesn't appear in the index.
It may (or may not) have been a bit of stretch on my part for which the only excuse I have is that I was going on memory and the phrase triggered my memory.
The first edition discusses Least Mean Squares LMS which is in a way [that way being my way maybe] the 'evil twin' of Least Squares. It fits the data points (in this case the values produced by the agent) to a known curve rather than the curve to the data points. In the first edition, 'long running' examples are on the order of a few hundred to a thousand epochs and fitting a curve via linear regression is similar computationally.
Anyway, I'll claim poetic license if I must. And I must if my hand waving doesn't work.
Thanks. The section "Regression and Classification with Linear Models" from the third edition and the chapter "Statistical Learning Methods" from the second edition do not appear in the first edition.
Robin's post reveals a couple fundamental misunderstandings. While he may be correct that, for now, many small firms should apply linear regression rather than deep learning to their limited datasets, he is wrong in his prediction of an AI bust. If it happens, it will not be for the reasons he cites.
He is skeptical that deep learning and other forms of advanced AI 1) will be applicable to smaller and smaller datasets, and that 2) they will become easier to use.
And yet some great research is being done that will prove him wrong on his first point.
One-shot learning, or learning from a few examples, is a field where we're making rapid progress, which means that in the near future, we'll obtain much higher accuracy on smaller datasets. So the immense performance gains we've seen by applying deep learning to big data will someday extend to smaller data as well.
Secondly, Robin is skeptical that deep learning will be a tool most firms can adopt, given the lack of specialists. For now, that talent is scarce and salaries are high. But this is a problem that job markets know how to fix. The data science academies popping up in San Francisco exist for a reason: to satisfy that demand.
And to go one step further, the history of technology suggests that we find ways to wrap powerful technology in usable packages for less technical people. AI is going to be just one component that fits into a larger data stack, infusing products invisibly until we don't even think about it.
And fwiw, his phrase "deep machine learning" isn't a thing. Nobody says that, because it's redundant. All deep learning is a subset of machine learning.
I'm skeptical of claims about a one-shot learning silver bullet, unless people are talking about something different from how it has been classically presented, .e.g. Patrick Winton's MIT lectures. Yes, you can learn from a few examples, but only because you've imparted your expert knowledge, maintain a large number of heuristics, control the search space effectively, etc. There's a lot of domain-specific work required for each system, so I consider it more an approach of classical AI and not something that figures out everything from the data alone, like deep learning.
But again, maybe people are talking about something different than my above description when they talk about one-shot learning today. Either way, I don't think having to rely on a lot of domain specific knowledge is necessarily a bad thing.
Disclaimer, I haven't looked at their work to closely.
I think the process they use to select the hyperparameters overfits on the labels in the validation set. The true size of the labelled data includes the 100 from the training set and all labels in the validation set.
>> One-shot learning, or learning from a few examples, is a field where we're making rapid progress, which means that in the near future, we'll obtain much higher accuracy on smaller datasets.
I'm really not convinced by one-shot learning, or rather I really don't see how it is possible to show that any technique used generalises well to unseen data, when you're supposed to have access to only very little data during development.
Even with very thorough cross-validation, if your development (training, test and validation) set are altogether, say, 0.1 of the unseen data you hope to predict, your validation results are going to be completely meaningless.
I think there are a lot of programmers try playing around with "deep learning" and it doesn't work for them. But they lack the knowledge necessary to make it work, such as calculus, statistics, signal processing theory, ect.
This article matches what I've been seeing anecdotally (especially at smaller tech firms and universities in the Midwest US).
I've been hearing more folks in research and industry express the importance of applying simpler techniques (like linear regression and decision trees) before reaching for the latest state-of-the-art approach.
Saying that linear regression is easier to do properly than more complex methods like random forests, DL, boosting etc is like saying that people should code assembly instead of python
This is a false dichotomy. Both OLS regression and, say, random decision forest regression have the same objective (predict values) and achieve it with similar means (build a generative model / function). They solve the same problem. Contrastingly, assembler and python are broadly aimed at completely different use cases.
Broadly, whether you should move from OLS to random forest regression = SNR increase / increase in manhours and money spent.
It is actually much easier to apply a random forest (or really gradient boosted decision tree, which almost strictly dominates random forests) than a linear regression. Decision tree methods require far less data preprocessing than linear regression, because the model is able to infer feature relationships. Obviously if your features are linearly related to your target than linear regression is much more viable.
This is absolutely true, the one caveat is that you can explain the significance of features and the relationship to the response variables in simpler terms.
Technically, it is an incorrect analogy not a false dichotomy. A false dichotomy means an incorrect assertion that you have to choose X or Y in a situation.
The GP compares python-vs-assembler and random forests-vs-linear-regression but the analogy breaks because python produces assembler and increases the programmer's general certainty concerning what they are doing. Random forests don't make their user more certain of the results as an application.
Basically, Python is a relatively "unleaky" abstraction whereas complex AI algorithms a very "leaky" abstractions.
Simple regression has a lot going for it beyond simply being a simpler model. For example, it produces models that are easy to interpret. That is an enormous advantage if you're looking to use data science to help drive strategic decisionmaking.
My personal suspicion is, in a market full of people who are using ever more sophisticated algorithms to ratchet up their customer conversion classifiers' F1 scores by .001 per iteration, the leader will be the company who's decided to steer clear of that quagmire and spend their time and money on identifying new business opportunities instead.
I think calling linear regression simple is misleading. While the algorithm is simple, interpreting the results and not falling into one of the many traps is quite difficult!
Mine was also. I am just pointing out that even with a simple algorithm like linear regression you can run into all kind of issues and it's not always obvious. Sometimes a slightly more complex method is easier to get right in the long run. It depends on many factors.
You might be being really subtle about python there ! I realize that one can create lines of code like :_ = (
255,
lambda
V ,B,c
:c and Y(VV+B,B, c
-1)if(abs(V)<6)else
( 2+c-4abs(V)-0.4)/i
) ;v, x=1500,1000;C=range(vx
);import struct;P=struct.pack;M,\
j ='<QIIHHHH',open('M.bmp','wb').write
for X in j('BM'+P(M,vx3+26,26,12,v,x,1,24))or C:
i ,Y=_;j(P('BBB',(lambda T:(T80+T9
i-950T 99,T70-880T18+701
T 9 ,Ti(1-T452)))(sum(
[ Y(0,(A%3/3.+X%v+(X/v+
A/3/3.-x/2)/1j)2.5
/x -2.7,i)*2 for \
A in C
[:9]])
/9)
) )
(stolen from : http://preshing.com/20110926/high-resolution-mandelbrot-in-o...)
which could be seen as retrograde vs assembler (but not really for the very funny and brilliant code above - you have to see in formatted nicely and run it to realize that there are some great people out on the web!) perhaps in fact I would agree with this dig - some people do write horrid bits in their python code and python seems to facilitate (or enable) this behavior rather more than other modern languages like Julia. But taking your comment more at face value, reading it to say that more complex methods represent an evolution and that they should be accessed by users as they are easier or better I would disagree. It is easy to screw things up with a random forest or a booster in the sense of overfitting, focusing on the method and not the features and not understanding what the model extracted is telling you about the data. Often a regression model or a decision tree can reveal that there are a few simple things going on which say more about how a process or system has been implemented than the generating domain that that process or system is operating in. This can be gold dust. So, I think that they can be easier to use and simpler to understand, of course when they don't do the job better model generators are required.
AGREE (but it's irrelevant to (1), and also a pretty old conjecture.)
5) Subpoint: It's not true that 47 percent of total US employment is at risk .. to computerisation .. perhaps over the next decade or two.
PROBABLY (that this number/timeframe is optimistic means very little. one decade after the Internet many people said it hadn't upended industry as predicted. whether it took 10, 20, or 30 years, the important fact is that the revolution happened.)
It would be interesting to know if those who are agree in the comments agree with the sensational headline or point 1, or the more obvious and less consequential points 2-5.
I think a lot of people are young and don't realize what it was like working 20 years ago. When I started working in 1994, most people didn't even have computers on their desk let alone an email address. The web was brand new (we used things like gopher and archie), and only a handful of people had ever used or seen the internet.
Sure, I suppose it's possible that advances we've seen in AI won't be translate into huge productivity gains, but I would think that extremely unlikely.
another point is that Linear Regression IS Machine/statistical Learning. Sure its been around for more than 100 years before computation, but regression algorithms are learning algorithms.
Arguing for more linear regression to solve a firms problems, is equivalent to arguing for machine learning. Now, if instead he wanted to argue that the vast majority of a businesses prediction problems can be solved by simple algorithms, that is most likely true. but economic impact of this is still a part of the economic impact of machine learning.
If we're classing linear regression as machine learning and agreeing it's a representative example of the type of simple algorithm that's most likely to benefit firms, I think it probably helps his point rather than harming it. It's a technique that's been around for ages, is far from arcane knowledge and every business has had the computing capability to run useful linear regressions on various not-particularly-huge datasets in a user-friendly GUI app for at least a couple of decades now.
For the most part they haven't run those regressions at all, and where they have, they haven't been awe-inspiringly successful in their predictions, never mind so successful the models are supplanting the research of their knowledge-workers.
This overshoots the target. It's like saying that we use algebra and therefore =/= AI.
LR and general regression schemes are captured in supervised learning methods. So yes, the systems use linear regression as a fundamental attribute but build on them significantly.
After a good look behind the curtain of Deep Learning, I've come to agree with Robin. No, Deep Learning will not fail. But it will fail to live up to its promise to revolutionize AI, and it won't replace statistics or GOFAI in many tasks that require intelligence.
Yes, DL has proven itself to perform (most?) gradient-based tasks better than any other algorithm. It maximizes the value in large data, minimizing error brilliantly. But ask it to address a single feature not present in the zillion images in ImageNet, and it's lost. (E.g. Where is the person in the image looking? To the left? The right? No DN using labels from ImageNet could say.) This is classic AI brittleness.
With all the hoolpa surrounding DL's successes at single task challenges (mostly on images), we've failed to notice that nothing has really changed in AI. The info available from raw data remains as thin as ever. I think soon we'll all see that even ginormous quantities of thinly labeled supervised data can take your AI agent only so far -- a truly useful AI agent will need info that isn't present in all the labeled images on the planet. In the end the agent still needs a rich internal model of the world that it can further enrich with curated data (teaching) to master each new task or transfer the skill to a related domain. And to do that, it needs the ability to infer cause and effect, and explore possible worlds. Without that, any big-data-trained AI will always remain a one trick pony.
Alas, Deep Learning (alone) can't fill that void. The relevant information and inferential capability needed to apply it to solve new problems and variations on them -- these skills just aren't present in the nets or the big data available to train them to high levels of broad competence. To create a mind capable of performing multiple diverse tasks, like the kinds a robot needs in order to repair a broken toaster, I think we'll all soon realize that DL has not replaced GOFAI at all. A truly useful intelligent agent still must learn hierarchies of concepts and use logic, if it's to do more than play board games.
> Good CS expert says: Most firms that think they want advanced AI/ML really just need linear regression on cleaned-up data.
Cleaning up data is very expensive. And without that, the analysis is good for nothing. AI helps provide good analysis without having to cleaning up data manually. I don't see how that is going away.
> AI helps provide good analysis without having to cleaning up data manually.
I don't even know where to start. I suppose you don't really think that what separates ML/AI (whatever that means) from your standard OLS regression is that denoising is not done manually?
> Cleaning up data is very expensive. And without that, the analysis is good for nothing.
I hope you are not saying here that linear regression cannot handle noise.
In the end, as this blog post also points out, ML/AI is just a vague blanket term. What you want is a statistical method that captures the signal as fast and efficiently as possible and often the gain from going beyond simple linear models might be marginal.
AI techniques can help with cleanup too. I've been using entity recognition to find speaker names to use in faceted search of lectures for https://www.findlectures.com. The NER library just finds low hanging fruit that I check manually.
When you say "clean data", what exactly do you mean? I've often seen this claim that cleaning data takes a lot of time, but it seems like an ill-defined term.
For user behavior: bots, clickfraud/clickjacking, bored teenagers, competitors who are sussing out your product, people who got confused by your user interface, users who have Javascript disabled and so never trigger your clicktracking, users who are on really old browsers who don't have Javascript to begin with.
And then there's bugs in your data pipeline: browser (particularly IE) bugs, logging bugs, didn't understand your distributed databases's conflict resolution policy bugs, failed attempts at cleaning all the previous categories, incorrect assumptions about the "shape" of your data, self-DOS attacks (no joke - Google almost brought down itself by having an img with an empty src tag, which forces the browser to make a duplicate request on every page) which result in extra duplicate requests, incorrectly filtering requests so you count /favicon.ico as a pageview, etc.
I agree that using AI to get something that sort-of works out of dirty data is one of the appeals and is going to get people using it.
If this works at all, the problem is that what you wind-up with is a system that's been heuristically taught to clean-up data for a single snap-shot of your data - and the teaching is expensive and requires experts who are going to move on.
Less expensive than clean your data but still a cost.
So when you wind-up with a different pattern of dirty data after a year's time, the system winds crappier than previously and no one will be able to agree how to fix it.
Eventually AI is going to get an evil reputation and that may kill its appeal.
There is a never ending confusion caused by the term "AI" to begin with. Term coined by John McCarthy to raise money in the 60's is really good at driving imagination, yet at the same time causes hype and over-expecations.
This field is notorious for its hype-bust cycles and I don't see any reason why this time would be different. There are obviously applications and advancements no doubt about it, but the question is do those justify the level of excitement, and the answer is probably "no".
When people hear AI they inevitably think "sentient robots". This will likely not happen within the next 2-3 hype cycles and certainly not in this one.
Check out this blog for a hype-free, reasonable evaluation of the current AI:
Thanks. This is why I love HN for finds like this, after poking around a bit this site looks like a really good blog that doesn't hype AI nor deny it which is congruent with my views.
The more I get into machine learning and deep learning it seems like there is an incredible amount of configuration to get some decent results. Cleaning and storing the data takes a long time. And then you need to figure out exactly what you want to predict. If you predict some feature with any sort of error in your process the entire results will be flawed.
There are a few very nice applications of the AI techniques, however most data sets don't fit well with machine learning. What you see is that in tutorials use the Iris data set so much because it breaks into categories very easily. In the real world, most things are in a maybe state rather than yes/no.
> In the real world, most things are in a maybe state rather than yes/no.
Not to get too far afield, but I disagree with this on a certain philosophical level. All states are yes/no. All states of all things should result in a yes/no and be differentiable, with enough data. This doesn't speak to the practicality of that but as far as I can tell the theoretical potential is huge, almost infinite even.
But there should be no ambiguity, with enough data. Maybe that means there will always be ambiguity, but maybe it doesn't, especially not with man-made things and complex natural objects, and also if you can contextualize the data over time and 'geographically', there is more 'signal' there to differentiate
> But there should be no ambiguity, with enough data.
What? I'm sorry but this runs counter to everything in my experience, both professionally, and just casual very day experience.
More data, helps to a point, but then there's diminishing returns, and it certainly doesn't eliminate the ambiguity. On the contrary, you discover diversity, and you still have a misclassification and perhaps even a harder data cleaning problem, because now you're seeing cases that aren't actually clear cut. Even if you're only talking about adding more features, well again, that works up to a point, but then you hit sparsity issues.
Yeah that's why I called it philosophical, because the idea is a little more involved, shall we say. I'm not a god of this, so speculation ahead bewarned. In cases that aren't clear cut, you would also need contextual data like bigger actual physical area, or over time dimension, really any data point that can help narrow down what the thing is. It wouldn't just be pure deep learning stuff, it would be some kind of memory and data store of already classified objects and contexts. In the ultimate end, ALL of it would be sparse, but classify perfectly just that one thing it is built to classify. And if that doesn't work, several sparse things combined would result in one unique thing. On the sparsity matrix wikipedia page there is an example of balls with a string through them, this would correspond to the data being the balls and the systems we build (or alternatively unsupervised learning methods for finding new strings), whatever they may be, would be the strings (assuming all the strings are actual informational and correct to natural world). But you need the balls to begin with etc. Since all of this information should be in the natural world by its own, and also accessible to us
This is literally a philosophical problem. It's called ontology. And no amount of data solves this problem, because ultimately it's a labeling problem, and the border between things is ill defined, and additional data doesn't help resolve labeling ambiguity, if anything it finds out just how ill defined the world actually is.
Think about it. Let's say you had a problem which was find the black squares. So you collect some data and you find that you have a whole bunch of squares that are on the blackness scale of 0.0, and bunch that are 0.1, and then there's one at 0.5. Is 0.5 black? Maybe not. What about 0.7? Maybe. What about 0.999? Probably, but is it? It's not 1.0. And if we say 0.9 and higher are black, why not 0.89? Even discounting measurement error, there's nothing that supports a threshold at 0.9 beyond, "Well, I think it should be this."
> if anything it finds out just how ill defined the world actually is.
Yeah I hear this but it seems only half-true to me. While for most intents and purposes the world is ill-defined, in another sense the world itself is "100% signal" and no noise. If we "zoom out" and take a grand view, imagining that we have a supercomputer and a huge database, and the algorithms are solved, I think every 'thing' in the universe has some unique features, and if you start to have them all in a database you may be able to uniquely identify any thing, at least those important to us. Everything one has excludes something else, but it also includes that specific thing. Every thing adds context to one thing and removes context from another. If you can draw a map of it, it seems to me like deep learning can, hypothetically, automatically differentiate it. Deep learning isn't just about one vector or one hierarchy of features, it's about how the world is ALL vectors like this, even if right now, the CS around it is pretty limited. It seems to me intuitively true at least. At the bare minimum, seeing as us humans are absurd about categorizing everything into objects, and it actually works very well functionally (we can manipulate, create and predict in the world)
If I understand your point, I'd suggest that it may apply best to the use of DL for low level AI -- seeing, hearing/generating speech, and recognizing/ navigating/ interpreting complex signals of other kinds. There classification is secondary to modeling the many subsymbolic facets endemic to raw analog signals.
I suspect DL will eventually settle into a less vaunted role in the historical saga of AI than it portends now. And that role may well be the 'grounding' of sensory experience -- the modeling of the world into something perceptually and cognitively manageable, like Plato's shadows on a cave wall.
What I mean is the Iris data set splits into 3 categories easily.
Right now, they are saying AI self driving cars can get their predictions right 95%+ of the time. However, the cases where they cannot classify the object is the problem. Those are the "maybe" cases I was referring to where they algorithm simply cannot classify the object no matter how much it has seen.
Nah. There is a difference in my mind between creating a 'map' of the natural world, and deciding which actions to perform in that world, and this is all theoretical/philosophical. If one were to do a pure AI type programming language it would have to come from the AI itself, and it would need some reason to create that language and who knows where/what all those rules would be and why (think of human programmers who make new languages).
Your average data-set does not fit into 3 nice categories like the Iris data-set does. For example, with the Iris data set if you know the sepal length and the petal length, you can say with near certainty which type of flower it is. Even trying to classify other objects in nature is much harder than this dataset.
Now let's take sentiment analysis which tries to determine if some words are positive or negative. If someone said: 'that new machine learning algorithm is so sick'. The algorithm has no way of knowing that sick may be slang to mean good, because the system looks up 'sick' and finds that is a negative word. Sentiment analysis has no way of defining sarcasm or other natural language terms.
> Sentiment analysis has no way of defining sarcasm or other natural language terms.
Of course it can. If humans are capable of detecting a given inflection, computers absolutely can as well (given enough data).
Any sentiment analysis algorithm which classifies "that new machine learning algorithm is so sick" as negative is not worth an ounce of consideration. Compared to other problems, that is absolutely trivial to classify, especially since you're typically training off data sets which already include such vernacular.
Here's why the pipes metaphor is a bad one: we already are doing everything we can and ever will do with pipes. Pipes have been around for a really long time, we know what they are capable of, we've explored all of their uses.
OTOH, the current progress in AI has enabled us to do things we couldn't do before and is pointing towards totally new applications. It's not about making existing functionality cheaper, or incrementally improving results in existing areas, it's about doing things that have been heretofore impossible.
I agree that deep nets are overkill for lots of data analysis problems, but the AI boom is not about existing data analysis problems.
> we already are doing everything we can and ever will do with pipes.
I'm pretty sure industrial engineers would disagree here. The romans certainly didn't clean their lead pipes with plasma and neither did they coat them with some fancy nano materials to reduce stickyness.
If there is a curse of our industry, it is almost willful ignorance of just how hard the physical engineering fields are.
The simple things with pipes are simple. Yes. However, to think we haven't made advances, or have no more to make, is borderline insulting to mechanical engineers and plumbers.
Ironically, deep learning will likely help lead to some of those advances.
> However, to think we haven't made advances, or have no more to make
Not what I was saying at all. My point was that pipes are used to transport something from point A to point B, and that regardless of what advances we make, they are still going to be used for that purpose, and that this is unlike the situation with AI.
My apologies for twisting your point, then. I confess I do not think I see it, still. :(
Pipes do much more than just transport from a to b. Though, often it is all a part of that. Consider how the pipes of your toilette work. Sure, ultimately it is to get waste out of your house. Not as simple as just a pipe from a to b, though. You likely have a c, which is a water tank to provide help. And there are traps to keep air from sewage getting back in.
Basically, the details add up quick. And the inner plumbing for such a simple task are quite complicated and beyond simple pipes.
So, bringing it back to this. Linear algorithms are actually quite complicated. So are concerns with moving all of the related data. And that is before you get to things that are frankly not interpretable. Like most deep networks.
> I agree that deep nets are overkill for lots of data analysis problems, but the AI boom is not about existing data analysis problems.
This hits the nail on the head for me. The author's observations in the first 9/10 of the article could all be perfectly valid, but the conclusions he draws I. The last two don't follow for exactly this reason.
The interaction of a system of many small pipes to accomplish computation is an active area of research, and new improvements are used in devices pretty routinely. (Really, the behavior of networks of pipes in general is still pretty open, if you want instantaneous details rather than statistical averages.)
Along similar lines, HFLP systems and systems that require laminar flow to be effective are both more recent techniques that come out of a better understanding and engineering of pipes. HFLP upgrades are a current engineering change over very recent and modern high-pressure systems.
Another way the analogy breaks down: Let's think about cars/trucks. Let's say in the 1930's , they're only availble to big business. What do we get ? A few big companies(Walmart/Sears/etc) doing all retail, at significantly lower prices. A big change.
It seems a little odd that the author is focusing on machine learning not being terribly good for prediction from data to counter the "this time is different" argument. The reason this time is different is we are in a period when AI is surpassing human intelligence field by field and that only happens once in the history of the planet. AI is better at chess and go for example, is slowly getting there in driving and will probably surpass general thinking at some point in the future though there's a big question mark as to when.
>The reason this time is different is we are in a period when AI is surpassing human intelligence field by field and that only happens once in the history of the planet.
Journalists and investors only seem to get excited about buzzwords - Maybe that's because they don't actually understand technology.
To say that technology is like an iceberg is a major understatement.
The buzzwords which tech journalists, tech investors and even tech recruiters use to make decisions are shallow and meaningless.
I spoke to a tech recruiter before and he told me that the way recruiters qualify resumes is just by looking for keywords, buzzwords and company names; they don't actually understand what most of the terms mean. This approach is probably good enough for a lot of cases, but it means that you're probably going to miss out on really awesome candidates (who don't use these buzzwords to describe themselves).
The same rule applies to investors. By only evaluating things based on buzzwords; you might miss out on great contenders.
I've read every comment in this thread and its filled mostly with peoples self congratulatory intellectual views. Nobody, not even Robin Hansen himself has given a good, detailed argument as to why the current progress in Machine learning will stop.
I doubt you'll get that, because nobody thinks that progress in machine learning will stop.
An AI winter doesn't mean that progress stops. It means that businesses and the general public become disillusioned by AI's or ML's failure to live up to the popular hype, and stop throwing so much money at it. The hype then dies down. Research continues, though, until enough progress is made that machine learning starts to produce results that excite the public again, and the cycle goes into another hype phase.
> I doubt you'll get that, because nobody thinks that progress in machine learning will stop.
Robin Hansen is notoriously skeptical about the possibility that Deep Learning can make real gains. He for some reason thinks brain emulation is more likely to make large progress in AI.
>An AI winter doesn't mean that progress stops.
It doesn't completely stop, but progress would be at a snails pace.
> The hype then dies down. Research continues, though, until enough progress is made that machine learning starts to produce results that excite the public again, and the cycle goes into another hype phase.
I think we as a community may need to take a good long look at the hype cycle theory and be skeptical it has any merit.
I don't know that I'm positing any sort of formal theory, just describing a pattern that's been going on for decades now. I'd suggest that unless something fundamental changes, there's a reasonable chance that it will keep on going that way.
Progress in machine learning has never been at a snails pace. What has happened at times is that the futurists of the world stop making breathless pronouncements about machine learning, and that creates a public perception that things are going slowly.
Case in point: Deep learning really isn't anything revolutionary or new. I first started noticing papers about stuff that now falls under the "deep learning" catchphrase about 15 years ago. Not because it started then, but because that's when I started reading that sort of thing. During that time, there's been relatively constant, steady progress being made. But you wouldn't know it unless you had been following the literature, which not many people do. And all of this happened at a time when everyone was convinced that neural nets were dead and support vector machines were the way of the future.
As far as Robin Hansen's skepticism about deep learning getting us to true artificial intelligence, meh. A technology doesn't need to make Ray Kurzweil's eyes roll back in his head for it to be useful.
Brain emulation? I didn't realize people seriously thought that could be good for anything other than research and investigation.
If we can successfully emulate the brain, it seems we would have necessarily acquired the knowledge needed to build models that are very powerful without having to exactly mimic the brain.
Sorry but I don't think "AI winter" as a term
is appropriate here.
The "AI winter" as I know it was around 1988.
I've still vintage issues of "AI Expert" and other publications of the time (which I kept for their great linocut-style artwork, and because they were expensive items here around).
Back then it wasn't so much about ML (aka "Deep Learning") but anything Prolog, expert systems, artificial neural nets and their generalizations, and Lisp.
Why, as presumably rational agents, is there so much hype with certain technologies, even though past history should have taught us otherwise? Is it all for the VC? Is it the media just needing to sell stories? Usually there are smart, knowledgeable people involved in the hyping. They should know better.
There is no such thing as complete rational people, every person goes through bouts of rationality and irrationality, some are just more often rational than others. With that said nobody is immune from irrational emotion sometimes, for example have you been angry from something that wasn't worth it?
I think the core argument against is the same as it was in 1990 (Dreyfus, Heidegger) - basically AI can solve problems in micro-worlds (chess then, Go now; blocks worlds then image annotation now), but it's not clear these micro-worlds can be fused together, unless they are embodied in an entity who lives in the world. The current deep learning & robotics work is exciting and promising, but it's still a long way from an embodied general intelligence. So - progress might hit the same wall - we can build better subsystems , but not the whole system
Fine - let's go for a walk together, build a fire, cook and chat and see what we have in common - It would be interesting. This is why TV and film like Humans and Bladerunner and I, Robot and Arrival and books like Ancillary Justice are fun - how much would we understand each other? how would it be different from interacting with a human?
I am a human. I think it's ridiculous to think I was a special entity so I won't. But I think we reached the point where humans became machines, and machines became humans. The first one to achieve superintelligence would guide the world. I think humans won the debate versus AIs on Twitter. A human effectively debated Big Brother and won.
Have you ever played one of those strategy games with a tech tree? Research A and it lets you research B, C, and D; research C and D and it lets you research E; etc?
That's based on the way discoveries in the real world build on eachother. And in the real world, the research tree seems to be "lumpy".
Think "agricultural revolution", "industrial revolution", etc. Something new comes available, and everyone rushes to pick off all the new low-hanging fruit. Eventually the easiest gains are all taken, and people lose interest and move to other things. And as people keep picking away more slowly at the more difficult/involved things, eventually someone will find something that -- probably combined with some completely different existing knowledge -- opens up another new field. And it repeats.
Right now we're in the "low-hanging fruit" phase of (1) computers that are powerful enough to run neural networks, combined with (2) feedback algorithms that allow networks with lots of layers to learn effectively. Sooner or later the gains will get a bit tougher as we understand the field better, and then research will slow even further as many researchers find something else new and shiny -- and with better returns -- to focus on.
> Right now we're in the "low-hanging fruit" phase of (1) computers that are powerful enough to run neural networks, combined with (2) feedback algorithms that allow networks with lots of layers to learn effectively.
I'm not so sure about that. Places like Deepmind are not satisfied with simply having AI that does straight forward pattern matching problems (Though that's very important). They are moving into more complex problems like transfer learning, reinforcement learning and unsupervised learning for more complex, real world problems solving. They also seem to be making good progress on this as well.
you might enjoy a book called "the structure of scientific revolutions" by thomas s. kuhn. a historian of science who basically argued this. that book was the origin of the phrase "paradigm shift"
The progress in AI didn't stop during the past AI winters, it just slowed as funding dried up when people realized the AI at the time couldn't possibly live up to the hype.
Those are just the two I had off the top of my head. We apply ML tasks for object/scene classification and they blow away humans. Not only that we're already structuring a GAN for "procedural" 3D model generation - in theory this will decimate the manual 3D reconstruction process.
I started question the credibility of the article when the author mentioned "deep machine learning". Not an expert in ML, but it should be "deep learning" referring to a type of neural network based machine learning technique with deep hidden layers.
A little off topic but I think the VR boom will bust much more sooner than AI.
I can't think of normal people wearing those heavy gears in their normal life. There will be its use cases in specialized applications like education, industry, games but I don't think it will get popular like an iPhone.
AR is still OK since it augments real life but there is a long way before it will become mainstream.
I reckon someone said the same about the first mobile phones when they were attached to a suitcase. Who would want to carry a suitcase around just to make calls. Now, comparing the current VR to the iPhone is a huge leap, the VR technology is still evolving and I think it will surely get lighter.
have you tried it? i own an oculus and every family member i've seen has been shocked and loved it. Obviously the oculus is prohibitively expensive but with the release of Playstation VR i think the mainstream is poised to adopt it. I really believe the next game consoles that come out will simply be vr headsets
I have tried it. I owned a PSVR for a month and tried everything available. It was a pretty incredible experience, more immersive than I expected for sure. Most of my family and friends had a great time with it as well.
Ultimately, however, I resold it after a month. There are too few interesting full games available. Nearly every game is mostly a short trial, and most of the games are also very experimental and uninteresting to me in general. As an example, a full 1/3 of games available were musical demos that seemed to be geared toward folks having fun experiences while presumably smoking weed or otherwise in an altered state.
There was never a reason for me to come back to the system, but I would like to see if there are very imaginative useful practical applications that eventually see light. After surveying the other VR options I'm not convinced anything exists yet.
I love that idea but I think we should be more social in person than electronically. I would better go out with someone on a run or a walk than this but again I would love that for sometime
I think almost everyone would prefer to walk with someone in person in nature. However sometimes the more realistic (or perhaps perceived) alternatives might be stay in, or go on a virtual tour with someone online. This makes it relatively attractive.
Game consoles will/should come with VR headsets. My point of view is daily normal consumption.
I wouldn't wear an Oculus while I am home sitting on my SOFA with my family members and not many people play games especially considering third world countries like India. VR would have its place but not to the likes of smartphones.
I use my computer through my oculus. instead of being in my shitty student apartment. i get to be in a movie theatre or nice apartment with a massive screen and watch videos on it. I find it pretty compelling.
I don't think there is a VR boom. There's just early adopters right now. I'd agree with you if you said that a VR boom will never happen... I think VR will stay niche among a core set of gamers.
I have a hard time understanding why even technical people use the term "AI" today. Its use should be limited to sensational media and cheesy sci-fi. It's roughly equivalent to saying "computery thingamabob". I would call a pocket calculator an AI too. Why not? It carries out certain mental tasks better than our brains do.
I think "AI" does still have meaning, in that the goal of AGI hasn't gone away. Yes, any goal that's tangential to AGI probably shouldn't be called AI. But as long as the constituent tasks needed by AI (vision, learning, speech in/out, etc) continue to improve rapidly, especially due to advances in a single technology like DL, it's inevitable and IMO appropriate that the umbrella moniker used to describe DL and its impacts remains "AI".
* The article is correct and the current singularity (as described by Kurzweil) will hit a plateau. No further progress will be made and we'll have machines that are forever dumber than humans.
* The singularity will continue up until SAI. So help them human race if we shackle it with human ideologies and ignorance.
There is no way to tell. AlphaGo immensely surprised me - from my perspective the singularity is happening, but there is no telling just how far it can go. AlphaGo changed my perspective of Kurzweil from a lunatic to someone who might actually have a point.
Where the line is drawn is "goal-less AI," possibly the most important step toward SAI. Currently, all AI is governed by a goal (be it a goal or a fitness function). The recent development regarding Starcraft and ML is ripe for the picking, either the AI wins or not - a fantastic fitness function. The question is, how would we apply it to something like Skyrim: where mere continuation of existence and prosperity are equally as viable goals (as-per the human race). "Getting food" may become a local minimum that obscures any further progress - resulting in monkey agents in the game (assuming the AI optimizes for the food minimum). In a word, what we are really questioning is: sapience.
I'm a big critic of Bitcoin, yet so far I am still wrong. The same principle might apply here. It's simply too early to tell.
I think you are making a false dichotomy here. It is perfectly possible for the article to be right, and there is still a future with general artificial intelligence and a singularity. If you believe the singularity is inevitable then you should read the article as saying that we are woefully misjudging where the asymptote is -- yes current progress looks impressive, but there are some really big steps that we are currently ignoring, and real human level AI is a century or two out, not a decade or two out. That's perfectly possible, and most philosophers who work in consciousness and philosophy of mind (as well as a very large portion of the machine learning community) will tell you that there is still some big hurdles that we don't even have the faintest idea how to cross (the whitehouse report on AI described it as a "chasm").
It's already happening. The world became hyper efficient. The hackers behind "Trumpbots" made money on the prediction markets. Understanding that mechanic makes me almost certain the singularity arrived after his election.
We're building an applied AI business by creating an experience through both hardware and software. You don't set out to create something with as big a breadth of vision by worrying about booms and busts. You continue your journey unwavering because the potential impact and fruitfulness of development is worth it.
This is why you should work on something you're passionate about. Your time on earth is limited, so strive to leave good work and contribute to the progress of humanity on a larger scale.
I meant that computer science has staying power, while particular branches (or JS frameworks) may rise or fall in popularity over time. Likewise, the "trunk" of data science knowledge is not a mere fad.
I'm not a data science academic or practitioner. My opinion is based on a small amount of tinkering and what I've read in various online sources.
I plan to enter a PhD program in 1-2 years to specialize in ML/Deep Learning. Assuming it'll take 5-6 years to complete my degree how applicable should my skill sets be in industry at that point?
The important thing about the PhD is that you've become an expert in conducting experiments and research. It really doesn't matter what ML techniques you've done, as long as you know everything else associated with building those types of systems. Just hone your research and experimentation skills and you'll be fine.
I do have one suggestion: learn to handle dirty data.
I work with ML researchers and notice two things: they're pretty bad software engineers (no knowledge of software patterns, bugs galore), and they almost never know how to clean their data. The latter is because they do a lot of their research using pre-cleaned, standard data sets. You never get that in industry.
you'll be a programmer - that is what counts. How good of a programmer you will be will determine your success. never put your eggs in one basket (not saying you shouldn't become an ML expert though, that's pretty damn nice). as a Phd, you are probably good enough.
as to ML, its adoption is hyped. it is powerful, but not as anyone really talks about.
support vector machines and Bayesian learning have been around since the 70s/80s (ninja edit: SVM's since 1963! Markov Chains 1950s, Bayesian Learning/Pattern recognition sine the 1950's), but adoption has been slow due to the nature of business, which is now drooling over it since neural networks beat a few algorithms.
due to the hype, more business will opt for ML now, but the craze will plateau and ML will become another tool in your arsenal.
so basically, you really have nothing to worry about - use your Phd to do interesting things, come up with novel and new research and/or develop your own product.
don't let your job security worries get in the way of enjoying what you want to do now, you're already good and in STEM (and if you don't feel good enough, work on yourself until you do).
> support vector machines and Bayesian learning have been around since the 70s/80s (ninja edit: SVM's since 1963! Markov Chains 1950s, Bayesian Learning/Pattern recognition sine the 1950's), but adoption has been slow due to the nature of business, which is now drooling over it since neural networks beat a few algorithms.
This is one of the things I find hardest about convincing managers and leads of. They think things like CRFs and Markov models are "new" methods and too risky. So they opt for explicit rule-based systems that use old search methods (e.g. A*, grid search), which hog tons of memory and processor. Those methods rarely ever work on interesting problems of the modern day.
They can understand the rule-based methods easily. They have a hard time leaping to "the problem is just a set of equations mapping inputs to outputs, and the mapping is found by an optimization method."
I explain it using the infinitesimal method, which if done right using the hill climbing metaphor, often delivers. But it does take away the magic of "wooo, neural" :p
I don't actually think that's true, if the "AI bubble" bursts at some point in the near future, the people who'll be in trouble with be those without formal education to back them up.
> Almost every good place pays their CS PhD students enough to squeak by without getting into debt. I wish this was more widely known.
Right. The debt problems that people have after PhDs are more often due to their undergrad loans sitting around for 4-7 years while they were earning enough to subsist and not more.
A PhD is, by it's nature, rather self structured. How much value will be added to your skills over that time will depend in no small part to how you spend it.
If you are aiming at the industry, I would say getting a research-oriented master's degree is much more cost-effective. Besides you can always learn the skills you need in your work.
If your plan ahead of time is to go into industry then a PhD is not a really great plan unless you will get a lot of personal satisfaction out of research. PhDs are incredibly inefficient from a professional development for industry perspective.
Robotics and automation have been improving for a long time, and especially recently. Look at the rise in consumer drones, enabled by improvements in batteries, sensors, and computers.
But the main thing holding them back is a lack of AI. Robots can do a rote action over and over again, but they have a hard time identifying where objects are, planning, and reacting to their environment. Just solving machine vision would be a massive step forward and enable a ton of applications.
And that has sort of already happened. The best nets are already exceeding humans at vision tasks. They are learning to play video games at expert level, which is not conceptually distant from robot control. Its taking time to move this research out of the lab and into real applications, but it is happening.
And so I totally believe that at least 50% of current jobs could be automated in 10 to 15 years. How many people are employed doing relatively simple, repetitive tasks, over and over again? Me and most people I know have jobs like that.
I think what we're seeing is an explosion of new approaches to computerized problem solving made possible by huge amounts of data and enormous computing resources. A lot of what has become possible in the last couple of decades is indeed new, but the apparent rapid advance is really just a matter of applying the brute force of a massively upgraded ability to process huge quantities of data in parallel, and this has led us to make erroneous assumptions about future progress in these areas.
Basically, these are new solutions to new problems, and we're rapidly seeing the easy 80% of this new generation of "AI" happen and it seems magical. But soon enough we'll hit the wall where further progress becomes harder and harder and brute force approaches are no longer sufficient to achieve interesting results.
Pretty much every large firm has multiple problems ML can solve better than linear/logistic regression. Smaller firms may still have one or two. In some industries the core competency will be how good your ML model is as everything else becomes a commodity. There are new advents that make ML better for small data-sets as well as opportunities for data-brokerage to increase access to data. And these are just current applications, new applications are still nascent (i.e self driving cars). Treating ML as a software problem instead of a science project - with a pipeline of adding/creating data, cleaning, modeling, analyzing, learning, and iterating is also incredibly important but it's not like most companies are doing this particularly well either.
Thanks for the study - Figure 1 is impressively self-explanatory. It's interesting to note that the article's author Robin Hanson[1] worked in AI & ML industry roles starting at the peak of the last AI boom in '84 and witnessed the AI winter through to '93.
"Ai secretary" in what respect? Natural language processing? Because simply making an entry into a calendar is just a 1980s' level computer programming problem at best, so I think your example needs some clarification/elaboration. Whether something is "AI" also depends on the presence of learning (the stuff you do with Windows voice recognition to set it up is not "learning/teaching" in the AI sense). Most of the things touted as "I" (intelligence) have all the intelligence in the programming effort, but none in the actual program (apart from what was put in).
I think the boom in AI jobs will go bust, when tools for data cleaning and injection, and for automatically building models will get so good that experts will no longer be required to use them. I have so frequently set up customers with procedures and code for ML, that I have seriously thought of writing a system to replace people like myself.
Until there is real AGI however, there will be jobs for high end AI researchers and developers.
I'm very curious to what degree there even is an AI boom right now, vs. AI and machine learning going through a phase as the buzzwords du jour used in corporate PR. People have doing all sorts of fascinating things with machine learning for decades, and (for example) Google has been arguably an AI-focused company from day one.
In the tech press recently, I keep hearing how every huge tech company needs to have some sort of AI strategy going forward, so they don't miss out on an industrywide windfall, or even become irrelevant because they didn't hop on the AI bandwagon.
I suspect that there are a few more people working in AI nowadays than we're doing so 10 years ago, but that quite a bit of the narrative surrounding AI in the press is some combination of corporate marketing and journalists eager to have something to write about.
I'm not saying AI isn't important, rather that it's an important field that's only a little more important than it already was 10 years ago. The difference seems to be how often it pops up in PR and tech journalism vs. 10 years ago. Just a theory of course; I would love to know what the reality is.
I work at a large pharma, and from what I've seen in this business and anywhere else where data is important (like R&D), deep learning is perceived to be revolutionary.
I think most of the rise in DL's corporate mindshare is the potential it has for disruption of the status quo. Business folks hate game changing tech. It destabilizes the priorities and practices they understand, and dishevels the grand hierarchy of corporate command and control they know and love.
If DL really does reinvent any significant fraction of their part of the business, they know 1) they won't know what to do in response, and 2) their daily competition is going to get even more challenging until a new status quo emerges. And both of which may cause them to lose their tenuous position/status in the company hierarchy, much less, lose their very job.
>> Good CS expert says: Most firms that think they want advanced AI/ML really just need linear regression on cleaned-up data
Not nearly true. The simple counter-argument is that prior to DL, we don't have good approach to really 'clean' data like images.
The author states this fact as if cleaning data is a piece of cake. No, it is surely not. In fact, part of the DL's magic trick is the ability to automatically learn to generalize useful features from data. From another perspective, the whole DL frontend, prior the very last layer, can be viewed as a data cleaning pipeline, which is learnt during the training process, optimized to pick the useful signals.
The author clearly isn't an expert on the matters he trying to put claims on. Yet his statement comes with such big confidence or ignorance. This shows why this revolution will be a truly impactful one, for even some of the claimed intellectuals cannot understand its importance and divergence of its predecessors. They will be caught off-guard then left behind. It would be very enjoyable to watch what their reaction would be once it happens.
The immense availability of financial instruments and VC makes investment in any mildly promising technology overshoot. There's nothing really mysterious about this, especially after the dot com bubble. Is the web useless? no, but there's a limit to the number of players doing the same things the same way successfully - because that's what hype does, make people focus in not just a technology but a particular way.
3d-printing, mobile apps, tablet devices, VR, cryptocurrency, 3D TV, Neural Networks (in the 70s, then again in the mid 80s, then in the late 90s, and now), you name it.
These are perfectly applicable technologies that may or may not warrant the swings in investment they attract, but they are valid and defensible nonetheless. They may also be monetisable - not always useful stuff brings proportional profit in the market. And of course, there's also timing. The exact same idea may work years later in a different environment. But the only way to find out is to try, and maybe try too much at some point.
It seems the nature of this and VR that they come to boom for a while and then bust, having stagnated. They then wait for the next alignment of underpinning technology, knowledge and culture to emerge again. Last last one I'm aware of was mid-late 1990s, where VRML was gaining traction and new ways of thinking about AI were emerging.
IBM Watson or similar (if I recall IBM was still calling their business AI system Watson back then) seems to be promenant in these two booms and both times the results it gives haven't matches it's marketing hype.
The technology, having been significantly furthered fades into the day to day of computing somewhat until the next boom that drives more short burst innovation and awareness.
Conscious AI and realistic VR is some way off, if we ever see it. Culturally and ethically we are not ready to answer the questions it poses and the cyclical nature gives us more time to digest the latest raft of questions in light of the progress.
While the article is right that in many situations a linear regression or tree based approach will be more effective, it downplays the real value add of deep learning which has been clearly demonstrated with image data and is likely to have significant impact in other areas (audio, biodata, etc.) where traditional statistical methods have failed.
In a couple of years you will be able to take a picture of a rash in your arm and get exact diagnosis with treatment instructions. You will be able to take a pic of a flower, a leaf or any tree and get accurate info about its species, plagues and best techniques for growing them. You will be able to take a pic of any insect, spider, snake, animal, or anything that moves, any mineral, element or anything at all and get accurate info about that.
AI is not only about robots thinking, it is about collecting information and making it available on demand. Agriculture and health will be the first beneficiaries, finances in a close second.
Data mining is where the first stage of AI is, and Google is moving ahead of everybody else with their search engine, maps, translation, and all the information collecting tools. Once you have enough data, knowledge is just a couple of programs away.
That is a valid use of deep learning. I think the article is talking about people thinking that because deep learning is very impressive for some things, they should use it for everything - even problems where a much simpler solution works fine, or problems where deep learning doesn't apply.
I work for a consumer product company, and there are often people talking about using 'big data' and 'machine learning'. They're just following the hype; they don't really know what machine learning is and I've only heard two or three potential applications of it mentioned that make any kind of sense.
So, imagine that in 2011, 5 years ago, you approach some VC and say:
"Hey, we are building this VR hardware and games for it, we would need ~1M to finish it".
I think that there is high chance that you would get some weird looks, and possibly few remarks how that is a "dead technology, tried once, and obviously failed".
And then, fast forward couple of years there is whole industry around VR, jobs, hardware, software, the whole eco system.
You only need one strong player in a field, and suddenly everyone and your neighbour kid is doing it.
What is "new this time" is that computers/machines can see and hear and even speak, whether we call it AI or "Deep Learning" or "Machine learning". So there is going to be more impact from Vision/Voice based applications rather than data analytics/predictions. Eg. Self Driving Cars, Medical Diagnosis, Video Analytics, Autonomous machines/Robotics, Voiced based interactions. All of these combined could be as disruptive as Internet itself.
Thomson Reuter's ET&O and Risk division laid off 2000 people recently to fund a new center near university of Waterloo to provide "answers"(Its their mission statement) by recruiting people to provide deep learning solutions. NY suffered deep cuts. The sad part is it seems the leadership just want to use deep learning as a way to justify doing what they are doing, which is "starting from scratch"
This reminds me of the stem cell research boom and bust of the late 90s and 00s. It turns out that the new and shiny toy doesn't work for everything. However, it's not really a bust, as AI, similar to stem cells applications, will continue to do wonders where it is the best tool for the job.
Hard to say. I'm seeing too many billboards near SF for "big data" and "machine learning". AI-type grinding on your stored business data is sometimes useful, but not always profitable. Everybody big already has good ad targeting technology, after all.
There is no AGI, or strong AI. There is weak or narrow AI. AI simply means that if a human performed the task, it would be considered intelligent. So Deep Blue was chess AI, but it wasn't remotely AGI. Same with Watson and every application of AI to date.
In a sense, Artificial Intelligence is the encoding of human intelligence for some task or problem domain in machines.
Until the taboo on talking about consciousness is broken and we seek to understand what role this incredible phenomenon plays in human cognition, there will be no progress towards the holy grail: true general purpose AI. That is my falsifiable prediction.
We already had the AI winter before. The worst thing about it is the death of Symbolics. The latest AI resurgence, unfortunately, created nothing comparable to the legendary Lisp machines.
Also - must remember that AI is a programmer on the sidelines letting data do the logic. It required a fairly different frame of mind than traditional programming
So data as employed by DL plays the same role that it does in Brooks' subsumption architecture -- grounding and shaping the knowledge model -- but now doing it emergently, albeit requiring a lot of parameter tuning from the human-in-the-loop.
An interesting prospect for the evolution of software developers.
I think a key differentiation between ML and more common statistics is that ML is designed to improve itself based on data. Statistical methods don't do that.
So maybe they're trying to do very similar things in a lot of cases, but self improvement is a major differentiator.
I'm old enough to have seen a lot of these boom/hype/bust cycles. I'm convinced that this time is, in fact, different.
To temper this I believe most decent user-visible changes will take ~5 years (as most actually useful software does) but the changes will be huge:
* The author cites computer driven cars. I think this will take place mostly on long-haul highway trucking instead of in cities first. Even so, this could mean a massive swath of truckers without work in a short 5yr epoch.
* We've already seem the effects of heavy astro-turfing/disingenuous information/etc in the last US election. This certainly changed the "national psyche" and may have changed the election outcome. There is heavy ML research going into making the agglomeration of ads and content almost compulsively watchable. Our monkey brains can likely only handle a few simple dimensions and only boolean or maybe linear relations and they certainly get trapped in local maxima/minima. Even trivial ML techniques can bring this compulsion from say 50% effectiveness to 95%+ (by some reasonable measure). Imagine a web that is so completely tailored to the user such that search results, ads and content is completely tailored to you. Verbs, adjectives entire copy all written to get you to that next click. This is different.
* Bots that seem like real ppl will be rampant. Are those 100 followers/likes/retweets actual ppl? Even years ago reddit (to gain popularity) faked users. Certainly this has only accelerated and will continue to as commercial and state actors see value to moving public opinion with these virtual actors. (ironically maybe only bots will have read this far?)
* Financial Product innovation - Few ppl actually understand this market (even within the banks) however the deals are usually in the 100+ million range. The products take advantage of tax incentives, fx, swaps, interest rates, etc in an ever increasing complexity. These divisions are still some of the most profitable parts of banks. It's likely that on deals where profits are measured in tens of millions on a single deal (several are made per quarter, per major bank). It's likely that ML algos will be put to use here as well not only optimizing current products but in current prod elaborations. I beleive these products to be a major source of inflation. Whereas the official numbers are ~2% I believe the actual inflation (tm) felt by most is more in the 7%+ range.
* State Surveillance and Actions - I hear ppl saying that mass surveillance hasn't been effective in stopping "terrorism", as if it would be ok if it did. Well, it will be effective and it will get very, very good at it. Of course terrorism is not defined anywhere so ...
* Customer Support - this, like transportation, is a major employer of unqualified workers. I believe in 10 years there will be maybe 1% of the current workforce in CSR work. The technology is here the software just has to be written.
It's not just the number of jobs displaced it's the velocity. If we look to the effective Predator-prey modeling:
AI will find it's way into specific areas in which the ROI is very significant, and though those will be hard to predict, I agree with the above.
It will make a huge impact in some areas.
Others, not so much.
It's funny that not a single person mentioned 'big data'.
Big data was all the rage for the last few years, and it would seem like an intuitively obvious fit, not? I don't think so.
I think it's a stretch to see how AI help's the gap design new clothes, or even optimizes sales approach. I suggest it will be things like customer service as Randy indicated.
The first one is Data Science. More and more businesses store their data electronically. Data Scientists aim to analyze this data to derive insights from it. Machine Learning is one of the tools in their tool belt, however often they prefer models that are understandable and not a black box. Sometimes they prefer statistics because it tells you if your insights are significant.
The second one is Machine Learning Engineering. ML Engineers are Software Engineers that use Machine Learning to build products. They might work on spam detection, recommendation engines or news feeds. They care about building products that scale and are reliable. They will run A/B tests to see how metrics are impacted. They might use Deep Learning, but they will weight the pros and cons against other methods.
Then there are AI Researchers. Their goal is to push the boundaries of what computers can do. They might work on letting computers recognize images, understand speech and translate languages. Their method of choice is often Deep Learning because it has unlocked a lot of new applications.
I feel like this post is essentially someone from the first group criticizing the last group, saying their methods are not applicable to him. That is expected.