Hang on. That argument makes no sense. We've had subsumption architecture for a long time now( https://en.wikipedia.org/wiki/Subsumption_architecture ). Subsumption architecture puts some of the intelligence in the lower level systems. While the higher level controls can tell the lower level systems what it wants, it can't do things that the lower system determines is dangerous.
So if a normal Mercedez has a collision avoidance system that automatically brakes, there is no reason why an AI based system can't be built on top of that and the collision avoidance system automatically braking without intervention from the higher AI systems. A subsumption system prevents higher level controls from doing something catastrophic like hitting a pedestrian or, in biology, a person holding their breath until they die.
Very true. Which is why I'm still waiting for what happened to all the lower level safety systems, presumably driven by lighting invariant LIDAR or RADAR systems, in the Uber situation.
The article kind of mentions:
"cyclist detection from 3D point cloud is much harder task than cyclist detection from an image"
The system didn't need to know that it was a "cyclist". There was something there, in its path. Stop.
In fact, the XC90 is already equipped with a radar-based emergency stop system from factory - that alone would have been 100% sufficient to prevent or vastly reduce the impact. But of course it looks like the system was disabled/overriden by the Uber's autonomous solution.
Yep! Every single autonomous system I'm aware of (and that I'm allowed to speak about) has such a system designed in from the ground up. The article presupposes we allow "intelligence" handle all decision making, which seems ridiculous even in biological systems, let alone those we design for a single purpose.
Yes. We had that in our DARPA Grand Challenge vehicle a decade ago. In addition to the main navigation and control system, there was a separate process taking inputs from the radar and speedometer, with the authority to slam on the brakes if the vehicle was about to hit something. Backing that up was a hardware stall timer reset every 100ms. If it hit 120ms without being toggled by the computer watchdog process, the brakes went on and the engine dropped to idle with no computer involvement. Then there was a DARPA-mandated remote engine kill radio system.
> So if a normal Mercedez has a collision avoidance system that automatically brakes, there is no reason why an AI based system can't be built on top of that and the collision avoidance system automatically braking without intervention from the higher AI systems.
In fact this 'bottom up' approach is exactly what all automoted driving efforts in the automotive industry follow. It's the biggest difference between how the auto industry and Silicon Valley are tackling this problem. A lot of digital ink has been spilled over how slow and supposedly prone to disruption by hip and agile software companies this approach is, yet it becomes ever more evident that it was the correct choice.
>A lot of digital ink has been spilled over how slow and supposedly prone to disruption by hip and agile software companies this approach is, yet it becomes ever more evident that it was the correct choice.
The car industry had 130 years to become the conservative, anti-agile, careful industry it is. They are in the business of costing human lives a lot longer.
The problem is "move fast and break things" works really extraordinarily well for software which is for entertainment and information purposes. Not for something serious.
That sort of thing is common in industrial controls. Start the big ass air compressor and if the oil pressure doesn't hit 100 psig in five seconds the interlock trips. That's handled by a controller on the pump itself.
Also really common or required that the interlock be totally separate from the control system.
Agreed. You use the lowest-tech option which is practical at every stage. So if you have a mechanism which can travel between points A and B and will cause damage if it leaves that range, your best option is a physical stop at each end. Second best option is a hard-wired limit switch at each end which cuts motor power. Third best is a limit switch which is used by software. You wouldn't use machine vision or LIDAR.
And then as you say, safety related functions (anything on which human life depends) need to be separate from general control code and to meet the requirements of whatever safety spec you're working to (e.g. SIL).
Yes. For new readers, suggestion for this article: start with the last sentence. Then skim the article re AI. Then respond. So we don't have to keep saying/hearing "Why not both?". The author's take is: both!
I guess author makes the distinction from deterministic Vs non-deterministic algorithms. As a layman in these matters I still don't fully understand how some people classify machine learning algorithms as non-deterministic...
...i started wondering about this when I was reading a few articles about the AlphaZero algorithm that learned to play chess entirely from self play and wondered if it would always play the same moves in response to a fixed set of opponent moves (assuming the opponent starts as white).
My guess was that it wouldn't always respond in exactly the same way in case there's any MCTS like step somewhere in there blended with the Machine learning algorithm.
For a game like chess it would seem to make sense that the overall algorithm would still include a MCTS step (like AlphaGo did) but for an autonomous car it would seem crazy to any human to imagine that there would be any random search for a decision in a tree of possible interpretations of the input for example.
Does any one have any detailed knowledge about this? Would a non-deterministic algorithm ever be allowed in an autonomous car?
> As a layman in these matters I still don't fully understand how some people classify machine learning algorithms as non-deterministic...
Some algorithms start with random numbers for the model and converge towards a better model. After the model is generated the input->output will be deterministic, but since the model generation is non-deterministic the algorithm overall is considered non-deterministic.
Not an expert on self-driving cars, but modern, statistical machine learning systems have to be trained sort of end-to-end. That is, if you want to machine-learn how to perform a task, you have to learn both the low-level and the high-level actions at once. And you can't add to the knowledge of the task, once the system is trained.
That's a limitation of the technology: statistical machine learning models are notoriously non-compositional. That means that basically you can't take one trained model and use it as a feature, to learn a new model for a different task.
Say, if you train a machine learning classifier C1 to recognise class Y1 from features F1,...,Fn, you can't then take the model of Y1 built by C1 and give it to a different classifier, C2, as a feature in a new feature vector Fn+1,...,Fn+k to learn a different class, Y2.
So you have to learn everything you need to know in one go- and that's terribly, awfully difficult.
There is a type of machine learning algorithm whose models are inherently compositional- Inductive Logic Programming (full disclosure: that's my research). These learn logic representations and from logic representations, so once you learn a model you can use it as a new feature and continue learning. Statistical machine learning folks have been trying to do that for a while now, without much success.
Actually they are extremely compassable. A feed forward neural network is often called a multiple layer perceptron, because it is multiple perceptron classifiers that feed into each other.
If you want to build a domain specific image classifier, you take the bottom layers of a generic classifier and train it on your otherwise too small dataset.
You can absolutely add predictions from one classifier as a feature to the next. It is quite common to have a pipeline where you augment text with Part of Speech, Named Entity Recognition, etc, before performing the final task.
What perceptrons feed into the next layer are not features- they're activations. They can't be used as features because they don't represent attributes of some concept.
CNNs invent new features that can be used further down the track- sure, but the end result is not something you can use any further. You can't, for example, take a learned concept of a dog and then use it as a feature to learn about pets.
With language learning pipelines again, you're not adding a newly learned model to a set of features- instead, you use that model to label parts of your data that were not previously labelled.
What I mean (what my research group does) is more like what is discussed by Francois Chollett of Keras, here:
> What perceptrons feed into the next layer are not features- they're activations.
> They can't be used as features because they don't represent attributes of some concept.
I think we have very different definitions on what a feature is. You can use these activations as features in a model.
> With language learning pipelines again, you're not adding a newly learned model to a set of features- instead, you use that model to label parts of your data that were not previously labelled.
I'm afraid I don't understand what you are trying to say. In an NLP pipeline you use the predictions of several models as features for your final model. How is that not composable and modular ?
>> I think we have very different definitions on what a feature is. You can use these activations as features in a model.
Let me backtrack a bit, to where I phrased the issue thusly:
Say, if you train a machine learning classifier C1 to recognise class Y1 from features F1,...,Fn, you can't then take the model of Y1 built by C1 and give it to a different classifier, C2, as a feature in a new feature vector Fn+1,...,Fn+k to learn a different class, Y2.
When you train a statistical machine learning classifier - let's take a simple linear model as an example, for simplicity; what you get in the output is a vector of numbers- parameters to a function. That's your model.
You can't use this vector of numbers as a feature. It is not the value of any one attribute - it's a set of parameters. So you can't just add it to your existing features, because your features are the values of attributes and the model is a set of parameters meant to be combined with those attributes.
What you can do is take your newly trained model and label the instances you have so far with the class labels the model can assign to them. Now, that's a pipeline alright. For instance, if you had a linear model with features "height" and "weight" and learned to label instances with "1" for male and "-1" for female, you could then go through your data, label every instance with a "1" or "-1" and then train again to learn a model of "age". At that point you have a new feature that is not the concept you learned in a previous session, but only a subset of that concept. Now you can try to learn a new concept from this new set of features, but the original concept ("sex") may or may not be part of it. It may turn out that "sex" is not necessary for learning "age" (it's redundant); or, it may be necessary, but in that case you have to learn the concept of "sex" all over again as part of learning "age".
By contrast, the class of algorithms I study, Inductive Logic Programming algorithms, can add the models they learn to their features (features are called "background knowledge" in ILP) and go on learning. For instance, such an algorithm can learn "parent" from examples of "father" and "mother", then "grandfather" from the original examples of "father" and the learned concept "parent" and "grandmother" from "mother" and "parent", then "grandparent" from "grandfather" and "grandmother" etc. Every time the new concept learned can be added to the learning algorithm's store of background knowledge, as it is- you don't need to go through the data and label it. That's because the representation of "data" and "concept" is the same, so you can interchange them at will.
Say, your background knowledge on "father" and "mother" might look like this:
father(Earendil, Tuor)
mother(Earendil, Idril)
From that you can learn "parent" that might look something like this:
When the collision detection system kicks in and determines that you're going to hit something in the next couple of seconds, then braking probably is the best option.
I know of an accident where someone overran a cardboard container used to package washing machines. He thought it was empty. Unfortunately it was not. Some truck lost part of its load.
I think its safe to break always when encountering an ambiguous situation.
Another anecdotal tale: My dad once didn't bother slowing down for "cardboard tubes" that had fallen off a lorry. It turned out they were ceramic tubes, and caused quite a bit of damage.
Integrating AI with a subsumption architecture sounds interesting but could still not work. You would be abstracting a lot of details (as well as potential action) from the training process, so the overall result you get still might not work. Training on everything might actually yield a better result.
Does anyone know how iRobot (Brooks' company) does it?
The AI can still drive the car normally, the only thing that changes is it's prevented from running into another car at full speed. Which is no worse than having drivers override the system forcing a disengagement.
Yes, as long as you add a huge penalty to the training algorithm for triggering any underlying safety system I see no problem in that. (Otherwise I can see a completely mad car that believes everything is safe as it never crashes even when reckless )
I believe it is subsumption based with random walk added on top: bumper hit means turn around and wheel dangling means don't go farther. I think the more recent models have more intelligence added on top but still has the lower level systems acting as safe guards. Of course it doesn't run into the problems you mentioned because it doesn't have any learning (at least not one of the earlier models I had).
As I was skimming the article, I was thinking some sort of system like this could be made, and I was pleasantly surprised to see the approach has a long history. This is one reason I like Hacker News.
So if a normal Mercedez has a collision avoidance system that automatically brakes, there is no reason why an AI based system can't be built on top of that and the collision avoidance system automatically braking without intervention from the higher AI systems. A subsumption system prevents higher level controls from doing something catastrophic like hitting a pedestrian or, in biology, a person holding their breath until they die.