It seems difficult to tell who is the “furthest ahead” because it’s a pretty complex race.
For starters, we can’t even say with certainty who all of the competitors are. Some are very quiet or at early stages but could be working on a breakthrough problem that turns out to be a pivot point.
Some are specializing. Who is going to scale up the best and most cost efficient solid state lidar? There’s a whole list of key problems being worked on.
Do you mean hardware or software? Just the algorithms are a huge piece. Just hardware is a huge pieces.
Beyond price and performance, it’s common that unforeseen factors end up being important in determining who becomes the most successful in selling traditional manufacturers components or whole systems. Surely we don’t realize all of them yet.
Also there is so much still being held close to the vest. These guys are keeping lots of secrets about how far along they are truly, roadblocks, etc.
I don't really trust any of the marketing (including videos) or friendly articles about how far along companies are. The best way to judge I've seen is by the mandatory reports companies testing in California must supply. By those, Waymo is far, far ahead as of 2016. They've driven 635,868 miles (two orders of magnitude more miles than their next closest competitor) and their miles driven per disengagement was 5,128 (an order of magnitude higher than the next closest, with many still in single digit miles per disengagement). I'm eager to see 2017 numbers. Perhaps one of the other major companies has caught up but I'm doubtful. That's a wide gap to close in a year.
Sure, the numbers could be gamed a little (just drive on the same road you can do perfectly every time every day all year) but doing that will never let you improve your real world performance (important for the big companies heavily invested in this, not so much for the smaller companies looking to get acquired).
I think it's reasonable to just ignore all the small companies working on this in stealth. It's all about testing for self driving cars. They can't perfect it just sitting in a garage and thinking really hard about what problems they may encounter and maybe a few prototype vehicles. They need as many cars as they can on the road driving and collecting data on real world situations.
Isn't Uber testing in Pittsburgh and Phoenix? They wouldn't report those miles to California.
What about R&D outside of the US?
I don't think California reports are an adequate measure of progress. Additionally, I am not sure mileage matters as much -- you even suggest why in the next paragraph.
>They've driven 635,868 miles (two orders of magnitude more miles than their next closest competitor) and their miles driven per disengagement was 5,128 (an order of magnitude higher than the next closest, with many still in single digit miles per disengagement).
Making the assumption that "disengagements == bad" is a bit suspect though. We have no way of knowing if the reason for disengagement among competitors is purposeful, or due to more strenuous testing environments. It's not like this is a performance metric we know they are trying to optimize. Without knowing the testing methodology, it's meaningless to compare these numbers.
Who is going to scale up the best and most cost efficient solid state lidar?
Probably some big auto parts company. Continental [1] and Denso [2] are both building solid state LIDARs. Quanergy seems to be all hype.
Maybe automatic driving is just going to be a set of components Tier I auto parts makers sell to auto companies. That's how ABS braking and stability control works. Those are the companies that can make electronics work reliably in the automotive environment.
It may not be a race, either. Many major auto manufacturers expect to ship some form of automated driving in the 2020-2021 model year. There's no reason to expect one big winner here.
As for "transportation as a service", Avis, the car rental company, is getting ready to do that. They're servicing Google's self-driving test fleet in Austin, so they'll be ready to go into that business when the time comes.
They started early and seriously, unlike manufacturers who embraced it with marketing in mind and then put "beta" products on the road. Google were the first to gather massive amount of data on SDV.
Maybe they're not the first right now, but they're safely in the lead, waiting for the others to fumble.
I work for one such "early stage" startup - all we do is bullshit VCs by showing them research done by other people, and then passing them off as our own.
Indeed, the only comparison we have for Google's progress in most cases is what they claim. The few times other parties have stepped in and spoken up, there's been some less rosy truths in there.
It's in all of these companies' best interests to look like they're in the lead.
The tech we can judge the progress of is the tech we can actually test and use.
Consider that at the time, an engineer told him that the car was dependent on a level of mapping detail that was impractical at a nationwide level, that the car would run a red light if it wasn't on the car's map.
The article also cites that the cars would have issues in bright sunlight, particularly determining the color of stoplights, that they couldn't handle construction sites consistently, etc.
Also, consider the Slate article mentions Google Self-Driving Cars don't know how to park, two full years after the PR stunt which shows a blind man doing so in a Self-Driving Car. (Which was almost certainly a heavily staged operation.)
It's safe to assume Google has improved upon some or probably all of these issues in the last couple of years, but I think that article shows a stunning difference between where a press team says a technology is and where it actually is.
The point of not having a steering wheel was to experiment with a platform where it is assume disengagements where user intervention is needed in an emergency won't happen, because Google's own research indicates that relying on disengagements is dangerous.
Level 2-3 cars are frankly, too dangerous for the road IMHO, and the rush by automakers to ship these out to the public is, I predict, going to lead to some serious disasters, class action lawsuits, and regulation, that imperils the whole enterprise.
If there's anything that you don't ship "until it's ready", it's a device that's lethal to human life when something goes wrong. Tesla already killed someone, and while you could argue he violated the beta test agreement, shipping cars which requirement drivers to be attentive and keep their hands on the wheel while self driving -- ready to be aware and take control at a moment's notice -- effectively are engaging in bullshit marketing, because we all know that people who use self driving don't want to sit there "engaged" but want to be free to do somethings like check their phone.
Public data on disengagements shows Waymo is far far ahead. Are disengagements important? Yes. Every disengagement is a failure of the car's automation to handle road conditions that would require emergency intervention by the driver. But keep in mind, Waymo drives off the highway as well, whereas many of the competitors with worse disengagement figures are triggering them on mostly highway driving.
Taking time to do this right might be frustrating to those who want products shipped immediately, but the first time a level 2-3 car runs over a schoolyard, people's minds will change very fast.
Yes, but computer vision has moved forward by leaps and bounds since 2014, and so have GPUs.
AD pipelines uses object-detection for traffic-sign detection. RCNN was state of the art for this back then and ran at what 0.2 fps and ~40 mAP ? SSD came out in 2016 and ran at 60+ fps at ~80 mAP. Google's mobilenets implementation probably runs atleast twice as fast and apparently has no noticeable loss in accuracy.
There are more performant and more efficient network architectures than mobilenets. Don't get me wrong, Google's DL research is good, but it's not significantly ahead of field SOTA like they would have you believe.
I don't think mAP on VOC is indicative of real-world performance - especially given COCO mAP is just now cracking the 40s with similar architectures. But the gains in proposal based detection/instance segmentation from 2014 to now have been staggering for sure.
It seems difficult to tell who is the “furthest ahead” because it’s a pretty complex race.
For starters, we can’t even say with certainty who all of the competitors are. Some are very quiet or at early stages but could be working on a breakthrough problem that turns out to be a pivot point.
Some are specializing. Who is going to scale up the best and most cost efficient solid state lidar? There’s a whole list of key problems being worked on.
Do you mean hardware or software? Just the algorithms are a huge piece. Just hardware is a huge pieces.
Beyond price and performance, it’s common that unforeseen factors end up being important in determining who becomes the most successful in selling traditional manufacturers components or whole systems. Surely we don’t realize all of them yet.
Also there is so much still being held close to the vest. These guys are keeping lots of secrets about how far along they are truly, roadblocks, etc.