Some of the news I've seen associated with Saildrone recently has really fascinated me, like sailing one of the drones into a hurricane [0] and returning back data (including video) from the event! Just wanted to say great work to you all on that, and I look forward to seeing what else you have coming up!
Is there a reading list to get into subjects like: bathymetry, acoustics, perception, navigation and propulsion? An open source software suite for ocean science, perhaps?
I'd dig a DIY kit for underwater photos. Analagous to a toy drone cam. Or a 1:50 model of your Sail Drone ;)
Boats have had a higher level of autonomy than cars for a while. In the early 2000s you could plot a course on a computer and that software would navigate that course, avoiding some (charted) obstacles and the like. See: https://www.sperrymarine.com/integrated-bridge-system (sorry for crappy product page)
One thing ships have is tons of redundancy. You can always go one level lower in control. Go from the charting system picking heading and coarse down to you setting heading directly via an autopilot. You can go from that to rudder control. Down all the way to manual control of the rudder.
That kind of redundancy, plus a trained and alert crew, make a world of difference and seems quite different than the car world.
> Boats have had a higher level of autonomy than cars for a while.
This is really a mixed bag - some categories of vessels have had more advanced navigation systems for decades, but on the the other hand a family minivan is about two decades ahead of a bowrider that goes for about same price when it comes to autonomy.
More directly addressing the article - marine autonomy is an easier problem technically, but there is a significant market problem. Aside from water taxis, there is not the same compelling economic case for autonomous small boats as there is for commuter cars.
Disclaimer: I work on ADAS systems for recreational boats.
The difference between a 50k 4Runner and a 50k boat is that while both are status symbols to show off you don't expect to be minimally engaged while operating the boat. Driving it is part of the fun.
40% of the world's populations lives within 100km of a coastline [1]
I could see some applications when combining autonomous small boats with multicopters, to manage last mile delivery along the coastlines and various rivers or lakes.
Even consumer level boats are starting to get this technology. You can get nav systems now which have fully autonomous control (I believe via garmin). They are extremely useful when fishing. You can set a waypoint and have the boat drive you there while you take care of your tackle, live well, etc. Then, once you arrive, the nav system will keep you in the same spot without having to drop anchor.
People have been using self-piloting sailboats / automated navigation on personal water craft for decades. The issue of course is collision avoidance, traffic navigation, and determining the best route autonomously.
Collision avoidance, traffic navigation, and determining the best route autonomously is already available for ships. A lot of the same sensors are used in cars as well.
Disclaimer: I'm the founder of blkSAIL (black sail) that builds AI and autonomy for ocean-going maritime ships. We are an MIT CSAIL spinoff. We started with an aim to become Uber for the waterways, then pivoted to large ships.
Many of the comments below touch on many key points: (1) most commercial ships have a crew between 8 to 20, (2) auto-pilot in ships exist since the GPS was invented, (3) when ship hits the fan, there's not much you can do.
(1) Autonomy in maritime is not to replace seafarers, the navigation crew is less than 30% of the crew. And crew is negligible OPEX compared to fuel etc.
(2) Because the industry is the backbone of global trade, it moves much slower in adopting technology for the sake of technology. However, when there's a simple reliable system that can clear benefit, the adoption happens. So given that there's few[1] lanes in the ocean, you can easily go from one waypoint to the next.
(3) Onboard a ship, there's no such thing as breaks. In most cases, you know about a collision or a grounding 12+ minutes in advance. The challenge is in getting the predictions right and abide by the rules of the road. Most advanced LIDAR have 1km range, which is too close, cameras don't see much neither[2].
[1]: there are quite few channels around ports to ease traffic that are one-way.
[2]: the resolution is too low to see far out. When using zoom, the stabilization is a nightmare.
The L3Harris Mk20 optical sensor supposedly has excellent stabilization when using zoom. But it's intended for warships and might not be suitable for civilian autonomous vessels.
> Where autonomous vehicles have had the most success is in environments with a lot of predictability and structure, which is why I really like the idea of autonomous urban boats designed for cities with canals.
I love a good autonomy challenge and love being near water, so this is close to my hearth.
That being said this sentence seems to imply that city waterways are predictable?
I'm not familiar with Amsterdam, my primary experience is with Oxford. Here you can easily see two crews trying to manage to leave the pier at the same time, while a third is rowing full speed towards them. Meanwhile kayakers and stand up paddle boarders are everywhere, and you can see the occasional swimmer bobbing in the water too.
Often as an onlooker or occasional participant I can't figure out who wants to go where, let alone what path they are actually going to take.
Its a ton of fun, and a great autonomy challenge I presume, but I wouldn't call it easy or predictable.
There is a code of practice for the roads (Highway Code[0]) and for the waterways (Col Regs [1][2]) apply more informally.
On the river you stick to the starboard, use appropriate nav lights after dark, don't create breaking wash and stay under 8 km/h.
On the road you don't run red lights, cycle on the pavement, cycle behind reversing vehicles with limited visibility mid-manoeuvre, collide with pedestrians, etc etc.
As someone intimately acquainted with both the Oxford waterways and roads I would say that road users in aggregate (mostly bicycles) behave far less lawfully, predictably and safely than users of the the Thames! If I were designing an automonous vechile I think it would be far easier to do it on the water!
All of these "oh it's easy" assume absolutely no external factors and fully controllable environment. Self-driving(sailing?) boats in city canals are super easy to do, if you assume nothing else is ever in those canals(which is definitely not true).
It's the same when people just say "oh once all cars are autonomous we'll just have cars driving through city centres at 200km/h!" - to which I say ok, but what about cyclists?
There probably won't be any cars in city centers by the time that autonomous vehicles become the majority.
Most cities are moving away from a car centric approach and move towards a more environmental and economical sustainable transportation system that is more efficient.
not if you're one of the top dog cybernetic driver mercs of Mackey's Kill Gang 2000, you can take what you want, do what you want, whenever you want and nobody better be foolish enough to get in your way!
It's a great city to live in!
Until of course a young kid with a secret past and the cool driving skills of an angel decides to stand up to Mackey and the Kill Gang 2000, then it's a great city to die in!
Waterways don't have clear lanes, intersections and turn signals, but you generally have unobstructed view, and every participant with the exception of the rare swimmer is much less nimble than a pedestrian or biker (and thus more predictable).
Waterways are very chaotic, but in some ways it's still more predictable than city driving.
>>the exception of the rare swimmer is much less nimble than a pedestrian or biker (and thus more predictable).
The problem is that boats are an absolute pain to turn and are the very opposite of "nimble" too. Yes if you see a swimmer in the water 20m ahead of you they probably won't move very far, but also to avoid them you should have started making a turn about a minute ago. Having done some canal boating in the UK it's just inevitable that you will keep colliding with things, and unfortunately you leave a lot of kayakers pissed off because it's almost always them having to take an evasive action.
It'd be an interesting challenge writing an algorithm for canal boats on narrow British canals that can deal with the effect of wind, bank effect and squat and underwater obstacles that simply shouldn't be there in a vessel that's 57' long, pivots in the middle only and generally between a few feet and a few inches narrower than the canal itself. Tbf kayakers are easy to deal with if you actually see them in time to slow down. Most obstructions don't move as fast to get out the way!
A robot to do locks for me when I'm feeling lazy would be a bigger deal though :D
Perhaps not ideal but as a swimmer you almost always have down as an evasive direction. You do need awareness but this has saved me from multiple shallow hulled fast boats and water jetskis. You can hear them miles away though directionality of the sound is poor or absent.
When I'm doing open water swimming workouts I'm usually wearing a wetsuit buoyant enough that I can't dive down. I probably wouldn't risk swimming at all in waters where jet skis are allowed because they're notoriously irresponsible. It's a good idea to also tow an inflatable buoy to increase your visibility.
The banks are often quite far or too shallow to realistically be a concern. Even when dealing with sailing vessels whose rigging extends beyond the deck and is subject to change position by the whim of the captain and wind. You’re very often not close enough to the other ship to matter. What makes it more fun is that sailing vessels have the right of way to all other vessels (with the exception of large barges and container ships) IIRC. This means an AI piloting a smaller craft would have to 1, recognize a sailing vessel, 2, recognize whether it is powered by wind or not and 3, yield the right of way.
Amsterdam is definitely less chaotic than Oxford, the canals are wider and there is less "pedestrian" traffic, the closest you'll find is water bikes or tubs with drinking people.
The main canals are mostly visited by touring boats or motorized leisure craft.
Autonomous Boats sounds great, through I do see a few things that will be needed that thankfully might have a better chance of being implemented on boats compared to cars. Coast guard and Maritime search and rescue will need to have the ability to shut down any boat in an radius in case swimmers or scuba divers get into the lane.
I can also see this being used in places like Norway and Sweden where car ferries are used as an alternative to bridges. Those are slow (<10 knots) barge-like boats, and in my experience quite overbuilt in terms of withstanding any kind of weather. They are quite bit more dangerous in term of mass and distance, but they can be quite cost-effective compared to bridges and environmental friendly if electrified. In terms of making those safe enough I suspect you would need more than just lidar, but since they are bigger and have a potentially higher budget I can imagine they could afford a bit more radar/sonar/radio technology to identify ships and other things that might be in the water.
What advantage would there be in making ferries in Norway autonomous?
I think that even the ferry where I live (shortest scheduled crossing in Europe I think, less than ten minutes) would still need someone onboard to deal with the customers and handle emergencies. So you might reduce the crew from three to one during normal operation but when something goes wrong those other two will be needed again.
I am unsure how much you actually need someone to deal with customers. Roads/bridges are already automated in terms of reading the registration sign and the payment is also automated. The biggest issue from my experience is customers not knowing which lane to go to or where to park, but that seems like a problem which could be fixed through clearer markings and signs. The ferry is basically just a floating road, with the driver inside the car at all times.
There are many smaller island with permanent resident, and only so much money to build bridges. Some places I know only have a ferry that goes there a few times a day. Making those kind of ferries cheaper would both improve those peoples lives but also enable more people to live in such locations. Without crunching the numbers, it seems possible that an autonomous ferry could be made cheaper than what an equal distance of road on land.
The ferry is typically much more cramped than the road. Cars, bikes, motorbikes, trucks, tractors, diggers all have to be able to get on board without interfering with each other and without trapping anyone in their vehicles. Even with a willing and cooperative population this gets occasionally difficult, especially when tourists or other people who have never travelled on the ferry before are involved.
> Without crunching the numbers, it seems possible that an autonomous ferry could be made cheaper than what an equal distance of road on land.
The existing ferry is already considerably cheaper than a road bridge would be. It seems to me that an autonomous ferry would merely shift the work to development and maintenance of the system and add risk.
There is some potential to automate navigation for ferries travelling on short fixed routes. But trained crew members will always be needed to deal with emergencies such as fires, man overboard, flooding, mechanical failures, etc. It's just completely impossible to automate that stuff.
Depending on the area there aren't necessarily defined lanes. Any vessel could by conducting dive operations so autonomous vessels would need to maintain situational awareness by monitoring VHF radio and watching for signal flags.
Civilian sonar systems aren't usable for collision avoidance.
Makes sense for larger boats in calm conditions as the input lag is enormous but very predictable, and the terrain is flat and simple, especially on open water. In fact, it makes so much sense that mechanical autopilots have been a thing for centuries!
So here's my comparison between cars, boats, and aircraft:
Car Boat Air
Worst case you can safely stop Sometimes Yes No
Reaction time required 100ms Minute Minute
Trajectory tolerance 0.5m (city) 5m (harbor) 5m (takeoff / landing)
Trajectory tolerance 2m (highway) 100m(open sea) 1km(mid flight)
Collision survivable Yes Yes No
Please add or point out my mistakes (I kinda pulled the numbers out of my ass).
I'm not saying they can come to an instant stop. But if they do stop they are in not in danger to themselves or the surrounding traffic. That means that if you see a sticky situation about to happen then you can start slowing down to make everything easier and you can keep slowing down until you are either able to safely handle it or completely still.
If the computer has no idea whats going on then slowing down and requesting human assistance is very viable.
Stopping in heavy weather or currents is not at all safe. The ocean is a dynamic environment. Vessels must maintain steerageway or else they risk broaching or drifting into obstacles. In shallow waters a vessel in distress can sometimes drop anchor to hold position but that isn't always a practical option.
(Semi-)Autonomous open sea cargo ships might be an even better use case. One could imagine totally unmanned ships cruising the planet and delivering goods without putting anybody at risk (or subjecting to the long days of confinment). But one might need people on-board for quite some time: eg mechanics when invariably something breaks down
A canal environment is still quite crowded and maybe the economic motivation rather limited but its probably an excellent test bed
Cargo ships are largely autonomous once underway. They have a small crew of 8-12 members who largely do maintenance, watch, and cook. Then, near shore it's all hands as they prepare the ship for docking.
> 8-12 members who largely do maintenance, watch, and cook
I suppose the "cooking" part can be eliminated once you don't need maintainance and watch :-)
"Watch" is an interesting (and solvable?) challenge. Understanding the sea environment (visually, sonar etc) from the local (ship's) perspective can complement e.g satellite imagery. But there is also e.g. watching against piracy
Maintainance seems the toughest. Unless the engineering is perfected to the point that it is only required at ports.
Most routine maintenance could be done in ports. The problem with that approach is that ships would then be stuck in port longer to finish maintenance work after unloading and loading cargo, which means ports would need more berths and shipping companies would need more expensive ships just to carry the same total load. It's actually cheaper to hire crew to do that preventative maintenance and minor repair work while underway.
Watch is not solved and most of our sea rescue services rely on cargo ship crew as first responders. Rescue planes don't have the range for recovery. Planes can't land in blue water. So it comes down to the small crew keeping a sharp look out for someone in distress.
I can see where this would work well in the scenario described: "in environments with a lot of predictability and structure".
Waterways don't generally have vehicles doing high speeds and do have wider lanes to travel within than roadways.
I live near two big lakes with a lot of boats on one of them (Table Rock Lake in Missouri) zooming all over the place. You could easily plot a course to get most anywhere on that lake without worrying about hitting anything or even coming close to any fixed structures or shallow spots, but you'd have to "see" or communicate with other boats to avoid those traveling or drifting or anchored most anywhere out there.
This article pretty clearly states this tech is not there yet and my experience is that there would be dumbasses that would run into it on our lakes if it were out there. Probably not very often, but for sure it would because it does now.
And it's also a lot less important. If you are using a boat for pleasure, part of the fun is in driving it. Meanwhile, for commercial vessels, the ROI is not going to be very meaningful. A ship with capacity to carry 20,000 20-ft containers only needs a handful of people operating, only a few of whom are actually piloting it. A cruise ship might have tons of crew, but again, only a few people doing piloting and navigation. There's probably a small market for small commercial vessels and maybe some recreational boats, but overall, there just isn't the demand for this in the way that there is for cars or trucks.
The are looking at things above the water. Boats are in the water. And that is where most autopilots can't help. APs know nothing about tides, currents, waterline obstructions. As a powered boat the Roboat has one of the lowest priorities in Colregs; behind human power, sail, under tow, and limited maneuverability.
There are no lanes to follow in waterways you can pass on the left or right. A big component of waterways traffic is using your mouth, getting on the VHF, and hailing the other captain.
Larger vessels already transmit their position via AIS, which is helpful for collision avoidance. The problem is that you can't rely on it. Transmitters can fail or get switched off. The smallest vessels don't have AIS and probably never will. How would that work on a Hobie Cat that has no electronics at all?
AIS has a 50ms window to transmit data on just 2 channels(69,70). Not much of a problem when there are only a few ships in a few miles radius. But in a congested harbor transmitters may never get that window in a timely manner.
I've experienced cargo ships popping up out of nowhere in AIS (I could see them on radar) in Valleta harbor. A proper AP will have to ingest information from all sources (vision, sound, radar, AIS) to be able to work outside of the low traffic areas.
Smallest vehicles likely never be automated. On other hand I hope everyone piloting them are smart enough to avoid the big ships. Most sufficiently big things are decent radar targets anyway. Or special military vehicles, which I expect to do the avoiding. As if you are the one that can't be seen, you better be careful one.
Under current rules of the road, there are circumstances where big ships are required to avoid the smallest vessels. That can particularly be the case if the small vessel is a sailboat or otherwise has restricted maneuverability (alpha flag), but there are a variety of factors involved. Small vessels operating close to shore often have no radar.
I didn't even realize it was unsolved. I thought it was like flying, where the pilot is only there during take off and landing, and the rest is handled by computers.
What about autonomous carts? Why aren't we seeing more attention on that, above autonomous cars, which drive fast, have passengers, and can hurt someone?
I'm picturing a slow moving crawler that has a public bin on top, slowly trundling along the sidewalk in a loop from the garbage depot through town.
Or a cart with space for dishes so that you can dump your used plates and get freshly cleaned ones from a central washery.
Something I don’t read enough about that would make autonomous cars a lot more reliable is smart tech in the road infra. Why don’t stop lights send a beacon? Roads and lanes should send a signal. Cars should send signals to each other about their intent eg. turning indicators should not be purely visual, etc. Are there standards for this stuff being worked on?
Every stop light is actually required to have a beacon to be fit for purpose. It's universal among all manufacturers in every country.
When it's time to stop, they emit 700nm waves, but crucially, only in the direction that they apply. In fact, brake lights are required to emit almost identical waves, but in a different pattern.
Having spent the last 30 hours autonomously driving from Texas to the east coast, I can confidently say it never failed to detect these lights, and the existing standard is both sufficient and not prone to invisible errors, which you seem to want to introduce.
