I'm in the terminal side of this business. While this seems very interesting, and I'm sure some line will poke at it, it seems very academic. I'm curious if they actually partnered with a shipping line to come up with this.
On the terminal side, all I can say is that I am knee-deep in container optimizations right now and it's a damn nightmare. Every terminal does things extremely differently, even if the terminal is owned by the same company. Even the terminology is often different within a company. You optimize for one terminal, and you have to build 80% of it from scratch for the next terminal. Any solution is incredibly difficult to scale.
When looking at industrial optimization problems, it always appears there's an insane amount of money left on the table (e.g. nurse scheduling in healthcare, train/cargo ship optimization, etc). In my experience there are always undocumented (human) constraints that massively complicate things:
- German engineers balked at tightening features on pre-production vehicles because it would prevent their use for recreational purposes
- Billions were spent in overtime in a healthcare setting, seemed trivial to improve scheduling to reduce overtime, but there were numerous union constraints and simply not enough hiring supply
I wonder how realistic this google solution is (Houthi missile constraints?). In my experience it's often more valuable to develop solutions that can easily adjust to unplanned changes rather than provably optimal ones
I work on these sort of industrial optimization problems, and you're absolutely right about the amount of money left on the table. I had one problem at a previous megacorp that I worked on for about 8 months, and then passed off as a two part solution. the first part was a new optimization engine design, but it also required a second part, a new physical process for crossdocking. To this day, 10 years later, my former colleagues still report actual measured savings of $300M/year, which I still find to be mind-boggling.
You're also right about the complications and the need to be adaptable to various situations that arise. Some of that just comes from people who are afraid of getting their hands dirty and really understanding the domain they're working on. Operations Researchers and Industrial Engineers are not much different from Computer Scientists: sometimes the desire for a beautiful and simple solution prevents you from building something useful. Sometimes in order to build something useful, your idealized fast-converging LP turns into a QP/MIP bastardized rat king that only converges on an optimal solution half the time, but still gets a better-than-before solution the rest of the time. And some of us really hate that it has to be that way. The rest of us laugh their way to the bank.
Indeed!
It is never simple. Sexy simplistic CP-SAT constraint programming solver versus containers with deep frozen goods or tons of fireworks. Both need to go into a very special place on the ship.
Then soft business requirements, keep a weekly schedule.
Plus the real business issue: 5-10% of invoices in logistics are incorrect! Mistakes like wrong rate.
(disclaimer: the stories I heard while trying to digitise end-to-end value chains. We failed.)
Yes, and that's leaving aside the explainability problem. Let say for arguments sake that you have an optimal planning algorithm, if you cannot explain to the people using it why it generates a certain solution, it will not get adopted.
As an employee at a terminal, you have a) no reason to trust some software people who may have never seen one of them steel boxes up close, b) if this works, you or your colleague gets laid off, and c) if it doesn't work, you're now stuck cleaning up the mess the computer will inevitably make for you, which is way less fun than planning things yourself.
The 5%-10% invoices being incorrect is probably on the low end, I'd probably say it's 5-10% in total invoice VALUE. Because of the sheer amount of containers, you need automated invoice calculation. But if you miss an edge case ("We are allowed to invoice a 5 buck surcharge if the container is filled with at least 100kg of explosive goods, AND was loaded during the night shift because dangerous goods handling requirements now require us to have a safety officer on call, which we normally don't have during the night shift") you are leaving serious amounts of money uninvoiced, and you will never know. One of the reasons not that much effort goes into optimization algorithms is that there is so much low hanging fruit left on getting your invoices out correctly...
>As an employee at a terminal, you have a) no reason to trust some software people who may have never seen one of them steel boxes up close, b) if this works, you or your colleague gets laid off, and c) if it doesn't work, you're now stuck cleaning up the mess the computer will inevitably make for you, which is way less fun than planning things yourself.
Also d.) It works sometimes so you get forced to use it and then it gets ransomwared and you're completely unable to work because they laid off so many people you can't fall back to manual operations.
This is so true! Worked in the past on flight optimization problems and the most obvious solution (fly in a direct trajectory, or almost direct but considering wind) would easily achieve a 10% reduction in fuel usage. Now implementing that would be a nightmare, with so many layers involved and the need for realtime comms between aircraft and ground to keep the solutions in sync and conflict free. Unfortunately reality is always more complicated
Nurse scheduling is weird. I worked on a project in this space. Basically the nurses had found their own optimization technique over the hospital’s scheduling methods.
I did not work in shipping, but I did work in manufacturing for 15 years, mostly with a focus on test engineering & automation, but also getting into warehouse/material management and shipping/logistics. My masters is in operations research for manufacturing.
I agree with you that objective optimization is largely academic. There are always reasons why it's impossible or impractical to adhere to the most efficient versions of standardized processes. Sometimes these are "stupid" (people) reasons, and frequently they're reasonable justifications that accommodate for any number of externalities (weather, downtime, supply chain disruptions, lumpiness in demand signals and forecasts based on seasonality, etc).
That said, it's almost always smart to start with the most efficient version of a process and then add exception-handling, rather than to create a standardized process based on known exceptions. If you let exceptions become the rule, you'll always be operating less efficiently than optimal.
Same here, I'm in the business of terminals as well as hinterland transportation companies. There is no shortage of people trying to solve the 'sexy' academic problems like stowage plan optimization, berth assignment problems, crane splits, etc, but only very little gets implemented in practice. It is a tough business, with an even tougher political landscape, since dock workers traditionally have very strong unions.
All of the problems you can neatly divide and name are interrelated in practice, and trying to have algorithms "magically" figure it out for users is pretty much guaranteed to fail. This business eats software hotshots who think they can swoop in and "just" run some algos, "because it's just a TSP / Constraint Solver / [insert your approach here] right?". This business can definitely use bright minds, but start humbly, and start by talking to real world users please.
You don't appear to have any contact details on your profile, but if you ever would like to exchange thoughts with another company in this business, shoot me an e-mail. I think we're doing some interesting stuff for the <1 million TEU/year segment of terminals, especially if they're heavy into multi modal.
> "it seems very academic. I'm curious if they actually partnered with a shipping line to come up with this."
Unlikely. I worked in this business and the idea of mathematical optimisation is always the first one to be suggested by someone who's new to this industry. Global shipping companies are clueless about tech and only do what they absolutely must to comply with the regulations, the rest is ignored. Give them an XML Schema for an EDI doc and they will stuff an Excel sheet into a CDATA. They know their game, they wore out the infinite patience of IBM who wanted to convince them to use blockchain to track shipments.
Shipping companies are clueless about tech, and tech companies are clueless about the day to day activities of shipping. They think they can tech their way through all problems.
We recently met with a big tech company who we were told to partner with. They start with an angle that made highly suspicious they basically wanted to build a fully automated terminal. When we told them just one fairly mild story of what union issues we deal with, they all gasped and said, "what?" and quickly dropped it.
Yes, as someone who used to work in that field for a bit in the customer side, I have to agree. Thisbis an operational topic, and those involve a ton of phone calls, people all over the place at multiple companies using multiple unconnected systems, a fair share of Excel and e-mail.
Some planning algorithm like this might work, for sure. But onpy if it is included in whatever software being used for this kind of planning. Otherwise, it is a nice academic excercise at best.
For the customers in my size category (50k to 1 million TEU / year) the answer is that they may have 2 or 3 planners, optimizing berthing slots and stowage plans, but other than that there is no coordinated effort.
You have to realize that these terminals are typically a lot smaller in terms of manpower than you might think. A terminal handling 1 million TEU/year may operate with 2-3 crews in a 16 hour/day schedule for handling ships and trucks, plus some extra people handling the empties. Let's say it's 60 dock workers a day. A typically office might only employ about 12 people in overhead: one for customs/quality stuff, one for invoicing and finance, one HR, a bunch of order entry / customer service people, and 2 or 3 yard/berth/stowage planners.
Unless you look at the biggest of biggest terminals, you have to essentially regard them as SMEs in a very non-sexy business, and they treat tech and fundamental research into optimization accordingly.
One day when I leave, I swear I'm going to write a tell-all book about this industry. Until then, here's some condensed info.
To set the stage, this is a blue-collar industry, and what happens, so I've been told, is quite common in blue-collar industries. A lot of good ol' boy networking, people getting promoted who have no business being promoted, and as a consequence, a lot of corruption, ineptitude, and inefficiencies. The software in this space just flat out suck and anyone at a FAANG company would be appalled at how we write software. It's literally longshoremen trying to run software businesses.
On the other hand, as a terminal, it's very hard not to make money, so there's a lot of players, and unfortunately a lot of, at least shady deals.
I'm involved with a company that sells to terminals that don't have home-grown software systems. I don't have that much info on terminals that have their own software staff, however, there just isn't that much breakthroughs in the space. It's a conservative industry with a lot of money already, so there traditionally have been little incentive with high risks.
In addition to the aversion to risk, there's a lack of software product and engineering strategy to invest in optimization properly. One major vendor, I was told by a colleague did invest substantially, failed, and "it almost bought the entire company down." My company has dabbled before. We've had a couple of PhDs on staff. We even had a neighbor search solution like in the article developed. It worked in simulation and testing, and failed miserably in real life. The customer didn't take it, and we never had the vision to actually salvage anything from the effort.
There are a few vendors in this space, but again, the landscape is pretty sad. I know of one that are flat out frauds. They exist because the CEOs used to be in the industry and have credibility. They're still around because they're able to con non-technical terminal managers and string them along for a while. A few more maybe have 1 or 2 competent data scientists, but that's not enough. I've worked with one company that is good and is easily the most qualified, but they have never had the marketing and leadership to make themselves the billions they should be worth. They also simultaneously price themselves out of range for most terminals.
The industry is changing, though, because private equity has gotten involved in many companies. They have taken over some internal initiatives and basically dragged many of these good ol' boys against their will, or forced them out. They've also forced companies to partner with tech giants to address these issues. IMO, this is the only way we will see supply chain optimization revolutions any time soon.
As someone in the industry as well for many years (coming from the business side having ventured into software), the main problem is always that the software just is not good enough and do not account for all the edge cases - leaving many users better off using pumped up Excel sheets combined with a few hours on the phone every day to account for all the nuances and changing customers/suppliers demands.
In short: It is not good enough. When it is good enough, customers adopt it.
This experience has given me first-hand experience of a PE takeover, and the standard line of PE guts the soul of a company and sells off everything is crap.
I don't know how many meetings I've been at where the PE team blatantly states over and over, let us help you, tell us what you want to do, tell us what you need, and we completely flub it. Sometimes our CEO is too paranoid, sometimes too prideful, sometimes too oblivious, but mostly just doesn't have the knowledge on how to run a software company.
PE is going to run out of patience soon, and it won't be good for us. There's going to be headlines of how PE gutted us, but in reality, we gutted ourselves.
I agree, mostly, but I've also seen (and been in) situations where the PE acquired a company for it's customer base and knew going in that it was going to gut the current leadership and install all new systems & processes [that would support 10x scaling]. This doesn't make PE bad or evil, but it's not accurate to state that PE overlords are always hands-off.
But what about the cases where the PE firm asset strips the company, loads it up with debt and then walks as the company implodes and everyone loses their job? That seems to be the model of some PE firms. But maybe those are the ones that mostly make the news. I would certainly hope they aren't all predatory vulture capitalists.
BTW the current state of English water companies is a not a good advert for private equity.
Not in shipping but in high volume manufacturing it's generally expected that anyone on the shop floor, or creating systems for the shop floor, is exerting at least some gray matter toward optimization. In most places of decent size, there will be at least one "manufacturing engineer" or "operations lead" whose primary job is to focus on optimization.
On the terminal side, all I can say is that I am knee-deep in container optimizations right now and it's a damn nightmare. Every terminal does things extremely differently, even if the terminal is owned by the same company. Even the terminology is often different within a company. You optimize for one terminal, and you have to build 80% of it from scratch for the next terminal. Any solution is incredibly difficult to scale.