I'm happy to answer myself. Previously unknown challenges are just the various facets of building and deploying an application. Its not so much that they were unknown problems so much as the abstraction we designed for doing so proved challenging to solve those problems.
Ultimately, Otto was trying to be a masterless PaaS (Heroku, etc.). When you frame it that way and think about all the things you'll have to solve it becomes challenging. On top of that, we always wanted Otto itself to be fairly "thin" and delegate its difficult duties to the rest of the stack. This required us to build a bunch of features we weren't ready to build into our other products OR risk bloating Otto itself.
Well, in Heroku proper, you feed its git repos an app, it figures out what type of app it is, and applies the right build pack and hosting environment for it. Keeping build packs up to date, keeping all the scripts running, and making sure an app has associated dependencies, etc--I imagine that's the difference between an independent setup you can self-host quickly and easily and one that's very dependent on an ecosystem of Heroku maintainers, tooling and existing server infrastructure...
It's a very hard problem to solve, and one which will likely only catch on as devops tooling improves and becomes expected for apps, and as app runtimes standardize. Alternatively, you could look at the myriad ways operating systems package applications, and the ways applications allow themselves to be packaged, let alone store data in production, and ... basically give up on this ever happening in an easy, hands-free automated way.
> Keeping build packs up to date, keeping all the scripts running, and making sure an app has associated dependencies, etc--I imagine that's the difference between an independent setup you can self-host quickly and easily and one that's very dependent on an ecosystem of Heroku maintainers, tooling and existing server infrastructure...
I work for Pivotal on the Cloud Foundry buildpacks team. 4 of our buildpacks (Ruby, Python, Go, NodeJS) are downstream forks of Heroku's.
We merge from upstream approximately weekly, but the pace has definitely dropped.
We build all the runtime binaries we ship with our buildpacks. We also build the rootfs it all runs on. Some of these pipelines are now fully automated. For example, when a NodeJS tag lands, our pipeline will build the binary, add it to a buildpack and put it through our battery of test suites. Our product manager can make a release with a few keystrokes and a button press.
The difficulty of engineering really comes down to the nature of the ecosystem you're turning into a buildpack. We did an article on writing buildpacks[0], taking Rust as our example. It was a doddle, because of Cargo. Meanwhile our PHP buildpack performs incredible gymnastics to make a 12-factor cloud platform look like a shared host circa 1999.
Ultimately, Otto was trying to be a masterless PaaS (Heroku, etc.). When you frame it that way and think about all the things you'll have to solve it becomes challenging. On top of that, we always wanted Otto itself to be fairly "thin" and delegate its difficult duties to the rest of the stack. This required us to build a bunch of features we weren't ready to build into our other products OR risk bloating Otto itself.
Overall, it was too early for us to do.