I don't think I've ever read about an engineering culture that I agree with more than this one. It sounds shockingly humane and practical.
From the somewhat... wistful tone in places, I gather things have taken a bit of a downward turn there. I wonder what pressures caused that? The acquisition, perhaps?
> But although technical management wasn’t an idea that lasted, it was a very good place to be an engineer while it did.
I suppose this may just be something that you either like or don't like, but my experience has largely been the opposite. I don't want a boss who can override my technology decisions just because they're the boss. Even if you only hire people who wouldn't, the fact that they can creates something of a chilling effect.
I think software teams should have a technical lead (who the team does not report to) in order to make sure the team's technical direction is cohesive and consistent. The manager (who the team reports to) should be responsible for their reports' professional development, and should be more of a "team coach" that facilitates interaction with other teams and other departments, and might do things like own (but not have final/unilateral say on) the delivery schedule and backlog.
If you have a manager who is responsible for a decent amount of technical work, the important work of actually managing -- of which there is plenty -- will suffer. I suspect this sort of leadership worked ok in the early days of Heroku because the company was small enough and the early engineers scrappy/independent enough that they didn't really need much in the way of management.
Some of my worst interactions with managers has been when they believe they should stick their nose into technical issues. Perhaps the problem there is that they're not technical enough, and that if they spent enough time coding, they'd be more competent when participating in technical decisions... But then I still maintain that 1) even just holding -- without playing -- the "boss card" creates unwanted tension, and 2) they'd be put into a position to be much worse at actually managing.
Has the Heroku story improved at all for stateful services? I've always seen clustering as essentially unsupported. Doing websockets with something like Phoenix/Elixir means that you have to bring in Redis for pubsub between nodes, which has all sorts of quirks and limitations.
Yo, full disclosure I work @ Heroku. The story is mostly the same on our standard product everyone is familiar with. We have a "private" version that provides a private subnet between the containers as well as DNS targeting (e.g. 1.web.app.zone).
We're working hard on this stuff and I'd love to hear more about what primitives you'd like to see us expose to ease any quirks you've hit.
It's still in beta, but https://www.gigalixir.com gives you a Heroku-like experience for stateful Elixir/Phoenix apps complete with node clustering, hot upgrades, remote observer, and high process uptime. You can use websockets and distributed Phoenix channels without Redis. Full disclosure: I'm the founder.
> Doing websockets with something like Phoenix/Elixir means that you have to bring in Redis for pubsub between nodes
I haven't even tried this myself yet, but it's been stewing in the back of my mind: you should be able to (efficiently) pass sent messages between your nodes using server-to-server websocket connections.
• a "server peer RPC" Phoenix Channel, that authenticates server peers in its connect/3 using a common pre-shared key (ala Erlang's own "cookie") or somesuch
---
Going a bit further: once you have these two components set up, you can actually extend it out to being a fully-fledged transport layer for the Erlang distribution protocol.
Between the two, the API an `inet_websocket_dist` would need to follow is pretty clear.
With such a thing, you get cluster-presence and cookie-based auth "for free", and automatic leader election for use with `global`. But most of what you want is probably just rpc:multicall/3.
> Any sufficiently complicated concurrent program in another language contains an ad hoc informally-specified bug-ridden slow implementation of half of Erlang.
In a similar vein, picking one true way forward from a collection of similar options helps keep engineers productive as they move between components.
I am a strong believer in this. However, i believe it's a wildly unfashionable idea in this day and age, where developer autonomy, 'best tool for the job', and polyglossia are the in things. I was interested to see it in this otherwise pretty hip and happening list.
The idea was that making coffee would be five to ten minute process, during which there would be time to interact with colleagues who happened to drop by the area. The system worked.
There's a similar meme in medical research institutes (at least, the ones i've worked in), which was popularised by the Laboratory for Molecular Biology:
If an engineer needed a new resource for a service being deployed, prototype, or even one-off experiment, they were at liberty to provision it and keep on working, even if that resource wasn’t free. Resources here might include everything from dynos to run an app, to a Postgres backend, to a few extra EC2 boxes for deployment to bare metal (relatively speaking).
I've come across this "self service" idea as a selling point for internal platforms and PaaS, but there seems to be a small but important difference - with a platform, developers can self-service anything that is part of the platform, whereas it sounds like at Heroku, it was anything - full stop. Although, as the author mentions, the fact that they were on AWS meant that that was ultimately still on a platform. Anyway, that difference can matter if there are things that aren't on the platform - if you're using Cloud Foundry, for instance, then you won't be able to self-service persistent disks, network configuration, or anything else at that level, which means that in practice, there is considerable drag from interacting with other teams to get that stuff provisioned. A radical commitment to self-service would be interesting to see.
But although technical management wasn’t an idea that lasted, it was a very good place to be an engineer while it did.
One of the genius things i learned at Pivotal was that a 'team lead' and a 'manager' don't have to be the same person, and that neither of them has to be a 'boss'. Every developer needs a 'team lead' (in Pivotal cant, 'anchor') to lead the day-to-day operation of a development team, and a 'manager' to channel feedback, guide career progression, ensure happiness, etc, but they it's better if they're separate people, and it's better if they're drawn from the ranks of the developer's peers (albeit the more senior ones), rather than being some formally separate layer of the organisation. You should always have technical leadership; you should usually have technical management too, although it probably makes sense for the most senior technical-management people to have non-technical managers themselves.
> 'team lead' and a 'manager' don't have to be the same person
This has been my experience at several places I've worked, including my current work at IBM Watson. I talk to my manager maybe once a month at best[1], whereas the team lead is doing the same work as the rest of the team and in communication with the other members almost every day.
[1]: this is partially because I work remote - I assume I'd speak to my manager a little more often if we worked in close proximity.
Adding just one additional framework, library, or language that performs a similar function to what's already used can add a huge burden on a mature organization. You end up with so much tooling around what you're using, that adding something new means your system is much less robust than if you'd stuck to the things the rest of your org uses.
At the same time, using the same stuff leads to a stagnant stack: it makes it harder to hire and retain, and you miss out on some genuinely better ways of doing things.
A "do whatever you want" policy is destined to fail, but being to strict and disallowing experimentation isn't the answer either, and I struggle to find where the balance should be.
I don't think the "best tool for the job" necessitates needless polyglossia. They obviously have a "best tool for the job" mentality to some extent as they have 3 official/primary languages, not 1 like some places I've worked.
How many different languages/technologies are required for it to qualify as polyglossia?
One more than however many I personally know, of course :).
I suppose the only defensible definition of the term would be more than one language, with some qualifier like "for any given context", because using C++ to serve your JavaScript app isn't polyglot. But then we just argue about how big a context is - are API servers and systems of record the same context?
A more fruitful perspective might be to look at how the decision to adopt a new language etc is made. If an individual developer can do it, it's a potentially polyglot environment, whether that potential has been realised or not. If it requires an organisation-wide process (whether CTO decree or consensus at an all-hands meeting), it's unlikely to be a polyglot environment.
This perspective is definitely a product of my own particular experiences, though, and might not be generally relevant.
From the somewhat... wistful tone in places, I gather things have taken a bit of a downward turn there. I wonder what pressures caused that? The acquisition, perhaps?