You are making it sound as if there was something special about my persona in this situation. Well, there was nothing special about "my skills". Anyone diligently following due process would've come to the same conclusion.
Another problem with this situation is that it went nowhere. The project disintegrated for unrelated reasons. So, there's no proof it would've been actually bad.
But, like I said, this is not an isolated incident. I have a better example of bad design created for the very same reasons (but, by a different person) that did and still does have negative consequences for the product. Bonus: making a test that shows the design is bad is extremely costly (basically, unrealistic).
A little bit of history first. A complex computer cluster management software was created without any plans for how to upgrade it. Traditionally, and very often still it's distributed to air-gaped systems, so it's not, and will never be provided as a service. Typically, such software manages anywhere from couple hundreds to tens and in some extreme cases hundreds of thousands of compute nodes, switches, storage nodes etc.
Eventually, the company realized they need some automated way to perform upgrades. Their first effort was to design a procedure for in-place upgrades. Customers rebelled. Not only does an in-place upgrade guarantee down-time, due to the complex nature of the software, usually (or, most likely, always), upgrades didn't go smoothly and required engaging the technical support with days or even weeks worth of email back-and-forth and potential loss of data.
This is when the company realized it needed a procedure for upgrading "in parallel". And here they came up with an idea of reusing existing code for in-place upgrades: they'd clone the management elements of the cluster, upgrade those, and then transfer the rest of the cluster under new management.
They saw it as a win. I saw it as an absolute disaster. My experience with the in-place upgrade was that it would typically leave behind a lot of shreds of the old system. Unused configuration files, or, sometimes, entire databases with confusing names would be left behind. Patched configuration would often end up having features no longer supported by the new system. If the old system lacked validation for some inputs, then new, improved validation procedures would either skip entirely over old data, or struggle with its indirect usage.
Finally, this management software had a lot of third-party integrations with popular tools. Upgrading it in-place often times made it impossible to upgrade both at the same time (eg. upgrading storage engine like Ceph while also using this storage engine to store management configuration). Every such bootstrapping required very convoluted and impossible to test code. In many cases customer would be told to simply remove the integration prior to upgrade and re-configure it afterwards. Of course in case s.a. integration with Kubernetes, this meant customer would lose data (in the form of container state that cannot be reliably serialized and saved somewhere).
To me, the solution was obvious: instead of clone and upgrade in-place, one should install the new system and import the configuration from the old one. This would not allow old stale data, would have to re-validate old input and prompt the user to fix retroactively validation errors. Upgrade and migration of integrations s.a. Ceph or Kubernetes would still be hard, but not as hard as before, as the same principle would apply to them too.
No matter how much I campaigned for my version of design, I couldn't change the situation. The person behind the existing design outranks me. Being higher on the management ladder, it makes them think they know better. They never even tried to estimate the cost the company had paid in providing tech support to a decrepit and poorly built system, or how much development hours had been spent dealing with the bad approach enshrined in bad design, while the same effort could've been applied to development of new features or, at least, to reducing the load on the tech support. I'm sure that to date the company wasted (or lost...) hundreds of thousands of euros or dollars because of the bad design. Probably more: it's hard to estimate the cost of the lost opportunities.
In order for me to prove myself correct, I'd have to build an alternative upgrade procedure. The original effort for this feature had a team of 3-4 developers working on it almost exclusively for at least half a year. Today, the company supports about four dozens of versions of its own product, and the new upgrade system would have to support at least half of that. I will never get funding for such a project, let alone on writing a test that compares the two designs (via their implementations). That'd be many years of work.
----
And the author of this design? -- Well, he's exactly the kind of guy who believes, literally, that the best is the enemy of the good. He works very fast, and often volunteers to help customers directly by adding more untested features to the product. Him being very senior gives him a free pass on code reviews and design documents which are otherwise expected from everyone else. He sees himself a hero, someone who day after day saves the company from devastating consequences of... his earlier bad ideas.
> Anyone diligently following due process would've come to the same conclusion.
> To me, the solution was obvious
> I will never get funding for such a project,
> bad approach enshrined in bad design,
> He works very fast, and often volunteers to help customers directly by adding more untested features to the product.
> He sees himself a hero, someone who day after day saves the company from devastating consequences of... his earlier bad ideas.
What I'm hearing is this guy delivers features to the customers, but your ideas are better, obviously.
If they were obviously better, you'd be able to secure funding to do them. What I think is actually happening is you've been unable to demonstrate the value of your ideas, or convince people they are valuable. The conclusion I am drawing here is that you haven't shown that your way is better - you just believe it is.
My belief is that the attitude that people suck if they write tests that must be ordered, "indicates a lack of understanding that finding the right balance of speed, cost and quality is a compromise." But, many developers believe that their pure code is just always "better".
> What I'm hearing is this guy delivers features to the customers, but your ideas are better, obviously.
Well, then you need to pay more attention... Your interpretation is wrong.
> If they were obviously better, you'd be able to secure funding to do them.
You don't understand the difference between quality and pricing policy. Best food is not the "fast food", even though customers love it and it's cheap. The "fast food" is causing obesity epidemic, diabetes epidemic and a bunch of other health-related issues that, in the grand scheme of things make the savings on the food quality not worth it. But, for some people it's easy to weasel their way out of this problem by pretending that repercussions don't exist, and that they will bear no responsibility for the consequences.
The "deliver features to the customers" guy is the one who internalizes profits and externalizes expenditures. He's a douche canoe. There's nothing positive about people like him. The fact that he gets budget to do idiotic and harmful things is not because they are valuable, but because this creates a condition for a scam that profits those funding him.
In the same way how not everyone is a dietologist and cannot be expected to assess the value of the food they eat, and so has to rely on experts, the customers using our software will not know how well different features of our software are implemented. All they have to go off is a changelog and some promotional materials. There's enough of a time buffer between the customers buying the product and discovering how crappy it is for the product authors to still make profits. And, given the conditions of our market, where the product we make is, basically, without competition, even if the customers discover it's true quality, they'll come back bagging to fix things rather than bail on us. Which gives even more freedom to the "deliverers of features to the customers".
> Well, then you need to pay more attention... Your interpretation is wrong.
> You don't understand the difference between quality and pricing policy. Best food is not the "fast food", even though customers love it and it's cheap.
> There's nothing positive about people like him.
This is the exact problem I'm getting at: Individuals don't get to unilaterally declare what good software is, what the worth of people is, what people do or don't understand.
This guy may in fact have a more nuanced and "better" understanding of software and what goes into making it, than you. Prove that he doesn't! If you can, you can take that proof into work and take his job!
Another problem with this situation is that it went nowhere. The project disintegrated for unrelated reasons. So, there's no proof it would've been actually bad.
But, like I said, this is not an isolated incident. I have a better example of bad design created for the very same reasons (but, by a different person) that did and still does have negative consequences for the product. Bonus: making a test that shows the design is bad is extremely costly (basically, unrealistic).
A little bit of history first. A complex computer cluster management software was created without any plans for how to upgrade it. Traditionally, and very often still it's distributed to air-gaped systems, so it's not, and will never be provided as a service. Typically, such software manages anywhere from couple hundreds to tens and in some extreme cases hundreds of thousands of compute nodes, switches, storage nodes etc.
Eventually, the company realized they need some automated way to perform upgrades. Their first effort was to design a procedure for in-place upgrades. Customers rebelled. Not only does an in-place upgrade guarantee down-time, due to the complex nature of the software, usually (or, most likely, always), upgrades didn't go smoothly and required engaging the technical support with days or even weeks worth of email back-and-forth and potential loss of data.
This is when the company realized it needed a procedure for upgrading "in parallel". And here they came up with an idea of reusing existing code for in-place upgrades: they'd clone the management elements of the cluster, upgrade those, and then transfer the rest of the cluster under new management.
They saw it as a win. I saw it as an absolute disaster. My experience with the in-place upgrade was that it would typically leave behind a lot of shreds of the old system. Unused configuration files, or, sometimes, entire databases with confusing names would be left behind. Patched configuration would often end up having features no longer supported by the new system. If the old system lacked validation for some inputs, then new, improved validation procedures would either skip entirely over old data, or struggle with its indirect usage.
Finally, this management software had a lot of third-party integrations with popular tools. Upgrading it in-place often times made it impossible to upgrade both at the same time (eg. upgrading storage engine like Ceph while also using this storage engine to store management configuration). Every such bootstrapping required very convoluted and impossible to test code. In many cases customer would be told to simply remove the integration prior to upgrade and re-configure it afterwards. Of course in case s.a. integration with Kubernetes, this meant customer would lose data (in the form of container state that cannot be reliably serialized and saved somewhere).
To me, the solution was obvious: instead of clone and upgrade in-place, one should install the new system and import the configuration from the old one. This would not allow old stale data, would have to re-validate old input and prompt the user to fix retroactively validation errors. Upgrade and migration of integrations s.a. Ceph or Kubernetes would still be hard, but not as hard as before, as the same principle would apply to them too.
No matter how much I campaigned for my version of design, I couldn't change the situation. The person behind the existing design outranks me. Being higher on the management ladder, it makes them think they know better. They never even tried to estimate the cost the company had paid in providing tech support to a decrepit and poorly built system, or how much development hours had been spent dealing with the bad approach enshrined in bad design, while the same effort could've been applied to development of new features or, at least, to reducing the load on the tech support. I'm sure that to date the company wasted (or lost...) hundreds of thousands of euros or dollars because of the bad design. Probably more: it's hard to estimate the cost of the lost opportunities.
In order for me to prove myself correct, I'd have to build an alternative upgrade procedure. The original effort for this feature had a team of 3-4 developers working on it almost exclusively for at least half a year. Today, the company supports about four dozens of versions of its own product, and the new upgrade system would have to support at least half of that. I will never get funding for such a project, let alone on writing a test that compares the two designs (via their implementations). That'd be many years of work.
----
And the author of this design? -- Well, he's exactly the kind of guy who believes, literally, that the best is the enemy of the good. He works very fast, and often volunteers to help customers directly by adding more untested features to the product. Him being very senior gives him a free pass on code reviews and design documents which are otherwise expected from everyone else. He sees himself a hero, someone who day after day saves the company from devastating consequences of... his earlier bad ideas.