What worked for us was to do two versions of the program at the same time.
Version one had simple functionality. Version two had all the bells and whistles dreamt up by the programmers and the customer while we were working on version one. Those were added to a todo list for version two. They were not implemented in version one.
That kept up the flow of ideas but didn't disrupt the project. It made the programmers feel appreciated while giving us an outline for the next version so our competitors couldn't keep up.
What surprised me was that the whole team got into the spirit of the thing. When someone came to me with an idea, cries of "Version two! Version two!" echoed around the programming room.
We were doing something similar but the result was a mess. Sometimes version 1 needed something that depended on unreleased stuff in version 2. It lead to fragmentation where we developed two implementations of the same functionality.
1/I often discover new things and have new thoughts and ideas along the way of an implementation which hadn't presented themselves previously.
2/I think it's a great way to avoid overthinking a feature or arguing too much for or against it. If nearing a majority consensus (with oneself or team) and there is time to spare, add it to Version 2 -- advantages being point #1 above, and having it prepared should it be deemed worthy to be merged to Version 1. I've heard that having features in the chamber ready to release is something big corps commonly do.
The problem is that as requirements grow, class structures are often kept the same but the 'glue logic' which operates on these structures becomes increasingly complex (you have to handle a growing number of edge cases).
There is a point where the glue logic becomes very complex (and brittle) - At that point, the best thing to do is to redesign part of the system's class structure.
If your components (at all levels of your class hierarchy) are specific about their own behaviors, it means that you can use simpler glue logic to make them work together.
When you need to handle a lot of complex use cases, it's often useful to have many specific classes which share the same interface and can be used interchangeably.
If we get several bugs or the team recognizes that an area is becoming brittle we make a bug that one/two team members work on for the sprint to clean that entire area up. The PO has become used to these kinds of bugs/user-stories and prioritizes them.
Unfortunately, sometimes the simpler system is the one you don't yet understand, and the more complex system is the one you can create on your own using tools you do know. This is the reason we still see homegrown business rule systems despite the fact that they were pretty much perfected them by the late 70s, with several open source implementations as of the late 90s. It is also the reason you still see deadlocks and race conditions in parallel code. And it is the reason you see matrix factorizations using hadoop map reduce instead of MPI.
And when these bullshit foundational design decisions are made, the "improvements" that the engineers make to the system later are almost always a reduction in fidelity to the real world process that is being modeled, instead of a move toward a system that drastially simplifies your system while simultaneously improving fidelity to the real world but might mean you have to read a wikipedia article for a few minutes.
There is a good book about this. It changed my thinking about complexity. The key insight is that if you don't manage complexity you will drown in it. Complexity is OK as long as it really earns its keep. And a lot of complexity is not worth it. (I speak as a person who just spent 20 hours - less than the IRS's estimated 30 hours - filling in form W-8IMY for the IRS, a form which started as a bright idea by someone in congress).
"Conquering Complexity in Your Business: How Wal-Mart, Toyota, and Other Top Companies Are Breaking Through the Ceiling on Profits and Growth Paperback"
by Michael L. George and Stephen A. Wilson
Thanks for this, very interesting, I'll give it a read. I've long wondered if the aversion to complexity in SV is at times pathological, that is, "it adds complexity" is often used as a reason not to commit to building something that would absolutely add long term benefits that outweigh the costs. I guess it's just harder to quantify potential future benefits against immediately obvious complexity, but all the same, I think pathologically avoiding complexity is just as bad as pathological complexity :)
The notion of holding a Code Purge Day (as Quora did) is excellent. Searching out and removing cruft in the code base results in a benefit that's hard to measure but easy to understand. Having more compact code without confusing distractions leads to easier comprehension and ought to make for faster bug fixes.
One of the neat things my company does is give out a monthly award for person who (net) deleted the most code that month. It's in the form of a little rubber duck called the deletion-ducky. Competition can get pretty fierce for it.
The nice thing about it being monthly, instead of one off, is that it keeps simplicity in the front of everyone's mind
Matching passengers and drivers while adjusting prices to balance supply and demand is a complex and hard problem. So is routing questions and answers to the people most likely to answer and read them
I used to work with a guy who worked on the compute infrastructure that CERN uses to sift the petabytes of data they get from the LHC. That's a hard problem.
And that's the problem with web guys, they think the web is all of computing, yield management has been done by airlines and hotels for DECADES. It is very, very far from breaking novel ground at this point.
Now the points he makes about the TCO of a new feature are valid but please, if you're going to brag, be prepared for people to yawn.
This point is quite nitpicky, no? The sentence you quoted could have been removed from the article and the article would have largely been unchanged - my point being it seems dubious to critique an article for a tangential point in the introduction which has nothing to do with the premise of the article.
I think what it really is, is some people just like to complain about "web guys" every opportunity they get. And while your criticism may have been perfectly valid, it is entirely irrelevant to this article.
Not at all; the structure of the piece is, these are all the impressive things I've done, so you should listen to my ideas. It would have been a stronger piece of writing if he'd just jumped straight in. Or, actually had done some impressive engineering.
There's a bit of ageism there too. Über's yield management code could probably have been written by one grey haired guy from United Airlines or Marriott. Instead they got a whole bunch of twenty something's who think they're doing "data science".
Sure you can. For example people need to move around in the game. You can add rideable objects - or not. Usually you don't need to ride something for the game - it's just cool. So this is complexity you can remove.
You can take it a second step - assume riding is required. Do you want various types? Say you ride a horse - do you need multiple colors and sizes of horses?
Or something as simple as allowing the character to change hats.
You can't do what for games? Avoid complexity? You can certainly try! Some bits of complexity will creep in. The important thing is to be aware of the cost and judge it appropriately, don't give low estimates for complexity-adding features. Include efforts to refactor complex stuff in the time budget. This is the same for games as other things.
As a one man army indie game developer I basically can no longer maintain my code-base without going crazy because it's grown to the point where it's no longer a one man job. I knew this would be a problem, so keeping code complexity to a minimum has always been a priority, but it's no longer possible to keep up. Thankfully it's reached maturity as a product and is a fun and profitable game. Many people think i'm 'brilliant' when they see the product and they hear i'm the only the developer and it's a custom 3d engine. But truth of the matter is much of my brilliance was stubbornness and cleverness in keeping things simple and code complexity to a minimum. The problem is you run out of memory in your brain. You forget how systems work. You have to trace back through the code and "reload things". When you have to make a modification.
When I hit my human ram capacity so to speak, i was extremely frustrated that I couldn't make the boundlessly quick progress that I used to. But then you get used to it, and just accept that it will take time when you switch from one subsystem to the other, to load up the relevant parts of the code into your human ram in order to do be able to do the work you want to do.
The simpler and more straightforward the program architecture, the easier it is to jump into and out of and write new code. Of course understanding what your product is and is going to be helps tremendously in making sure you can design your program in a sensible way.... I've been lucky enough to have a firm intuitive understanding of future the design of the game and it's features as it was being formed. (not being at the mercy of clients or bad managers is a wonderful thing)
What i've noticed is a big difference from me and friends with small development studios that accomplish less with a lot more sweat. Is that I don't develop fully featured flexible systems when simple ones will do, I don't roll my own code when good/simple libraries or apis exist. I'm trying to make my product a work of art, and the code is functional and simple. Where often I see other good engineers try to make their code the work of art. Which leads to lots unnecessary code complexity because they engineer systems designed to be future proof for everything, instead of of what realistically will be necessary for the future of the project or projects. A system with 30 components is easier to understand and easier to maintain than a system with 100. More functions, more objects, and more abstractions. Often mean more components and code complexity. Which ultimately means it's going to take more brainpower to get anything done. Which is why it takes teams of 100s to maintain large enterprise software projects where the incremental updates change very little even though the engineers working on it are educated well trained people. shrug
> Which leads to lots unnecessary code complexity because they engineer systems designed to be future proof for everything, instead of of what realistically will be necessary for the future of the project or projects.
This has been a bane for me in my work. I have termed it lasagna code: lots of layers with a little filling in between each. Often, each layer is isolating things for which there is little-to-zero reason to ever suspect that they will need to change. I've even seen attempts to abstract away data model... As if the entire program wouldn't have to be modified if the data model changed. I have learned to design and code only to known requirements. All the extra time saved in design, coding, and maintenance of the easier reduced system more than pays for the small redesigns necessary when things shift.
> I knew this would be a problem, so keeping code complexity to a minimum has always been a priority
I'm in the same position, except that it was never a priority for me. Even if I knew it would create a disaster in the future I prioritized getting things done. Basically I had the game idea and started coding without even thinking about architecture or maintainability so you can imagine what a mess my codebase is.
Still, I regret nothing since without this mindset there would be no game to speak of.
im also a one-person developer (mobile, utility) of an increasingly complex app. i totally get you on the having to load different subsystems into your brain thing. sometimes i will leave a separate, complex subsystem for the next day because i know my mind will be cleared of the current day's cruft, and ready to work optimally.
do you have a link to your project? i think one-person, deep projects are interesting.
In the world of enterprise software it's also easy to overlook the cost of implementation/setup of features on the services teams and customers. Is helpful to define nonfunctional requirements that deal with these ancillary concerns.
Also all of these issues highlight why product management needs to have a working knowledge of technology and the related costs of various trade offs.
I am totally agree with 'Use data to prune unused features' point.
It might happened that some features were introduced years ago based on demand. But It would be not usable for now. Still those features are maintained. I believe those features should be removed which eventually gives you space to work on innovative features for same product.
This article repeats a meme, but is total BS. It's similar to saying "the hidden costs of being a founder" and then stating that if you do ever make a successful exit, you will have a lot of your time wasted by having to select bedlinen and other hidden costs of having made an exit. (the hidden cost to being a founder - you will be forced to work unpaid hours shopping for luxury bedlinen. so consider carefully.) Well guess what: that's not real technical (or founder) debt!
The idea of calling this stuff technical debt is simply laughable. there's no debt. you don't owe shit.
hint: if you code something up in 20 minutes you still don't have to do all that other stuff, you can just ignore it, and throw away the twenty minutes of code if it's not better than not having it. if you become a millionaire you don't have to do all that stuff, you can just ignore it.
Let me put it another way. Say you're an MBA who can't code anything but excel formulas, yet you figured out how to get excel and powerpoint onto the web as a web app (wat). you create something and get 5,000 paying users and raise a $500,000 investment.
Have you created technical debt? No. You're at the same square as if someone gave you $500,000 to spend on developers against your mockups, except that you've validated them as well. There's no debt here. This article had it right:
The people that call this scenario (excel on the web) technical debt, think that somehow the MBA 'borrowed' the web app from a real dev, and now owes it in real development costs. That's a wrong way to think about it. In fact it's a ridiculous way to think about it.
I think you're arguing against people using the phrase "technical debt" as a criticism, but those people are missing the point.
Technical debt isn't a perfect metaphor, but there's an aspect of it I think you're neglecting - the interest payments.
The point about debt isn't that you'll have to pay it back [1], but that you pay an ongoing cost until you do. In the technical debt case, the ongoing cost is opportunity cost - the initial quick hack may make it harder to evolve the code, or you may have to spend time fixing bugs instead of adding features.
As with financial debt, if that ongoing cost doesn't matter - because the payoff outweighs the compounded ongoing cost, or because you can offset the ongoing cost with other sources of cash flow / bandwidth - then taking the debt was the right call.
If you abandon the project (declare bankruptcy) then you never have to pay off the debt :)
If you "make it big", though, someone eventually will want to pay it off. Either you make it big by becoming a successful business, in which case your time horizon extends, and at some point the interest payments compound enough that you don't want to pay them any more. Or you sell to someone else, in which case they have the same problem, and (if they're smart) they should have paid you less because you sold them a liability along with an asset.
[1] Most financial debt has a maturity date, so eventually you do have to pay it back, although not if your lender is willing to indefinitely roll over. Technical debt generally has no maturity date, so you really don't have to ever pay it back!
nothing personal, but turns out you happen to be totally wrong. :) we can just look through examples until you realize this.
It is like saying if I sell you a luxury mansion in the Hamptons that comes with nice hedges, for $1, I've really sold you an $800 interest payment, since you'll want to keep paying a gardener to trim the hedges, since if you don't it will hurt the price of your property long-term. you want to retain the value.
Do you see how considering the $1 as coming with an $800 interest payment (until you replace the hedges with gates at even greater cost) is a totally wrong way to consder what has just happened?
The fact that someone (you) will want to pay something doesn't make it anything close to debt. It's an unhedged (heh) call option.
I think I see where you're going wrong here. We need to put some numbers to it.
Suppose you can make $20 if you spend $15 adding a feature. Obviously you should do it, right? You'll make $5. So you do.
Now the customer comes back and says the new feature doesn't work in some edge case and they won't pay the $20 unless you spend another $15 fixing it. At this point the original $15 is a sunk cost that you can't get back and you're presented with the original scenario again: Spend $15 to make $20. So now you've spent a total of $30 to make $20. This can repeat arbitrarily many times, putting you even further into the hole each time, until you recognize that hidden future costs mean that you're actually spending far more than $15 to make that $20.
If I can choose to pay full price for a luxury mansion or pay $1 for the same mansion, of course I should pay $1. That choice is not usually available, either in real estate or in software. More likely, I'm choosing between a studio for $200k and a 4 bed for $1m (adjust for regional house prices), or a cheap fixer-upper vs an expensive new-build.
The point of the debt analogy isn't to think about which parties are involved in the transaction; it's to think about the various kinds of costs involved in a decision, and the timescales over which they are paid.
From that point of view, if I can't afford the $800 hedge maintenance, then I shouldn't buy the mansion, even for $1. The strict analogy with debt breaks down a bit because I'm not paying you the $800, but it's still an ongoing cost I have to bear.
Edit to clarify, I'm still not saying debt is always bad. If I enjoy DIY and expect to have spare time, the cheap fixer-upper might be a great investment.
While I agree with you that the notion of 'technical debt' is being overused, I don't think the article is talking about hidden costs as technical debt, but is instead about how product and code complexity incurs hidden costs, some of which but not all can be construed as technical debt.
Code complexity is a good example. A part of a codebase can be complex (sometimes the problem that is being solved necessitates a complex algorithm / codebase) but yet well written. That does not incur technical debt, but there is a hidden cost to maintain it given it's complex. On the other had, a complex code base, that was done hackily in a few hours just to get to the solution without any thought for maintainability or has no test coverage, is technical debt, because the person 'borrowed' time amongst other things with the hacked up codebase, which in the future he/she must pay down by refactoring / writing tests.
I think simplistic was meant to imply that it wasn't really a solution, just good enough for you.
That said, you can say the same case for your matrix: if there's a simple solution to a complex problem that actually solves it, is it really a complex problem?
Sure. "Complex" problems are problems with many facets/moving parts (I recommend watching the Rich Hickey talk "Simple Made Easy"). These problems can in some cases have simple (as in, few facets/moving parts) solutions, although those solutions may not be easy to implement.
On the other had, a complex code base, that was done hackily in a few hours just to get to the solution without any thought for maintainability or has no test coverage, is technical debt, because the person 'borrowed' time amongst other things with the hacked up codebase... Yup, usually because some sales/marketing/managing idiot had to have something look nice at a show and then wonder why an engineer is wasting time stabilizing the code base before he can give it to a customer. "It worked at the show...WTF?"
>On the other had, a complex code base, that was done hackily in a few hours just to get to the solution without any thought for maintainability or has no test coverage, is technical debt, because the person 'borrowed' time amongst other things with the hacked up codebase, which in the future he/she must pay down by refactoring / writing tests.
This is simply not true. It is exactly the same that a non-developer who managed to get an Excel spreadsheet on the web 'borrowed' time from a real developer if it takes off. How can you borrow time from a non-existing person? This just doesn't make sense. It's just an unhedged call option.
Let me give you an offer for your services. Suppose I ask you for a quote for a very simple database app where employees can punch in and punch out with a password, and it tracks their daily time between the two. Nothing exists including a mock-up. You give me a quote. Now suppose that some genius actually got excel online witohut password authentication, in twenty minutes, they can put in a URL and click a button for a timestamp. That's what we've been using three weeks.
Does the existence of this change your offer? Yes: it reduces it, because you can see exactly what we'r ehappy with. (Unless you're trying to take advantage of the shop for being non-technical and therefore increase your quote; in this example we can just hide the existence of it from you. But this is a pathological case.)
So how can you call it debt, when in fact all it is is a slight asset (we got three weeks of utility from it) that took a non programmer half an hour, and reduces the price you quote to write it properly by a certain amount?
It's not debt in any sense of the word. You don't 'have' to pay it, any more than you 'had' to pay it before the hack.
In the other poster's example. if the salesperson sold something under cost, he sold it under cost. But if he sold it over cost, then his demo didn't become technical debt of the actual cost, it was just worth 20 minutes by a non-programmer, exactly as though he hadn't done it. Except he did do it, becuase it helped him get the sale. It was an option.
A third example would be saying that any time I buy something crappy that I expect to break, I am creating technical debt. For example if I buy a crappy screwdriver that will break on my third project, I've really signed up for the debt of a real screwdriver. Huh. This is not a very sane way to think about it. (It's actually a good analogy for technical debt, if I only bought it to produce the first two projects, and wouldn't have started on the third if the first two didn't get traction and get sold, or maybe even result in an investment.)
Technical debt isn't a perfect analogy, but people understand it. It's like explaining that programming is just writing a recipe for the dumbest chef in the world--it's a good analogy, but doesn't capture the complexity or techniques of programming. Which is fine, because almost everyone is familiar with the idea of a recipe, and your explanation is for someone who probably thinks programming is this magic thing you do that makes Facebook pop up out of the ether!
An analogy that isn't understandable by its target audience is useless.
Given that most programmers' idea of finance is of some kind of magic that happens that makes money pop up out of the ether, you need to use the appropriate analogy.
And then your teammate then takes a week to build an API that's only compatible with your hacked out code you wrote in twenty minutes. Now you've wasted both your time and her time when both need to be re-written.
Yes, consider what has happened here carefully. Maybe she shouldn't be allowed to, and should consider that 20 minute code as not existing. (Because on day 1 of her week, she could have written it properly.) That does not make it technical debt of 1 day. It means it didn't actually exist.
Maybe this is really a variation of the sunk cost fallacy or something similar, where if something seems to exist in some form, then people can't get their minds around the fact htat only 20 minutes of it actually exists and the rest simply doesn't and needs to be written if they want it to.
EDIT: all these downvoters are wrong. You say "now you have both wasted time" but the 20 minutes wasn't debt. It was the cost of the call option. For example in the other poster's example where a salesperson sold some code that didn't exist, his non-programming version of it wasn't debt toward the final thing: it was the cost he paid for the call option of being able to sell the whole thing. Hopefully he sold the option under cost :) his payment on the option has nothing to do with it.
Maybe she shouldn't be allowed to, and should consider that 20 minute code as not existing. (Because on day 1 of her week, she could have written it properly.)
Completely agree with this, and I wish it was said more often.
Prototypes are a valuable tool for testing hypotheses, but you have to discard them when testing is no longer all you're doing.
That means the key to responsible use of prototypes is to make sure you remain able to discard them! This is why shipping the prototype is so dangerous - because now you're locked into either maintaining the prototype, or a rewrite followed by a tricky, risky live migration.
Version one had simple functionality. Version two had all the bells and whistles dreamt up by the programmers and the customer while we were working on version one. Those were added to a todo list for version two. They were not implemented in version one.
That kept up the flow of ideas but didn't disrupt the project. It made the programmers feel appreciated while giving us an outline for the next version so our competitors couldn't keep up.
What surprised me was that the whole team got into the spirit of the thing. When someone came to me with an idea, cries of "Version two! Version two!" echoed around the programming room.