That being said, scalability should definitely not be ignored altogether.
Yeah. You have to think about what's possible to scale.
You're planning to scale to 1,000,000 or 1,000,000,000 users eventually? OK. Well, what's your service? Are you planning the next Facebook where everybody can talk to anybody? Then you have some challenges you better start figuring out.
Or, is it a SaaS where nobody needs to access anybody else's data? OK, then you can probably simply scale vertically for a while up to $SOME_MILESTONE, and then scale horizontally via some fairly simple sharding or maybe even just database partitioning or something. No need to do that now.
> You're planning to scale to 1,000,000 or 1,000,000,000 users eventually? OK. Well, what's your service? Are you planning the next Facebook where everybody can talk to anybody? Then you have some challenges you better start figuring out.
Thinking this way actually leads to one of the most common failure modes. Specifically the day-dreamer death where the principals become so enamored with a fantasy of what the end-state could look like that they lose focus on the next step to improve their odds of success. This is under-reported because day-dreamers are so common that when they try a startup and fail, it's not really an interesting pattern to report on. Such founders will tend to cite specific reasons that sound better in context (eg. wrong market, product flaws, sales funnel flaws, etc). But as someone who's been in the web startup space 25 years, I've seen it over and over again where founders fail to fully engage with the hard problems right in front of them in favor of working on what's fun, easy or tied to some dopamine hit of imagined future success.
In practice, you should not spend one second thinking about the requirements of 1B users until you have at least 10M. You will need to completely rewrite everything multiple times along the way to that scale anyway, and more importantly, you won't know the product requirements to actually get there until you achieve the earlier milestones and get the necessary user feedback about what's working at each size of user base.
There's a misunderstanding. I may have miscommunicated.
I'm advocating for understanding your business, not for prematurely building things.
If you're in the extremely rare class of businesses trying to build something on the scale of FB or YouTube where that sort of massive scaling is intrinsic to your mission then yes, you need to start thinking about those problems early.
For everybody else, yeah, absolutely focus on product and getting to 100 or 1000 or 100K or whatever users first.
Precisely. It'd be like trying to make an F22 fighter jet right off the bat when you're still in the era of biplanes.
I think people often underestimate the strata of learning, and the essential feedback that gets you to the next level. Or at least that's how it works for mere mortals like myself :)
I'm advocating for thinking about if you'll need an F22 fighter jet someday.
If the answer truly is "yes", that's something you'll need to figure out early or at least plan for.
For the other 99.99% of situations where the answer is "no" or "almost certainly not" then absolutely, you should not be wasting time building an F22 or even thinking about it.
> Are you planning the next Facebook where everybody can talk to anybody? Then you have some challenges you better start figuring out.
Even Facebook didn’t start trying to figure that out until they had to. Famously you needed an invite for a very long time. I’d imagine infrastructure limitations factored into this.
You’d be surprised how many successful companies push off dealing with scaling problems until they absolutely have to.
> Famously you needed an invite for a very long time.
And then it opened to needing a college/university email address. Then it opened to everyone.
I got in with my college email address, and at the time it was considered a feature, not a limitation: a combination of higher quality since people couldn't just make new spam accounts whenever, plus a bit of a status symbol compared to just being on MySpace.
That's exactly right: it depends so much on the context and (expected) access patterns. Sometimes you really do need to plan ahead, but even that's more about not painting yourself into architectural corners rather than specific tech.
What I see also is not just scalability over-engineering, but the same in the solution domain. When you don't really understand your customers needs sometimes there's a desire to create something with ultimate flexibility.
In the days of Xp we used to say YAGNI when people would be overly general. I'm trying to bring that back into fashion.
You're planning to scale to 1,000,000 or 1,000,000,000 users eventually? OK. Well, what's your service? Are you planning the next Facebook where everybody can talk to anybody? Then you have some challenges you better start figuring out.
Or, is it a SaaS where nobody needs to access anybody else's data? OK, then you can probably simply scale vertically for a while up to $SOME_MILESTONE, and then scale horizontally via some fairly simple sharding or maybe even just database partitioning or something. No need to do that now.