I asked that in an interview with Netflix, why they were building their own vs open source. The answer I got was "It doesn't work at the scale that Netflix operates" , not quite sure what that means but I didn't press him further.
I work at Cloudflare, since joining almost 3 years ago I've seen multiple pieces of technology get deployed and then replaced because "it just doesn't work at our scale".
It feels kinda weird to say that, but then I see what our DNS servers are doing, or I learn about some customers that struggle to consume their own logs due to the speed at which we produce them, and things that you think are not an area of concern become that... when you have enough traffic flowing through your systems.
Pretty much the only thing that has really stood out is Kafka. Kafka does work at scale.
It's rather rare to see off the shelf technologies (open or closed) that scale to the top 5%. This makes sense because the techniques required at that level are pretty different, and don't apply well to other situations.
So, making an off the shelf product that actually does scale to the top 5% rarely makes sense, since, at best, the customer base will be limited. In the average case, though, the customer base will be almost nil, because of the per-site quirks and customization that are always present.
I experienced this on a weekly basis in the late 1990s and early 2000s at WalMart, where we were centrally managing a network with well over 10 million nodes.
Every vendor ever constantly tried to get in with us, so we had to come up with a triage system or an enormous portion of our time would be spent evaluating.
Even those best of breed would almost never work for us, because of our unprecedented scale (at the time).
In the rare case where an off the shelf product was selected and successful, it had to be heavily customized.
I am of course talking more about 'framework' kinds of tech, or, I guess, things that work at scale. A lot of off the shelf tech was and is used that isn't scale related.
Funny you should mention Walmart. I went to visit them in ~1992 when they were the biggest user of Teradata at the time. Never used Teradata after that, but i'm interviewing with them in the UK next week...
Yup. WalMart was doing big data analytics, mixed in with a lot of real-time processing, many years before those became a thing in the rest of the industry.
If you don't mind me asking, what kind of position are you looking at in the UK next week?
Yes Teradata were awesome at the time. Their secret sauce was the hashed index stuff,which I think was the source of their major patents.
Our PM was not amused when I pointed out that, although hashed indexes were awesome, they did nothing when you were doing a wildcard search at which point you're doing a table space scan. Nobody has thought about this wrinkle...
Re: Teradata: I sat next to that group for half a year, but I had no other direct exposure to it. I did what one might call 'devops' for Network Engineering.
Most (but not all) open source projects aren't built or tested with huge scale in mind, and often you have to do things different enough that it's not worth the effort to change an existing project when you could just build it yourself. Especially if you have a lot of custom environment thing to integrate with. A simple example is projects using MySQL rarely have support for separate read and write servers, but that's a pretty common way to scale out, and plumbing that through is going to be more painful than to do it right in the first place.
The other thing is 'early optimization' when you know millions of users are going to use something, you have to build things right to begin with.
