kanu666's comments

On: We Put Half a Million Files in One Git Repository, Here’s What We Learned

[kanu666]

Sorry, there is a quote from the article: "Kudos to other members of the Source Control team — Alex Sadleir, Wesley Li, Adam Murray, Matthew Chhoeu — who work on improving git performance at Canva"

So I assume 5 people working specifically on Git performance ;-P

[auscompgeek → kanu666]

Heh, I can see how this can be misinterpreted :P

That's the entire Source Control team (of 5 people) today. Among other things, they work on improving Git performance ;)

On: We Put Half a Million Files in One Git Repository, Here’s What We Learned

[kanu666]

Git staring to break at 200-300 engineers pushing into it. Scaling git would take 200 more :)

[hamandcheese → kanu666]

Everything described in the post can be done by one engineer.

On: We Put Half a Million Files in One Git Repository, Here’s What We Learned

[kanu666]

Thanks for the clarification. How many people work on Developer Productivity in total and what is the percentage from the total engineering headcount?

On: We Put Half a Million Files in One Git Repository, Here’s What We Learned

[kanu666]

Usually it is considered good proportion to start with to be 10% to work on a Developer Productivity and it should increase to 15% as company grow. But regarding source control it really depends on the approach: if you are on polyrepo - then 0 is enough. If you decide to write you own source control system like Facebook or Google, then maybe couple of hundred engineers full time until the project is finished :)

On: We Put Half a Million Files in One Git Repository, Here’s What We Learned

[kanu666]

If you don't have a couple of hundreds engineers who would write a custom file system for git (Microsoft), or who would take an existing source control system and nearly fully rewrite it (Facebook), or who would write a custom source control system from scratch (Google, Yandex, etc) DON'T USE MONOREPO.

Otherwise you are risking to end up in a situation when hundreds of your engineers have to spend tens of minutes every time they need to simply push their code changes.

[jeffbee → kanu666]

Google did not need to write their own SCM to reach the scale mentioned in the article. Off-the-shelf Perforce served that company well enough when it had tens of thousands of engineers and millions of files. They had 11 million files in a monorepo with one 256GB machine which most of you would consider a bare bones desktop computer at this point.

https://www.perforce.com/sites/default/files/still-all-one-s...

[danielheath → kanu666]

I mean, if you have hundreds of engineers, that might be sound advice; if you have 20, you’ll probably be fine.

[kanu666 → danielheath]

Of course, you shouldn't start a project from 10 repos. But the downside is while you live in a single repo all the tooling get tuned to work with a single repo. And then there is no way to go multi-repo when you need to.

So advice would be: start small but think big :)

[bregma → kanu666]

There is no point in making any argument with someone who has the conviction of their religion behind them. Followers of the monorepo religion are no different.

On: We Put Half a Million Files in One Git Repository, Here’s What We Learned

[kanu666]

But that's what those responses are all about: git doesn't scale, so don't use it if you need scale. If there are no tools that scale available, then change the approach so you don't need face this scaling problem at all.

On: We Put Half a Million Files in One Git Repository, Here’s What We Learned

[kanu666]

But in the monorepo you almost never can do the change in a single commit as it will cause incompatibilities during gradual deployment

[mbo → kanu666]

Canva engineer here: we do compatibility checking of interservice contracts (Proto) to ensure that gradual deploys are always safe and can always be safely rolled back.

[compiler-guy → kanu666]

Google does such a thing in its monorepo quite routinely.

On: We Put Half a Million Files in One Git Repository, Here’s What We Learned

[kanu666]

How would you do refactoring over monorepo if you have sparse checkout?

[blackoil → kanu666]

CI/CD build the entire project. So if you make a breaking change in library the build will fail.

[kanu666 → blackoil]

Sure, and what next? So considering that on big projects CI/CD may take up to an hour (in one of my projects it took 4 hours) the feedback loop would be great

[krageon → kanu666]

> on big projects CI/CD may take up to an hour

No, it may not. Perhaps occasionally it does. That is a bug that you must fix - a pipeline that takes even 30 minutes is horrifically slow.

[auscompgeek → kanu666]

One of the nice things about using Bazel is caching builds, avoiding rebuilding parts of the monorepo that are completely unaffected by someone's changes.

On: Monorepos: Please don’t

[kanu666]

Don't mix up downsides of multirepo and bad composition of your microservices