This is a good write up of why you shouldn’t give away the house when you open source things. The issue was really this: Earthly being open-source, Earthy Satellite users were already seeing the benefit from 95% of Earthly CI.
I’m a huge fan of open source, however, if your business model includes an open source model - you need a differentiator. Beyond blazingly fast(tm). You need a reason for people to offer up their credit cards, or better, their accounting PO process. You need a feature you withhold or a service you provide from the open source offering to make it appealing.
GitLab restricts CI/CD to paying customers. Travis/CircleCI limit on build time/credits unless also open source. Azure DevOps is the devil. ArgoCD is complicated. GitHub actions are nice if you have hardware to run the runners. Swarm’s of Jenkins is what most are used to in the enterprise.
As a former Director of DevOps, my immediate question will be what features make me want to buy vs host myself. I’m technically capable. Why should I buy with you when I can run my cloud and my DevOps pipelines for my workforce the way that makes the most sense for my business? Help me make the most sense.
*EDIT* I'm not suggesting withholding features that are critical for the software to operate, but withholding business features that provide support or enterprise-level integration. Even making it tiered so the power users pay less because they support it more themselves?
I'm not sure feature withholding has traditionally worked out well in the developer space. The churn rate for tooling is so high that if Earthly were to intentionally cripple their product I'm sure the majority of developers would just move on to other inferior, but free, options (eg. taskfile). Maybe those that remain could be converted successfully but it's quite a gamble.
A great turn around story in recent times though is probably Docker. But that required them to abandon strict open source and make controversial license/product changes.
For the open core model to work, the OSS product should be standalone and featureful. If users feel like it's just used to advertise a commercial product, it doesn't work. And the commercial product should offer value-added services and features that are not strictly required for the base OSS product. These should be features directed towards advanced or corporate users, and preferably something that competitors lack, or don't do a good job of.
It's very tricky to balance the open core model correctly, which is why not many companies get it right. You need to invest equally, if not more, in the OSS product, and make sure users are happy with it before you even mention a commercial addon. Grafana is another example of companies that understand and do this well.
> I'm not sure feature withholding has traditionally worked out well in the developer space.
I think it's worked out well for Sidekiq (https://sidekiq.org). I really like their model of layering valuable features between the OSS / Pro / Enterprise licenses.
The trick is you have to do that very early on. Nerfing your open source offering by making some existing features paid-only is always going to get some backlash.
In this particular case, developers could also move on to a superior and completely open alternative (Nix), so making even a semi-closed alternative with much fewer people contributing etc. was already a hard sell.
Seriously, is there an email distro or something to notify Nix users when there's a new post to go troll? It's not suitable for 90%+ of users, which seems a far cry from superior.
It's just very noticeable when half of what the industry seems to spend its time on is solved problems. You and all the others have to learn why touching the hot stove is a bad idea the same way most children do. It's frustrating to observe.
It's fine to criticise Nix for its various flaws, but any suggestion of an "alternative" that has learned nothing from it is just a collective waste of time.
I’ve recently had to learn some nix to build an IHP app (Haskell framework). Most hosting providers require a dockerfile and/or image. I couldn’t get nix to build an image on time within the hosting providers’ CI. So I gave up.
For any other project, I can spend half an hour max to get a decent deployable build. The trade offs is Nix doesn’t seem to work for me.
Why are people ignoring you though? Because if it really was the promised land we'd all be there already. Must be something important y'all are missing about the good old fashioned hot stoves.
> You and all the others have to learn why touching the hot stove is a bad idea the same way most children do. It's frustrating to observe.
You say that while your ecosystem still can't get away from cutesy childish terms like "flakes" and "pills". Newsflash: nobody cares. I want "a package" and "a system".
I am not impressed. Before you look down on others make your own thing look uber-professional and then uproot any and all nerd-friendly marketing from it, now and forever. "It was in fact just a phase, mom" -- that's the aura that Nix should radiate before I give it a second chance.
Nix is supposed to solve a very serious problem and if it actually succeeds in that technical goal, the next step is for them to act like it. Very often you're marketing to CTOs or other tech executives -- working programmers are only one chunk of your audience. So that ecosystem has to do better -- starting with you. Your two comments here are basically a text-book argument against using Nix.
I've been a CTO and VP of Eng and I have skipped projects that can't sound professional. I know I have missed out on good stuff; I am very well aware. But we all have only 24h in a day, and only so much energy. We have to filter. One of my filters is: perceived / marketed professionalism.
(To be fair to all sides, I like the nixos.org website. I don't like practically anything else about the ecosystem though.)
> It's fine to criticise Nix for its various flaws, but any suggestion of an "alternative" that has learned nothing from it is just a collective waste of time.
Let's start with that one: nobody needed one more programming language. Nobody will "see the light" if every new technology requires one more knowledge investment. Somebody must tell you that programmers, DevOps, full-blown Ops, CTOs etc. are already overworked and already have to know and regularly catch up with way too much.
If Nix is aiming to help people work less on very annoying problems then it should, you, actually make them work less.
Nix's marketing is a complete mess. "We want you to think less about A and B so here, now learn, X, Y, Z and you should still know A and B -- but it gets better after, one day!".
Nah.
It's fine that you found technology that you love but your bias is very clearly visible and you lack the introspection skills to see that Nix is not that impressive.
Give me a `curl ... | bash` installer plus instant onboarding plus zero upkeep (an oft complaint about Nix btw) and I am sold. Anything less than that, to me you are just one more voice in a huge crowd where everyone screams "use my thing, it's amazing!".
Differentiate yourself and people will flock to you without you having to lift a finger. The fact that this hasn't happened yet is a strong hint that Nix fans are refusing to notice and take a lesson from.
I tried to write a serious response to your comment, but as I got further into it and you got further into personal attacks and away from substance, I decided to delete it and I'm instead leaving you with this:
You should get off your high horse. Nobody cares about you having been a "CTO" of some random venture-backed company, it doesn't give you any credentials which you can fling at people to "win" debates online. 80% of the people on this forum have some useless title like that in their CV. You being able to write lots of words about irrelevant superficial topics doesn't mean that you have anything useful to say, even if you've previously been led to believe that these are equivalent.
I don't care where you flock. I publish all my Nix related work for free, I don't have a personal stake. Your decisions are your problem.
The open source version has to be good and valuable enough for you to bring it in as a dependency in a basic setup, and then the paid versions provide support for larger scale setups where the requirements are a bit different.
There are many ways to do that, depending on what you're offering. Limits and quotas are easy enough.
plugins, hardware, hosting, support, there's a number of ways to offer a service for an open source project. Open Core models come to mind. Another is not having enterprise SSO as part of the core. Or not having clustering. Something that would make a VP think twice about adopting your open source solution vs using you as a vendor.
I know these are examples, so I don't want to pick on you too much, but I have one nitpick: Reserving SSO for enterprise customers is awful. Please don't do it.
See https://sso.tax/ for details but I'll quote this from it "SSO is a core security requirement for any company with more than five employees"
If a company requires SSO, they'll surely have a budget for paying for the products and services they use. The majority of OSS users won't have this requirement, and likely won't have the budget to pay for commercial features. So it doesn't make sense to give away a service that has a maintenance cost to everyone, as long as it's fairly priced.
As others are pointing out, once you've grown big enough that you can't trust everyone with the keys go the kingdom: enterprise pricing is for you.
SSO Tax is a fun meme, but it misses the point: Whether it's 5 people or even 3, you've scaled your business enough to hire people who aren't in the inner circle.
That's a fair place to consider you an enterprise.
Not sure what you're talking about. It's not like without SSO everyone in the company has to all use the same password. Services allow people to have separate users with separate privileges.
It sounds like you're unaware of why SSO is considered a security feature at all them, but it's covered right on the site: https://sso.tax/
It's to allow centralized access management. Stuff like firing someone and revoking their access from one platform instantly, instead running around and changing permissions in every tool manually. Or ensuring people in department A can't be invited to some platform for people in department B in order to limit information access.
SSO tax is predicated on the idea that the moment you outgrow the informal arrangements and liberal access, you're really a business. Seems pretty fair?
> It sounds like you're unaware of why SSO is considered a security feature
Technically it's an anti feature...
The "feature" you are talking about is really identity management not sign on (which, btw, users have different identites, outside of the scope of a single company).
At its core it's just a delete function for a username. Putting that in a script with http access should be enough (and not put behind a ridiculous price tag).
Separate services otherwise are enough, no need for SSO.
> At its core it's just a delete function for a username. Putting that in a script with http access should be enough (and not put behind a ridiculous price tag).
Can you explain more about how this works for SSO?
Let's say I'm trying to launch a product (which I am), and want to have SSO as part of the package (which I do), how do I implement what you just said?
I looked into SSO services and they're bloody expensive, so if I can cheaply do it myself, I will.
The comment about being a 'toy service' is disingenuous...
The comment about customers is possibly valid, the difference entirely your use case. If you consume services as a corp, and your goal is to be able to decouple any employee quickly, then you do not need SSO, assuming each service you use allows you the option to delete by username (with proper auth, ofc).
SSO is only needed when you want to give a customer federated access (ability to assign roles without activation on their part). For an internal corp it is largely unnecessary and a convenience thing not a security one (it is much more secure to require different passwords for every service versus a global SSO credential that gets you everywhere).
If your "customers" are your own employees, then this is your decision not a use case or a business requirement.
You're barking up the wrong tree: I communicated how SSO's value prop is defined by essentially the entire SaaS industry... if you have a problem with that value prop, Okta alone is 13B worth of short potential for you once everyone realizes they could just use a script with http access.
Let me give a contrarian argument. I dont think it is awful to have sso reserved only for enterprise customers. I think enterprise customer who are going through compliance will definitely have to but into that tier therefore.
If you think you need SSO and but you do not have the budget for upgrading to their enterprise tier, may be you are not the right customer that they are targeting to sell. It is not you, it is them.
> I know these are examples, so I don't want to pick on you too much, but I have one nitpick: Reserving SSO for enterprise customers is awful. Please don't do it.
SSO is expensive, adding maybe $10/user/month if the team adds in a purchased solution (passing the cost directly onto the client).
If the customer doesn't want to pay for it because they aren't an enterprise, you can't blame the supplier for not supplying it at a lower price than $10/user/month.
I am going to the competitor. I am paying for the software, but I don’t want your “enterprise tier”, which is really designed for 1000+ employee companies.
BTW, I did just this this morning. The AE came back to offer the enterprise package for the price of the mid-tier package.
I see, I see. The world isn't fair. We have 5 employees and we want SSO but don't need SSO. So we'll go without it. It's annoying for us but it allows your business to exist because enterprise needs SSO.
Co-founder of BoxyHQ here - We've crafted an open-source enterprise SSO because we firmly believe that robust security shouldn't be a privilege limited to large organizations. Ideally, essential enterprise-level features like this should become commonplace for all.
While we acknowledge the reasons behind SSO being in the enterprise tier, we're all on a collective journey to enhance our security measures. Open Core models are indeed a good option (my preference), yet the dynamics vary across solutions and industries. It's up to each of us to explore, experiment, and discover what resonates with our market. In doing so, we can foster growth while maintaining our commitment to supporting the community in the long run.
Well stated. I was merely showing how one could implement an open core strategy. I’m a firm believer in SSO for all (and not limited to social platforms or GitHub).
I think a nice compromise is the CircleCI/Travis model of providing limited build credits/hours. If you’re a small shop or single dev, you can use the core product for free, with the hope that you’d convert to a paying user if your business grows past the free constraint.
I’m not claiming it’s easy. Just that withholding features while still offering free tiers is a tried and tested business model that has demonstrated a lot of success for a lot of companies.
It isn’t the same principle though. Slack and GitHub can sell data to make a profit etc but the problem with open source is people can just host it themselves meaning the creators don’t even have that option.
Plenty of organisations prefer to have managed options and thus pay for SaaS, even from open source vendors, rather than hosting their own.
Also you’re getting too caught up in specifics and missing the broader point. You want open source examples? How about Nexus, Redhat, nginx, prettier much all of Hashicorps offerings, basically half of why AWS resell.
The real issue with commercial open source isn’t restricting access to customers, it’s restricting access to resellers. As demonstrated with the examples above.
And honestly I can't even imagine ever using the public runners because they're permissionless. It's way easier to have them live in your own network and doling out permissions via tags that get picked up by different runners with different iam roles.
When it was an open source project, yes. You are correct. Ever since GitLab went commercial, it's been behind a paywall. Apparently you can run runners on self-managed CE so I'm going to look into that.
(GitLab employee but familiar with this because I use it personally as well)
You can always connect your own runners to GitLab, whether SaaS or self-hosted, and whether free/CE or premium/ultimate/EE. GitLab SaaS also comes with free use of the central runners up to a quota. I'm not sure about the history here but as far as I know this has always been the case.
Circa 2013? I remember vividly implementing gitlab at a certain cable company I worked at because they were too cheap for GitHub. I remember sometime in 2012 some merging of worlds of GitLab CI and GitLab.com, it was around that timeframe until 2015 or so when the unofficial runner because official and CI and Repos were merged? You work there, ask the oldest grey beard if this was the case. I could very well be mistaken and have false memory.
you might be right, especially in that time period new features were often EE only initially and then merged into CE. When CICD was first created it was its own software project basically, but it may have required APIs that weren't merged into CE yet. To be totally honest I forgot that CICD was around that early on.
GitLab is still an open source project (the core of it is MIT licensed) and using your own runners has always been an option. Also, free accounts on the SaaS version get free CI/CD minutes as well.
On Gitlab.com you need to input a valid credit card once in order to validate your account, but you can still get 400 minutes on shared runners for free.
This is used to combat abuse and crypto miners, they don't actually charge the card.
You can also connect your own private runner to either the saas or self-hosted gitlab-ce for free, you just have to provide your own runner.
I know this is controversial, but I wish commercial "source available" were the more mainstream / not-shat-upon approach to services, rather than "open core".
I find it very frustrating to not be able to read the code or contribute bug fixes or self-host anything outside the "open core", and I have no need for any of the things that distinguish "open source" licenses from "source available" ones.
> I have no need for any of the things that distinguish "open source" licenses from "source available" ones
This strikes me as short-term thinking.
If you've ever used OpenSSH or MariaDB, say, then you've benefited from the right to fork, granted by Free and Open Source licences. [0][1] There are of course thousands of other examples.
This is what the FSF calls Freedom 3, [2] and what the OSI's Open Source Definition calls Criterion 3: Derived Works. [3]
Needless to say this kind of thing typically isn't permitted by source-available licences.
I mean, I'm very glad that open source (and also free) software exists. I don't think there should only be software with source available licenses, and I certainly admire and appreciate the giant amount of foundational work that has been done under those licenses.
But I think it comes with its own trade-offs. For instance, I might say that preferring open source is short-term thinking, because projects can become both critical infrastructure and essentially un-funded, and use one of the same examples you did, of openssh.
And I lament that people seem to be less critical of entirely opaque proprietary software than of source available software. I think if the only viable choices are nonprofit+open and commercial+closed, then we are going to end up with more closed software that is of higher quality than its open alternatives.
In general my interest in this topic flows from dissatisfaction with funding models for open source software. I really hate that so much important software relies on under- or even entirely un-paid labor to create and maintain it. I'm not under illusions that alternative models are a panacea, but I'm also not satisfied with the status quo. I get paid well for my work, and I hate that so many people whose labor I rely on aren't getting paid well for theirs!
> I might say that preferring open source is short-term thinking, because projects can become both critical infrastructure and essentially un-funded, and use one of the same examples you did, of openssh.
I don't follow. All told, OpenSSH is still going strong, and it owes its existence to the right to fork.
The FOSS model has a good history of software project longevity, especially compared to, say, proprietary software developed by start-ups.
More generally, the rights granted by FOSS licences are a good (albeit imperfect) protection against software turning user-hostile. To that end, a FOSS licence might not be sufficient, but it seems to be necessary.
> people seem to be less critical of entirely opaque proprietary software than of source available software
The Free Software Foundation used to consider semifree software to be its own subcategory of non-Free software, but they no longer make the distinction. [0] This makes sense given the organisation's goals - they don't want to be seen to be giving partial credit to licences that, while not as bad as some others, still deliberately deny the user the '4 freedoms'.
I agree that non-Free-but-source-available is better than non-Free-and-source-unavailable, but there are good reasons the Free Software and Open Source software movements have the principles they have.
> my interest in this topic flows from dissatisfaction with funding models for open source software.
Agreed. It's frustrating that so few people or organisations think to contribute to FOSS projects, even ones they deeply rely on.
There are ways to monetise aside from patronage/donation, such as paid support and paid managed services, but unfortunately it seems these are rarely successful in generating appreciable income.
> I don't follow. All told, OpenSSH is still going strong, and it owes its existence to the right to fork.
Sorry, this was too implicit. What I was alluding to was the period of time when openssh had some showstopper security issues, and everyone realized it was being maintained by essentially one person with essentially no funding.
I don't want to downplay that the copyright owner of source-available software taking it in a bad direction or stopping maintenance or going out of business altogether is a big risk of that model (and of fully proprietary software as well). It totally is, and it is definitely something that the ability to fork solves.
But my point is that it's a trade-off. Both models have failure modes. And they both have (different) imperfect solutions for them.
> Agreed. It's frustrating that so few people or organisations think to contribute to FOSS projects, even ones they deeply rely on.
This is probably where our fundamental difference in philosophy comes from. I don't think it is at all the case that people don't "think to contribute to FOSS projects", it's that the incentives don't work out in favor of it, for most organizations. And I don't think the answer to that is to wish for altruism from organizations and free labor from individuals.
> There are ways to monetise aside from patronage/donation, such as paid support and paid managed services, but unfortunately it seems these are rarely successful in generating appreciable income.
Yeah, as I said, I find all of these solutions deeply unsatisfying. Patronage/donation has the "relying on altruism" issue. Paid support misaligns incentives, because if you manage to make perfect software, you've also managed to put yourself out of business, whereas if your software is difficult to use, that makes it lucrative. Paid managed services would be great, if Amazon couldn't just slap your code behind an official AWS service that 99% of people are going to use instead of your also-ran; but keeping them from doing that requires the restraint on commercial re-distribution that everyone seems to hate.
Stallman was right. I think it's inevitable we'll see the next generation of developer focused tooling go in the direction of AGPL or source available (eg. BSL). Ironically enough Earthly started off as BSL.
I feel almost certain someone could wrap ArgoCD into a product if they handled the 'patternization' of applicationsets. D2iQ already do it with Flux, but we bailed on D2iQ before we even got to try it.
ArgoCD is almost worthless if your DevOps teams are IaC wielders. Honestly, just a bash script to push a container to the pods is all that’s required for DevOps deployment. A build pipeline should be get artifacts -> put artifacts in container -> push container to registry -> trigger a pull on your pods. If you didn’t have a standard cloud architecture, I can understand the frustration and reasoning behind looking at ArgoCD. If you have a standardized cloud practice, it’s moot.
AzDO is "decent", depending on your needs, but it's in an unfortunate zombie state where Microsoft is supporting it just enough to keep it stable and keep certain enterprises happy and consistently insisting that AzDO has a roadmap and is still beloved, but it is very clear that all of the actual resources are going to the GitHub side of the house in 2023.
(One of the most recent signs of this crazy zombie state that a recent feature for AzDO was given the absurd only Microsoft could do brand name "GitHub Advanced Secure for Azure DevOps", and it is indeed a bundle of GitHub features several years old at this point finally provided to AzDO users. Other signs include all sorts of AzDO libraries and roadmaps and related repos openly hosted on GitHub with last updated dates in 2019 and 2020.)
From an outsider perspective, Microsoft probably has a lot of sentimental and internal engineering reasons it doesn't want to truly wrap up so much of its operations in North Carolina (AzDO's ancestral home going back to early days of TFS), but the writing seems to be on the wall that what's left is a skeleton crew mostly working for GitHub full time and then sort of applying things part time back to AzDO. It doesn't seem to me to be sustainable long term, and every day it feels more like an Old Yeller situation where Microsoft is just prolonging the inevitable and making the pain worse for everyone involved.
Ugh, they’re busy bringing bad project management to GitHub, in a quest to make it suck as much as all the others.
The old issues + tags + milestones was perfect. Now it’s the same needlessly-heavy thing as all the rest, with their reimagined “projects” thingy.
But then again I’ve not once seen a PM embrace any version of GitHub’s project management, with the only explanation forthcoming being “it’s confusing for non-technical users” (fucking how? More confusing than Jira or Asana? No friggin’ way) so maybe they have to shit it up and make it hard to use and easy to get lost in or miss info in, to get any traction with PMs.
It does seem a case where Microsoft dogfooding GitHub may be actively making it worse. They've done several BUILD and GitHub Universe videos "How Microsoft Uses GitHub to Build [X]" and their PMs certainly need all those complex features to track stuff and those videos are the closest to training materials on how to use some of it, and it is definitely a lot going on.
It definitely seems one of those sorts of "every system PMs touch is doomed to become unmanageable complexity" laws and GitHub "Projects" does seem to be exponentially closer to being "Jira 2: Electric Boogaloo" every day.
This. AzDO will die. It will be replaced entirely by Github. It originally was only designed as a way to get people onto Azure using their already-existing MS partnership to give it away for free. The only purpose, drive adoption of Azure Cloud. In runners, resources, stickiness, and preventing abandoning ship.
As I said the positivive sentiments I heard came from developers, and TFA is about CI (rather than boards etc. that JIRA handles, so is there maybe anything remotely positive in how AzDO implements CI, git integration, testing, etc? Or is it just rotten to the core?
Hard to imagine someone would hate it more than Bitbucket Pipelines or, god forbid, Bamboo. Buuut I’m more than open-minded if need be.
AzDO Pipelines was directly the first draft of what became GitHub Actions, so technically it does have a lot to recommend it, especially if you learn the Multi-Stage YAML approach which does sort of directly feed into GitHub Actions for the eventual day you decide to migrate.
I think the main competitor is gitlab and GitHub, it’s very hard to compete with them. It’s not competing with earthly OSS. Personally the reason we haven’t used earthly satellites or CI is because we can’t use them inside the corporate firewalls, nor does it integrate with our existing CI tools super well, e.g. gitlab.
We use it and love it but it’s missing important enterprise features like that and SSO.
This is why I don't believe in businesses that offer both open-source and a paid cloud version. Their business goal is literarily: make sure the open-source version is not so good and easy to use/install/maintain, so that people do need to use our cloud offering.
I mean that node (not js) dashboard to build pipelines and make things look logical instead of writing 4 lines of bash. I'm not a fan of anything that adds complexity. ArgoCD/WF/ALL ARGO... is just a pretty dashboard over cli tooling.
./deploy.sh <repo>@<version> <pod-name>
This is the contract you should focus on. Whatever you need to make ./deploy.sh do this, do it.
Isn't ArgoCD more of a GitOps tool? The pretty UI is mostly secondary to its main purpose for me, which is to keep the declarative "truth" in source control, and have ArgoCD be the control loop that keeps the cluster in sync with that truth. Accidentally nuked namespace? No worries, ArgoCD's (or whatever alternative, like flux) got your back!
Exactly this. The critical difference being who deploys and when/how your resources are deployed - a human, or a machine.
Having git be the source of truth of your production environment is a blessing and a curse which is directly related to the maturity of the deployment system.
If you have low confidence in your deployments (as in, you don't deploy very often and don't have full e2e tests & monitoring), GitOps is nightmarishly scary compared to classic, battle tested Ops team CLI scripts.
I don’t accidentally do anything in production. Sorry. Nor do my teams or any developers who have access to my clouds. These kinds of failures don’t happen in my world. You will not have a mutable environment outside of data storage. Period. Nuke all you want, it will repair and redeploy itself.
I used it at a Fortune 500. I’ve used it at a Fortune 100. I’ve used it in a small startup. I stand by my statement. It solves problems self inflicted due to crappy cloud architecture and mutable environments. Running a simple deploy script means your DevOps pipelines and cloud are mature enough to handle rollbacks, failures, blue/green deployments, and any migrations applied after a data snapshot (whatever that looks like in your architecture).
Very weird that you have such an strong opinion of having some self written deploy script over a argocd setup.
ArgoCD is super easy to setup, has a declarative setup and is a great platform tool.
I introduced it to big systems and i really can't imagine how we would struggle if we would deploy those infra components with some bash scripts.
Could you be more specific?
Like assume you have some monitoring stack (kube-stack or grafana etc.) and either as a helm chart or as something else: What do you do when a CRD needs to be replaced? You write it in your deploy.sh script through some bash magic? Handling error cases in there?
What do you do when you want to see what you have deployed? Like checking if your test setup is rolled back? You always use kubectl to check some helm values? Or do you always annotate all pods with git commits which you then c&p?
A basic ArgoCD declarative setup takes perhaps a few hours if you do it for the first time, than you can give teams access to namespaces, they have log access, exec access if you provide a platform that is, in my opinion, quite a big advantage over just letting them use kubectl. Most developers i know do struggle with kubectl.
And for my own infra ( i also even setup argocd at home ) the dev cycle is much faster for me.
What "crappy cloud architecture" problem are you talking about specificly argocd apparently solves?
How do you test components? Like we have a test cluster and everyone uses argocd to just switch to a dev branch for a few days while testing. This works really well and you can switch back to main in argocd. And due to Argocd App Def you can even see all apps out of sync due to testing.
In a small business (read: startup), `deploy.sh` is all you should have. When you go on vacation and it breaks, I just need to read a bit of bash/python/php/whatever-script-you-use.lang and fix the broken stuff.
I introduced ArgoCD to a project with 20 teams and the acceptance from developers was very high.
What they had befor was a terraform based pipeline (i would argue similiar to a deploy.sh) and they struggled with those scripts hard.
ArgoCD with dex is easily integrated in your auth system and you don't need to write some scripts to handle issues like login errors, timeouts etc.
I have seen plenty of bash scripts which do exactly what they should do with 0 error handling and it was always a pain in the ass to analyse and fix those scripts.
From my experience hosting our own instance of Gitlab + few CI/CD workers was few weeks of initial work (integrating with company's LDAP etc.) then around one ops-week per year worth of maintenance over last few years (50+ developers, dozens and dozens of projects).
Before that we had just "a git server + jenkins + some slaves" that took even less maintenance (althought I'd imagine more fumbling on dev side with job config)
Earthly isn't cheaper than any of those options and none of our clients use it (and we occasionally have to work on client's gitlab instances).
You just described a great starting point for a small shop, I would recommend this route 100%.
Initially your CI/CD needs to be close to the code. It needs to be close to developers. Not a DevOps department. It needs to be almost forgettable. Include a config in your repo, bam, welcome to the pipeline.
Where this breaks down is when you don’t have LDAP, you have Azure IDP + Amazon’s Cognito + Keycloak 3rd Parties + SAML (barf) + Form Login for contractors + JWT/OAuth for services.
This is the inflection point where you need DevSecOps + Architecture + Cloud Engineers to ensure everyone’s playing nicely. Enjoy your pods. Let us know when you need that VPN tunnel to your customers backend because they don’t trust SSL.
There are actually authentication providers that will pretend to be LDAP server for your apps, regardless of what auth backend you use.
But yeah, once your clients are big corporations you'd at least need to know how to implement some kind of cloud mess of authentication provider(s), still, doesn't need you need to infect your own stack with it.
That is not at all the conclusion the author draws from this experience of his. Here's more of the paragraph, you quoted:
>Earthly being open-source, you could already run your own Earthly remote runner (since we use Buildkit underneath, it was essentially a remote Buildkit), connect Earthly to it, and get similar benefits to Satellites even before this was a commercial offering. And so people did this on their own, hosting it in their own environment, without us managing it for them. Once we had this packaged up in a managed offering, people were flocking to it, mainly because they did not want to manage remote runners on their own.
The author blames the failure on not validating what the were building was what clients wanted.
Not to rain on the parade here but this is literally a copy-pasta product.
Jenkins, Google Borg, Cloud Foundry, Concourse Pipelines...and surprise surprise when you look at where he came from...Ex-Google, Ex-VMW, RabbitMQ...
If OP wasn't in and around the source of all these tools above then they were at the very least first cousins to the story.
The sales cycle is long, integrations require multiple dimensions of executive buy-in, especially security and networking.
I think they made something nice but it felt like a nothing-burger story about something that is constantliy oscillating between bespoke and commoditized due to upstream problems that are such a mix of issues. Lack of oversight to micromanagement, inexperience to too much experience that they cannot let go of "the way it's always done".
It may be just my unpopular opinion but you are boiling an ocean of problems selling toolchains. Business and Tech are like water often finding the holes that lead to a path of least resistance, even when these erode the foundation of "core business". Toolchains that behave like guidelines and "parenting strategies" with removable guardrails have always offered the greatest rewards in my experience.
> Not to rain on the parade here but this is literally a copy-pasta product.
More to the point, the product offered no compelling reason to use it, let alone pay for it.
Being "fast" is not a selling point. Developers don't want slow pipelines, but that does not mean they want fast pipelines. The speed that the pipeline works is more depending on how the pipeline is setup than the overhead of the pipeline service, and other CICD services are already blazing fast. Take for instance CircleCI, which is already a tough sell over GitHub actions and GitLab CICD: how did the service stood out? Did it added tangible value over GitHub/Gitlab/CircleCI/etc? Shaving milliseconds off a minutes-long build ain't it.
It could be, if that is a pain point for the customer. You could be 10x faster than any other option, but if "slow builds" are not on the top 10 list of problems they are dealing with, it's a meaningless statistic.
> It could be, if that is a pain point for the customer.
But is it, though?
I mean, are any of the competing CICD services noticeably slow, specially if we focus on the system's overhead?
I don't think there are, at all.
We're talking about at best millisecond differences in workflows that take minutes just to do the things we tell them to, such as setting up a build environment.
I'm only hearing about Earthly through this blog post, but is their optimization just shaving time off of overhead? From their landing page, it sounds like their system made caching and build targets a first class feature.
> Rebuild only what has changed, automatic parallel execution, automatic caching with no configuration required for the entire pipeline, no upload/download of cache – instantly available
You _could_ get these benefits on other providers like CircleCI, but it would require a lot more work/customization.
"This pipeline will automatically perform all of your [insert line of business] testing, batteries included.
XYZ has signed off that if you get to the end of this pipeline you do not need to jump through any hoops to get to production." I would be screaming on a soapbox
There is too much process to know about a company to make any claim with any certainty and these companies will not tell you their process because you are a salesman, not their friend. Technology Sales is "Frenemys" at best and "Inherited Cancer" at worst.
Here's a pitch for OOP. If you do this you will have a line around the door:
"You can run this on your own internal cloud, public cloud, or mix of both. Hit the eject button and it will give you a [Docker, CircleCI, Jenkins, Travis] playbook to lift and shift your pipeline. Every app that goes through alerts each C?O that cares how it meets their KPI, Security Whatever, etc.
Set the bar as high as you want to get through the pipeline. Mission Critical? Core Business? Skunkwerks? This can be in the DMZ, this can't...you make the rules, we are the engine that enforces them."
I can tell you right now, your success will be predicated on Networking and Security teams that work together effectively and selling high enough to start the ball rolling.
We have this absurd "Throw paint at a wall and see what sticks" approach to doing business and then "People don't really want what I made" response when it fails. Look inward.
Reliable, configurable, and compatible. That's what, at least from the work I do, seems to be what the market is looking for.
Most CI/CD for sizeable companies have integrations into other systems, such as static analysis and things lime SBOM. Those tools tend to take a bunch of time outside the CI. They also need to ensure the results from those tools can break builds along with other alerting. Eventually the CI itself becomes a small fraction of total execution time.
It failed because the marketing is outright bunk and overly dishonest.
If I compile with Jenkins, Actions or Earthly, that compile time is going to be the same under each build system assuming the same build node. Claiming you're 20x faster when CI is firing within seconds is kind of meaningless. Caching and parallel execution are age old concepts when it comes to CI and every modern build system can do it.
CI is all about feedback, I didn't see much here in terms of collaboration and bubbling data up, I didn't look very hard, but this should be frontline and center. Lastly, I'm not interested in adopting a DSL for my builds ever again, sorry.
In your opinion, regardless I have to set those systems up anyways to use the Earthly system. Jenkins caches by default and has parallel(), actions uses actions/cache, actions is also async so it's parallel by default. Both can use docker and take advantage of dockers caching management and make is a great tool, but again, any build system can use make.
I don't think it's harder at all, in fact in my mind Earthly is more money, more steps, more time and energy, and more vendor lock in for the same output you can get in just about any modern build system.
How do you usually do caching for complex, multi-language projects in this case? How do you guarantee _only_ changed targets are rebuilt when dependencies, like a language version or external library, change?
With Earthly you just... write your Earthfile. You get everything for free. Caching and parallelization _also_ works locally, so I see that speedup in development.
Dependency graph is a feature of the build system with better understanding of the language import rules than the CI system can have (or whatever terminology you want to use). Good ones can hermetically, concurrently and deterministic build only the files that were changed.
Earthly is just a thin wrapper on top of docker multi stage builds[1]. Such docker layers are one-dimensional, meaning if one of the steps are calling “make”, that step will have to rebuild that entire layer, not just the one c++ file that changed. There are solutions to this with --mount,type=cache but they take away the reproducibility and and are equally possible with plain dockerfiles.
[1] tbh I don’t know if this is the case, but that’s how others explain it. This is core problem with Earthly. They don’t say how it works, just that you will magically get 20x faster. Without understanding how, I don’t know how to take that seriously and I can only assume it’s snake oil.
I mean, there is a need for a better Dockerfile syntax. But then market it as such! Not as magic 20x faster builds.
> How do you usually do caching for complex, multi-language projects in this case?
As a software engineer, I do my best to keep my systems as simple as possible. Each build stage either caches artifacts or delivers deployable packages, and they consume dependencies that are either built in each pipeline run or already delivered through other means.
> How do you guarantee _only_ changed targets are rebuilt when dependencies,
Why do you want that? Other than saving a few pennies here and there in pipeline runs, you do not have anything to gain. Your builds should consume dependencies already packaged and delivered, either from a OS image, your own container image, or a package deployed by you or a third party, and your build must be reproducible and tied to each and every pipeline run.
> like a language version or external library, change?
If you're mindlessly bumping up language versions and external libraries, you have more pressing problems with your setup than what's supported by the CICD system you're using.
I think there is certainly some value to that (the other posts don't appear to be fully grasping the complexity of the problem), but Earthly feels closer to an open source consultancy project than venture capital funded startup.
It's one thing to know a build failed, it's another thing to know why and how a build failed, where it failed, and what the cause was. If I have to swim through or context shift to a separate build system's output to get that, what is this actually doing for me?
That looks really cool, thanks for the link. Something like this well integrated into Jenkins would be super useful.
Our DevOps team build something custom: they check all test logs of all builds of all branches and aggregate all that into a Grafana dashboard. We use it to monitor tests that are failing, to get a better grip on flakiness. Works okay but could be better.
That doesn't help with non-deterministic failures and I've also yet to see a true "write once, run anywhere" system ever. It may be 99% "write once, run anywhere" but there's always that 1% edge case.
Exactly. This just reminds me of "works on my computer". Building locally is more advantageous for developing CI workflow, but I want to be as close to prod as possible and doing that on snowflake developer workstations is an exercise in futility.
Earthly makes use of BuildKit, which essentially executes the build steps in containers. It provides more isolation from the CI runner / dev workstation. Instead of having developers manage their own build tools, Earthly makes it easy to have the build definition manage them.
All flakiness I’ve observed in code I’ve written has been dominated by things like “this job isn’t getting enough cpu cycles and the test assertion is too aggressive in such a scenario” or “lack of CPU cycles is triggering a race condition” or “statistical test X isn’t written robustly”. Not sure how containers solve these problems. My point being that this tool solves some problems for some teams and maybe for a lot of teams that struggle with this problem, but hard problems remain and this isn’t a silver bullet for that. You can’t outsource stability of the project-specific test infrastructure which is where most of the cost lies these days I think.
Nix solves all these problems for good.
It's a monster of a system, but if you're migrating anyways then might as well migrate to the real deal, not to some fly-by-night half-baked thing that gets you halfway there.
CI is an overgrown shell script running your build and telling you when it fails. In general build tooling has gotten so slow that the cost of whatever CI runner relative to it should be nil.
If you want fast CI you need fast tsc, clang, rustc, etc... not a faster program that calls exec on them.
A bit more on topic, if you're selling CI and your business fails it's because people you're providing no value - people can run their build scripts fine without you.
Perusing this article quickly, it means a CI that is automatically handling things like caching build artifacts so that you don't need to recompile your entire repository every single commit. It's not about a faster program to call exec; it's about a program that knows it need not even call exec.
Incremental builds are something even enterprise build systems fail at.
I've had Visual Studio and MSBuild fail on me multiple times by deciding not to recompile a certain .cpp file, because they got the dependency graph wrong. This results in a running but inconsistent executable that's a bitch to debug.
If you're selling me another tool that duplicates this work, but that has even less context on my project than the build system, I'll pass.
I don't have a ton of experience in Bazel, but from what I have experienced with Bazel I think it would be more accurate to say that you configure the dependency graph yourself in Bazel. Then Bazel can determine what to build from scratch and what to take from cache
One thing these tools should do automatically is at the end log to the console (or whatever system) what was compiled and what wasn't.
This made debugging so much easier for these issues. They're bound to happen, because we're all human and bugs are a thing. Automatically being able to audit the build as step 1 makes this easy to spot
Earthly is explicit about inputs ( using COPY) and outputs of each build step. This let's it be 100% certain about whether it can cache parts of the build.
If you are familiar with docker layer-based caching and Bazel, you can imagine how Earthly works to eliminate rework in builds.
I looked at Earthly just a couple of weeks back. It’s a tool that makes perfect sense - especially when I was deep in a makefile wishing the caching was as automated as docker.
In the end I decided that I couldn’t face the complexity of dealing with someone else’s hosted tooling. Luckily in this case.
except not really.. their tutorial [0] includes no-compile languages like python and JS, with only non-trivial compilation being and Java... and how do they do it?
RUN gradle build
At least in this case, Earthly has no insight into how repository is organized, and it _will_ recompile the entire repository with every single commit.
So what's the real value of it? this basically seems like a better "docker build" alternative, giving nicer inter-layer caching. Maybe will save some time on dependencies if they take some time to install and devs were too lazy to set up own workers and used ephemeral runners instead.
That's step one, 'how to write a basic Earthfile that just does the thing', so they don't delve into the details of caching and how it works, but look at the whole block and not just that one line; it operates like a Dockerfile:
build:
COPY build.gradle ./
COPY src src
RUN gradle build
RUN gradle install
SAVE ARTIFACT build/install/java-example/bin /bin
SAVE ARTIFACT build/install/java-example/lib /lib
The COPY phases copy files into the build context. If the files haven't changed, then the commands don't need to be run. In other words, if `build.gradle` and `src` are unchanged, it won't run `gradle build` or `gradle install`, and it'll just give you the artifacts from the previous build.
They have a golang example in part three[0], which they redesign to use caching effectively:
build:
# Download deps before copying code.
COPY go.mod go.sum .
RUN go mod download
# Copy and build code.
COPY main.go .
RUN go build -o output/example main.go
SAVE ARTIFACT output/example AS LOCAL local-output/go-example
They copy in go.mod and go.sum and then run `go mod download`. If the mod and sum files haven't changed, then `go mod download` doesn't need to be run.
Then they copy `main.go` in, and run `go build`. If the `main.go` file hasn't changed, `go build` doesn't need to be run.
Gradle can do the caching of tasks itself in a level of granularity no CI can. If you keep your build cache "live" in the CI, Gradle knows to only recompile what has changed, and to only run tests which changed (or tests something which changed). I am not sure if many other build systems do that, but this is something I would absolutely expect from the build system, not CI. Do you not expect caching when building locally??
The whole idea with CI doing this is wrong IMO. All I want CI to do is call a couple of commands after setting up the environment (which would include the Gradle cache or whatever to speed up things - no need for your revolutionary CI build system).
A CI system like TeamCity will give you that sort of thing for free though, without needing an Earthfile. It just dedupes builds at the same commit, and keeps checkout directories between builds on the agents, so if the build tool has a cache then it's automatically reused. Works great.
Given that it's hard to understand what exactly this is for? Build systems normally try to avoid redundant work for local purposes anyway. Is it some hack for companies that insist on resetting build environments to zero on every single build, and then decide they want caching back? If so, why not just ... let the builds take place in directories containing previous builds, and fix the problems? That's what we do at my company and it works great (using gradle), there are only rarely problems with unclean builds and it's easy to rerun with a clean checkout if that's suspected to be an issue (the bugs are mostly in our own code which does a layer of caching above the build system anyway).
That only splits dependency-build from app-build. Any change to the app source code still causes a full rebuild, unless you figure out how to share the cache directory between builds (and then parallelism gets sketchy).
Generic CI just simply can't parallelize+cache on a level comparable to systems that understand the build itself (Bazel, sccache, etc).
sure, but in a java project, how likely you are to have a commit with no changes to src/? or for a go project, how likely is it that you want to duplicate you entire dependency tree in a clunky, non-parallel dependency description language, vs just doing "COPY * .; RUN go build" and letting golang figure it all out and build in parallel using all the cores?
Don't get me wrong, there is definitely value in caching dependencies, and I can see Earthly approach help if you have, say, a Python app, with 10 minutes of dependency install time, 0 build time, and 10 seconds of tests - you'll see huge improvement. But you'll also see almost the same improvement if you use Dockerfile on a persistent host instead, and as a bonus you won't have to learn new system.
One place Earthly might shine are monorepos with multiple independent projects that even use multiple build systems... but I am not sure how many of those exist _and_ don't have some other caching solution already.
> CI is an overgrown shell script running your build and telling you when it fails.
Oh, I wish it were that simple...
But, I also don't know what they measured. But, I'll give you just one example. Jenkins default landing page is a disaster in terms of speed. It tries to display data on recent builds everywhere across your cluster. For even not so big clusters this can be hundreds to thousands of items to pull from individual nodes running those builds. I've managed to crash Jenkins more times than I could count by simply loading the landing page w/o a particular configuration added that prevents it from its default behavior.
CI server will have its own database, typically, managing all sorts of CI entities: jobs, artifacts, users, secretes... It could be quite big and you'd do well taking care of proper indexing etc.
CI will have its assortment of runners, which are often provisioned dynamically (think VM or Docker images that need to be deployed to runner nodes), distributing them across the cluster is also not an easy task, if you want that to work fast. Of course you also want to be able to distribute the artifacts across the cluster -- this also takes time and resources.
CI will have its own bookkeeping necessary to, essentially, do garbage collection, reporting and self-diagnostic.
In big enough clusters this stuff can have a very high latency, if you don't specifically work towards reducing that latency.
> If you want fast CI you need fast tsc, clang, rustc, etc... not a faster program that calls exec on them.
Ever heard of ccache?
But, really, this is also super-naive. To make builds fast in a distributed system that CI is you would also need to figure out how to distribute this cache, how to modularize the builds... You might have heard about how one of the most difficult things in programming is cache invalidation? -- That's only partly a joke.
The only part of the Earthly pitch that resonated with me was the ability to run the CI locally. When debugging your CI, it is just faster to find problems when you can run on your own box.
I have a gnarly GitHub Actions script that took forever to get right because the debug cycle was 10 minutes long.
That's one of my biggest complaints about gitlab-ci. Theres basically no way to debug complex rules without just running it. They've got a way now to simulate for default branch, but that's like bare minimum necessary. No way to simulate based on vars or anything else. And no way to try locally, just gotta fucking send it live.
Yeah, the faster but is also a bit of a non-seller, they have to integrate the world/deal with bad build times that can't easily be parallelized (what if there are no modular dependencies, that your tool can understand?) except by tools that have nothing to do with their product.
But there have been many times when debugging a CI has been necessary, having to figure out what has gone wrong on a remote system is often an issue...ofc often the remote system is itself the problem...
Sure, but you have to be 100% perfect in guessing when the cache file must be rebuilt. If there is any situation at all where you use the cached file when you should have rebuilt your tool is not useful.
Note that you can go too far. If you rebuild a file on linux even though the only changes were in a "#ifdef freebsd" (windows, mac...) section that is a waste of time, but not harmful. However we already have tools that do that.
This is a solved problem; modern build tools and workflow orchestrators face the same challenge. You declare your inputs for a given task and cache the output as long as the inputs are unchanged, as determined by their hash. So there is no wrong guessing, only an incomplete specification of inputs. It's an elegant solution, if I do say so, and has worked well at companies I've seen it used.
It's not a solved problem, very few builds in the wild are deterministic and reproducible. You have to build up the universe from scratch (like guix and nix) for the caching to be sound enough to be reliable.
Just because an algorithm exists that can help doesn't mean that the universe of software can fit cleanly into a model where that algorithm is perfect.
The other fundamental problem with this model is that it cannot model cyclical dependencies (a degenerate case: glibc needs python which needs glibc). Since you can't hash the output until you've hashed the input, and the input hash depends on the output hash you need a way to break the dependency cycle. Tools have varying ways of mitigating this but it's not exactly "solved" in an elegant sense.
And finally, a lot of build steps are faster than hashing their input and outputs. So if you want the utmost performance you need heuristics or a model that allows for some things to be hashed and others to always be considered fresh.
It is a solved problem - you need to use a build system with sandboxing like Bazel, and also vendor all your tools.
Almost nobody does that because very few people realise it is the solution and even fewer realise they're going to need that before they set up their build system.
> very few builds in the wild are deterministic and reproducible
There's a difference between 'deterministic and reproducible' and 'predictable'. Sure I can (probably) build curl ten times and get ten technically different binaries, but the differences aren't relevant to the functioning of the actual result.
Assuming my build environment isn't changing out from underneath me (i.e. if I'm using the same versions of the same dependencies on the same platform with the same compiler toolchain, and I'm not passing in different arguments or environment variables) then the end result should be functionally identical.
> You have to build up the universe from scratch (like guix and nix) for the caching to be sound enough to be reliable.
This is a false assertion. If I'm building (again, as an example) curl, then I don't need to be rebuilding the kernel, glibc, kerberos, libssh, gnutls, brotli, etc. all in order to get a working curl build; I just need to make sure the libraries and toolchains I have installed are valid. If I create a docker image with all of the dependencies and toolchains preinstalled, and then I build the exact same curl code inside of that docker container on two separate occasions, then the end result is going to be indistinguishable without hashing the build artifacts.
> Just because an algorithm exists that can help doesn't mean that the universe of software can fit cleanly into a model where that algorithm is perfect.
It doesn't need to be perfect, it just needs to be correct for the vast majority of cases (which are not somehow inherently broken already and just haven't failed yet).
> a degenerate case: glibc needs python which needs glibc
Because you're not rebuilding the entire world (in 99% of cases), this doesn't actually matter. If you're building glibc use the existing build of python that you already have installed, which is the same identical version to the one you used last time you built glibc.
If I'm rebuilding glibc, do I also need to rebuild python? If the glibc code hasn't changed, but maybe python has so I get a different result? Well okay, has the python source changed? No? Okay, have the dependencies for python or glibc changed? No? Okay, well problem solved then.
I'm not sure why people have this philosophy of "this problem isn't solved because of some imaginary, unrealistic corner case that either shouldn't exist in the first place, isn't actually a real problem, or that no one will run into". In 99% of cases it works, and if it's not right for a particular circumstance then just don't use it in that circumstance. Sometimes makefiles aren't the right tool. Sometimes a GUI isn't the right tool. Let's not argue that we shouldn't have GUIs because sometimes running a command on the CLI is easier, and let's not argue that an extremely basic caching system that makes sane assumptions that are almost always valid isn't a good idea.
And the parent is right: if you correctly define your inputs and outputs, then the system works. What those inputs and outputs are is up to you to determine. Maybe for you, the outputs are a working build of glibc, and maybe the inputs are all of the tarballs necessary to build Linux from scratch and a working bootstrap environment, but if all of those inputs and that whole environment are identical to last time, what's the point of rebuilding any of them? Maybe there is one, but that's up to you to determine and model.
I agree with you, but my point is ultimately that content hashing inputs like GP is arguing "solves" the problem doesn't actually solve the problem.
Take your curl example. Its inputs are its dependencies. Now say you depend on libcurl and a more recent version of glibc than curl was compiled against. If you only track the hashes of inputs to determine if something needs to be rebuilt, then in this scenario you will have to rebuild libcurl.
What you're talking about (determining if your dependencies are "valid" and don't need to be rebuilt) involves tracking compatibility metadata out-of-band of the actual content of the build artifacts, which is not a solved problem.
The point about reproducible builds is that they're required for any kind of build system that depends on content hashing to determine if something needs to be rebuilt, because it's actually impossible to track that based on the content alone.
It gets hairier when you start abstracting what a build artifact that depends on another build artifact is, because there's more to builds than object files, dlls, and executables.
From my experience with Gradle, it's fairly tricky at the build tool level, trying to do it at the CI level must surely be harder.
If (for example) I change a value in gradle.properties at the top level, do I need to rerun every test in every sub project? You need detailed knowledge of how the build is structured to decide that.
Where that world sucks is that it forces you to ignore or duplicate work from the native build system. E.g. Go or Rust know which source files are needed to build something, yet Bazel makes you write that out.
No, generating the file isn't "the fix" if you're still committing those generating files into the source control; that's still duplication and manual labor.
> you have to be 100% perfect in guessing when the cache file must be rebuilt
Or, you make reasonable assumptions that are correct in most cases, and rely on the user to provide hints as to when the cache should be invalidated.
Assuming your code, environment variables, and `args` are all factored into invalidating the cache, there are few situations in typical builds where using the cache is a bad idea. The biggest ones I can think of are if your build system is going out and fetching data itself to decide what to do, e.g. if you're downloading source code (e.g. a shared library) from an external source which may have updated it.
That said, if you're just arbitrarily pulling down whatever the latest build of libcurl is or something, you're going to have a bad time eventually anyway, so you should either put a static version into an arg or just put the download into a separate step; at which point the cache will correctly invalidate when it's changed.
Maybe if a build is only valid for 90 days from the build date or something (e.g. a preview build) and that's built into the binary at compile time, but in that case you can pass that in as an arg anyway and then your cache is going to be invalid every morning regardless.
I'm not going to say it's never an issue, but the same code with the same environment and the same arguments, built on the same platform with the same versions of the same dependencies, using the same versions of the same tooling to run the same commands and copy the same artifacts to the same locations should very nearly never produce a valid output now which is not an equally valid output in the future unless it is fetching and acting on external information (which can be easily parameterized).
I'm glad that they're just shutting down the service and not closing up shop entirely. I really like the tool (to the point that I regularly stop by their careers page); the syntax for an Earthfile is a very sensible and incremental evolution of Dockerfile syntax, and it makes a whole bunch of things easy that were either previously impossible or very awkward with a Dockerfile alone.
I remember when Docker introduced BuildKit and buildx, they were trying to push Dockerfiles as a general-purpose build system (i.e. not just for containers but also for producing file artifacts and the like) - Earthly takes that idea and really delivers on it.
It was honestly a bit hard for me to understand what happened from this write-up. I read it a few times, and the terms are still bit confusing. Seems like there are at least two separate issues
- migration of the CI configuration -- from existing CI YAML (github/gitlab) to Earthly (Makefile/Dockerfile hybrid)
- migration of the job runners -- from existing CI to a service hosted by Earthly
I would have thought the first thing is the hard thing. Changing the language could take months or even years.
But I don't understand what the blog post was saying. I thought they validated that part? People were able to switch?
But then it wasn't validated at the end? The customers were faced with 2 migrations, not 1?
But now what's happening? You're still keeping the Earthly syntax but abandoning the CI? Isn't that the hard part to migrate to? I'm still confused.
I'm glad it wasn't just me. I'm interested in both sides of this, the product development side and the dev tools side. But with all that somewhat confusing prose, I ended up in the neighborhood of: "I'm happy for u tho. Or sorry that happened."
Was the heart of this that they assumed that radically faster builds was the killer feature, but they hadn't actually validated that assumption before building it? Or was it more they hadn't really segmented their users, so they hadn't realized that the high-dollar customers had different needs? Or that they just gave away the thing that was the real-problem solver for paying customers?
And given that the somewhat chaotic article was written by the CEO, I have to wonder to what extent the chaos was just a lack of editing versus there being a lot of actual chaos in the company throughout the process. Long ago Steve Blank wrote an article called "Founders and dysfunctional families" [1] where he talked about how a lot of founders are really good at managing chaos because they grew up in it, something that certainly describes me. He added that something that can make the difference between success and failure is the extent to which founders can also handle non-chaos. Because the ones who can't tend to "throw organizational hand grenades into their own companies" to get things back to the level of chaos that they're good at. That observation has given me pause for thought many times in the years since.
It's that people's CI over time become some kind of amalgamated model encapsulating how a firm makes every individual piece of software it makes and lands it places, think (ab)using a CI as Airflow (arbitrary automation job runner), and that the migration is first reverse engineering what people used to know, before even starting untangling all of that to express it some different way.
Nobody really wants to stop the world and sort that.
I think the key is that nobody "wants" to spend any time at all on CI or their build system. It is one of those necessary evils that you have to do to have functioning software. Once you do it, even if it's a slow buggy pile of hacks, you will just ignore it until the pain gets very very bad.
This was the same problem that caused our team to abandon CircleCI. They kept wanting us to rewrite our CI configs, touting various benefits that we just didn't care about. All we wanted was for our janky broken builds to keep working 90% of the time, like they did before Circle tried migrating this or that.
>I think the key is that nobody "wants" to spend any time at all on CI or their build system.
or if they do, they're hosting their own.
if you're selling a hosted CI platform, your sales pitch is "pay us $$ instead of paying your staff $$$ to do devops". if your pitch is "pay us $$ and then also spend $$$ on staff hours to manage our service", it's very unlikely the CI is doing anything useful enough that the staff couldn't just run their own thing in-house.
it doesn't take much configuration to be more difficult than an EC2 instance running a script on a webhook from your version control.
Some people do want to spend time on that! It depends whether you see fast feedback as a competitive advantage.
I once worked on a system where we had excellent test coverage and all our unit tests ran in under a minute. We worked hard to keep it that way, because it was a huge boon to productivity, especially when improving design issues. It also enabled frequent commits and merges, which made collaboration easier. It was a great experience.
But yes, from the business perspective, the leading edge is a pretty small audience. (Circa 15%, per Moore.) And the bottom half of the audience by competence, which Moore called the Late Majority and the Laggards, can be well more than half your revenue in a SaaS context because they will create problems for themselves (like slow/buggy builds) that mean higher per-user revenue. And they may also have special feature needs where they'll pay more rather than clean up their messes.
In that context, doing the right thing for your business can be very different than doing the right thing for users. Sort of like a physical therapist who makes most of their money from renting crutches and in-home meal delivery, and so doesn't push people to actually get better.
I had a talk with a circle CI rep at a conference and I tried to explain to him that yes I’m not happy with our custom Jenkins setup (agents, CasC, pipeline library), but that I also want to spend time and money on a migration project to end up at the status co again. If I would go to my boss and say we can either spend some time and fix our setup or spend time plus running costs for this external solution then the answer will be clear. I think convincing teams with huge stacks of CI setup to just switch because of marketing promises isn’t working. I would think of using a new setup when I start from scratch. At the moment at least.
I have been using CircleCI since the beginning and the only breaking change I saw was when they introduced workflows. This didn't need any changes to the difficult parts of my own code and was mostly copy-paste.
I believe that my configs are fairly complex using almost all existing features across many different projects and build targets.
We started using CircleCI in early 2016, I believe. There have been two major migration points. The first time was when they went from circle.yml to .circleci/config.yml, around 2018, which might have corresponded with Workflows. The second time was when they got rid of their old converged docker containers and moved to custom container runtimes (and a new set of images, none of which did _quite_ the same thing as their old images).
Neither of these took tons of effort to resolve, but both broke our builds and made us spend time thinking about CI when we would rather have been iterating on our products. When their later migration happened, Github Actions was just coming around, so we figured that if we had to rebuild and re-test all of our CI, we might as well move it to a more reliable and cheaper runner. Now we only use CircleCI on a handful of legacy repos.
I started work in December at an awesome company as a build/release engineer. Our current release workflow for our open-source components:
1. Run the CI build/test pipeline
2. If it passes, run the 'Publish' promotion (manually, on any pipeline you want)
3. The pipeline runs a make target
4. The make target starts up a VM (from the VM it's running on)
5. The make target SSH'es to the VM to run a make target
4. The make target builds a binary
5. The binary runs more make targets
6. The make targets run various scripts in docker containers on the VM the other VM started
7. The scripts in the docker containers build the components
8. The binary from step five runs more make targets to publish the releases
9. We test everything with another promotion which runs make targets which build docker containers to run python scripts (pytest)
This is also built by a complicated web of wildcarded makefile targets, which need to be interoperable and support a few if/else cases for specific components.
My plan is to migrate all of this to something simpler and more straightforward, or at least more maintainable, which is honestly probably going to turn into taskfile[0] instead of makefiles, and then simple python scripts for the glue that ties everything together or does more complex logic.
My hope is that it can be more straightforward and easier to maintain, with more component-ized logic, but realistically every step in that labyrinthine build process (and that's just the open-source version!) came from a decision made by a very talented team of engineers who know far more about the process and the product than I do. At this point I'm wondering if it would make 'more sense' to replace it with a giant python script of some kind and get access to all the logic we need all at once (it would not).
Taskfile looks worse than make to me in every possible dimension (except maybe windows compatibility, but I don't believe that it meaningfully supports that).
You should at least read and understand the paper "Recursive Make Considered Harmful" before attempting to replace make with something "better":
Most people use make incorrectly (and it sounds like the system you describe makes most of the classic mistakes). The paper I linked explains how to use Make's templating language to help it scale to complicated setups.
Here are a few critiques of taskfile (from skimming their documentation):
- The syntax is incredibly verbose and weird. For instance, you have to write a tree of yaml nodes to loop over a set of files. Make's syntax is weird, but at least it is not verbose.
- The documentation makes no mention that I could find of deriving a set of tasks from a list of files (Make's $(wildcard) macro, or stemming rules like "%.o : %.c"). So, I guess the taskfile will end up being huge (like the size of autogenerated makefiles from autoconf or cmake), or you'll end up using additional ad-hoc build systems that are invoked by your taskfile.
- From the documentation, they haven't thought through having more than one project per unix account "When running your global Taskfile with -g, tasks will run on $HOME by default, and not on your working directory!"
- I suggest taking an idiomatic but non-trivial bash script (which is 99% of what make is invoking, and that taskfiles support), and then trying to port it to python directly. There's usually a 10-100x line of code blowup from doing that, and the python thing usually bitrots every year or so (vs. every few decades for shell).
The space was destined to collapse these two products not because it makes sense conceptually but because their implementations are almost identical and maintaining fewer systems is great. One was always going to eat the other and CI systems are turnkey, hosted, and live close to the code that it will be running.
I wish they had either remained separate or gone the other way but it's not worth swimming upstream for it.
This article is confusing because the author is too close to the problem, and it needs to be explained from an outsider’s point of view before diving into the details. But it looks like the information is there.
It sounds to me like they ended up with an encapsulation layer between language-specific build systems and the continuous build systems that kick them off.
A comparison might be with something like Bazel, which does everything, but full adoption requires going all-in on using their build language to replace language-specific build systems, often including moving source files to make it work. This feels foreign compared to using a language’s own build system, but it might seem natural for C programmers, where the language doesn’t have its own build system. Similarly, Java went through several unfortunate build systems before unfortunately settling on Gradle.
Language-specific build systems end up not doing everything because each language has its own conventions and ecosystem. Modern ones don’t try, they know what they’re good at and stick to it.
So, the tool that actually knows about multiple languages and their artifacts is often a shell script, makefile, docker file, or the continuous build itself. Or even doing it by hand. That’s the layer they’re trying to improve.
I don’t have a clear idea what their fundamental insight is that makes their way of doing it better, though.
I looked briefly at Earthly a while ago for work because I have to write CI integrations for a whole bunch of CI platforms (GitLab, Azure DevOps, Jenkins, and maybe GitHub Actions). I ended up going with a very similar product (i.e., another nice BuildKit frontend with integrations into a bunch of CI systems), Dagger, because the DSL it used for defining pipelines at the time seemed better to me. I ended up deeply regretting this decision, as the Dagger devs have de facto abandoned that language entirely in favor of churning out SDKs for the most popular programming languages— all imperative, all Turing-complete, and all, imo, bad fits for the space. So now I'm back in hell, integrating with all of these systems by hand, hesitant to trust this kind of tooling to a startup again.
To me this is still the most appealing use case of something like Earthly. Push-and-see-what-happens is still the norm on almost all CI systems, and it's an abominable workflow. If you are at a large organization and have to support teams that use a wide variety of CI/CD platforms, something like Earthly could potentially save quite a lot of pain. But the appeal is precisely in supporting existing CI platforms, rather than in adding another one.
CI/CD & provisioning are inherently order driven tasks, I actually prefer the SDKs over the CUE implementation. The main issue with the old CUE implementation is that they had to try and align BuildKits DAG solver with CUE's DAG solver, and they worked in opposite ways. I tried to help them through this about 3 years ago (being a CUE expert). I definitely think the SDKs is a much better solution for them. Whether we like it or not, a lot of the industry is moving this way. Pulumi is another example of this, though I'm sold on imperative cloud infra, builds do seem to have alignment here.
Side note, I will be exploring a new CUE + Dagger setup, though it will work differently than the old Dagger engine
I'm not totally opposed to powerful (Turing-complete) languages being used for configuration or defining builds— I think Nix works really well in that space, and someday Nickel may as well.
But wherever I've seen build tools from a language ecosystem that just embed a DSL in the host language (e.g.: sbt, Gradle), I've also seen projects get way too clever with their builds and create total nightmares for downstream packaging efforts. I anticipate similar problems for other kinds of configuration tools based on general-purpose languages, like CI or cloud IaC.
> Side note, I will be exploring a new CUE + Dagger setup, though it will work differently than the old Dagger engine
> Dagger devs have de facto abandoned that language entirely in favor of churning out SDKs for the most popular programming languages— all imperative, all Turing-complete, and all, imo, bad fits for the space.
Disclaimer: I’m the CEO of Dagger.
You’re right that we offer SDKs for popular languages, which happen to be imperative. But Dagger remains a declarative system, so what you liked about the earlier versions is still there. The trick is that we moved the declarative layer from static CUE configurations to dynamic GraphQL queries. Then we generated client libraries in various languages from the GraphQL schema. So not only can you still declaratively construct your DAG like before, you can do it in whatever language you prefer. You can even run a DAG directly in pure GraphQL: see https://play.dagger.cloud which lets you do this straight from the browser.
A useful analogy is SQL: it’s a declarative language that is commonly combined with another, often imperative, language.
I hope you find this message useful and consider giving Dagger another chance.
While you are here: the Node.js syntax highlights on you QuickStart docs seems to be broken. Tested with Safari and Chrome on a recent macOS. That’s not a huge issue, but makes it a bit harder to grasp what’s going on and how the Dagger SDK actually works.
> Why not just simplify the stack? Why pay both the CI vendor and us, when you can just pay us?
Coz the CI they "pay" for comes with rest of the stack, and is not just CI product. Both Gitlab and Github have far more than just CI
>These weren’t the raving fans we were used to talking with. New people would look at Earthly CI with a skeptical eye. They were mostly thinking that “all CIs are the same - they just have different syntax,” and then they would not really look any further as to why we might actually be different.
we don't care about CI, we just need it and we need it to work.
If we have working CI manifest for our app, the new app will have same manifest with few find-replace in it. The pain might be there to create it. And the example ones don't really look any easier than doing same thing in Gitlab.
Frankly, putting some transpiler that takes Gitlab or Github CI config and just makes Earthfile on the fly would probably convince some people to at least try.
I've used Earthly for a year and absolutely love it.
It allows you to have two dependencies on the host system for any project -- just Docker and Earthly.
I've used it to build cross-compilers, system software, web apps, my resume, school projects.
The speed up is real. You can definitely get caching and parallelization and containizeration with other build systems, but Earthly's value is that it comes for free with a familiar syntax.
I feel like it's just a patience game of staying alive long enough that organic traction takes off. Earthly is bloody fantastic, the flexibility and expressiveness IMO is hard to beat.
Judging by the description, that project seems to be another DAG-planning meta-program with input-based memoization. This is the superior approach to fight complexity but, unfortunately, most engineers just don't understand the problem and the solution.
Complexity, not speed, is the problem with builds. It makes perfect sense that "another layer of complexity that wraps up all the other layers" was tough to sell.
This is true to a certain extent. If a build lacks the automation required to run without significant human oversight then it will be slow because it is complex.
But a normal build should be completely automatic (or at least close to it) and at that point whether it takes 30 seconds or 3 minutes doesn't really matter.
At least for me, whenever I have to set up CI or touch CI, it's incredibly painful. I usually end up having to push a bunch of commits to debug stuff, and there's a lot of configuration hell. I don't know if Earthly solves this, perhaps it does. But if I were considering moving onto a new CI, I'd immediately think about having to redo this painful process. And that's at the team level. At the org level, switching CI would be even worse.
That does make me wonder, how much of CI adoption is actually moving from one provider to another versus new projects adopting CI for the first time? I'd guess that barring a really extreme reason, most projects pick a CI provider and stick with it.
CI systems are important to me, but my requirement is pretty much: Can it run a bash script. Every CI I've ever used has degraded into a bash script runner. It starts out well enough, you try to use all the build in features but at some point you fallback to running a shells script and then slowly more and more move to the script, because it's easier.
Someone should just build a bash CI, something that can manage secrets and then run one or more shells scripts.
Once you have hooks into source control for commit-based callbacks into CI, everything else need only be shell. Like, if you build a triggers-shell-scripts-off-source-repo-hooks web service, does putting the results in a pretty web server really matter for anyone besides executives who want dashboards dashboards dashboards? I'm already sending script output to text files and our chat system via curl to boring web service APIs.
In a lot of ways that's pretty much what Earthly is, except those bash scripts run inside containers (that you can configure inside Earthly itself) and you pass files/artifacts between steps/containers.
The book talks about the state of the business Earthly CI was in: They tried to go from enthusiasts to generalists... But with a completely new product!
I don't think it was wrong to build a new CI; instead, I think the thing they did "wrong" was build a super-complicated CI; or that when they tried to cross the chasm, they tried to sell a (cough) completely different product than what their early users were using.
I'm quite passioned about CI/CD and working in this field for the last 7 years.
I'm keeping a close eye on jenkins, tekton, circleci, travis, github actions, gitlab etc. and never ever heard about earthly.
Btw. one thing the person doesn't get: Everyone wants to have fast and reliable builds but no one cares.
No one.
CI/CD Systems in a developers eye are blackboxes. They don't care.
And its already a solved problem somehow. I was waiting for Jenkins X to happen (something like jenkins working nicely in a cloud native way because Jenkins Plugins!) but that didn't happen.
What happend instead is Github Actions. Holy shit are Github Actions good. Srsly.
GitLab stuff was slow and effort. GitHub Actions just do what they should do 99% of the time.
If you do not have a huge build, you will not care about your CI/CD System and everyone who has real build issues, has to solve it by another strategy than just switching the build system.
That's an interesting perspective. I used to work at a large company with a massive Jenkins setup. The nice thing about Jenkins is that your unix skills translate. It basically just runs a shell script and you can do anything.
In GitHub actions, the process is super abstracted and opaque. It doesn't really match anything I've used before and knowledge of unix doesn't help top much. I can't really reason what will happen, things are the way they are because that's how GitHub actions work, rather than something I can work out.
I can see a certain part of the build is slow, I know how I'd fix it if I were on Jenkins land, but here it's just too abstracted. I end up just googling and finding someone else with the same problem and copying how they fixed it.
I do agree that gitlab is worse somehow than GitHub actions.
GitHub Actions is the same as everything else. It doesn't work any better or worse. But the UI is clunkier, you can do less with it than other systems, it lacked important features forever (still does). The only reason you think it's significantly better is psychological.
The "market" of actions is better than competitors because the incumbent gets an advantage, and it trades on the idea of a generic executable as a 3rd party plugin, unlike other systems that try to force you into a particular language or paradigm. But other systems do this too.
I build enough pipelines in GitLab way to be able to tell you that GitHub Actions is much easier for me to implement.
Its much closer than if you would have a second system like jenkins, i already have access to git and the packages.
I do use much more container images than i did 6 or 10 years ago, that might have something to do with it but nonetheless, i do prefer GitHub Actions.
I also stoped liking GitLab though. The Autodevops was horrendes, the security issues is concerning and plenty of ideas or features are half done. this is also something which changed. I liked GitLab much more than i do now. But looking at the year, i haven't looked at GitLab for at least 2 years now.
For anyone looking for fast ci, I'd highly recommend webapp.io. it automatically caches layers if it detects it doesn't need to be run, and you can split a VM to shard test execution. Runs way faster than GitHub actions, you can ssh into any layer to debug, and it's just like a Linux machine/dockerfiles so no crazy syntax/mental model to learn. Support is great and we've been super happy with the experience. Best CI platform I've used by far.
There are a bunch of these next-gen CI / container building systems. They all pretty much build on BuildKit like Docker, but unlock way more of the potential, namely DAG layers (think multi-stage docker on steroids), and better / more granular caching. These next-gen CI tools are fighting to be the UX on top of BuildKit.
Dagger is where it is at, and it's built by the creators of Docker, based on their lessons learned. If you're looking to try something out, try Dagger. You write in the same languages you do every day, no need for some weird syntax like Earthly or the flatness of a Dockerfile. Their CLI/TUI is pretty sweet too, but you don't even need it.
That line caught my attention too, and it looked superficially true. I have certainly never purchased a tool based on a phone call from a vendor. However, that only pertains to cold-calling, which is the absolute worst way to sell anything.
Advertising is a trillion dollar market for a reason – it works. The trick is to find the correct message and media for the audience.
Effective advertising takes a lot of skill, a lot of money, or both. A large portion of HN posts (including this one) are contain "hidden" advertising, and they absolutely work.
I have purchased more than one developer tool based on reddit and HN reviews, which originated as posts by the seller. On the "lot of money" side, a poster in the arrivals tunnel at SFO is an excellent way of raising brand awareness; even for developer tools.
For me the killer quote was “if the prospect requires a demo, then they’re not worth going after” — because as a prospect, if your product requires a demo, I’m already halfway to no. I am so tired of “enterprise” products that have to be purchased as a pig in a poke because they can’t stand on their own during a trial period.
I think HN will eat this up because few of us enjoy being sold to but like most absolute statements it's probably only applicable in specific circumstances. Enterprises sell tons of developer tools and move millions because of demos done well. AWS and Azure are definitely getting on planes and doing demos.
I have no personal experience with selling a SaaS, but read that AWS mostly sold to startups at first because there were no migration costs yet. Out of curiosity, has the same been attempted with Eathly and, if yes, did that work?
Is the concept that requiring a demo for developer tools is a negative signal that strange?
CI is also notoriously expensive to migrate, which the author alluded to. The pumpkin spice latte of software engineering is disdain for things like Jira and Jenkins but what keeps them in their seat is all the custom integration work that has to be done.
The thing with Jenkins is you can also make it custom enough so you have different teams handling different parts (say the building and the deployment, or the releases). In most CIs this is a bit harder to do - I think the Azure Classic pipeline also had interesting ways to delegate responsibilities. In the end the good thing of the different CIs is they apply to different problems and one can use them all at different parts of a company - teams that work in more innovative developments vs teams that maintain products that have to be trusted and adhere to many regulations.
I'm having a very similar issue getting https://brisktest.com/ out there. People complain about things like the speed of their build system, but convincing people that your CI system is faster just seems like an uphill battle. Maybe the era of launching new indie dev tools to developers is over and you just need a massive marketing budget to bombard devs with advertising and lots of sponsored content.
There is no real business advantage or value add to switching CI/CD systems. The only people that do it no longer have anyone working for them that knows how the existing CI/CD system works, so the new person just replaces it with something else. They're all basically the same crap. Run a job, evaluate its output, maybe run another job, send a notification.
Whenever you run into a problem with a CI/CD system, it's because the system is "opinionated" and "tightly integrated", but of course lacks some obscure and subtle capability that you need to solve a problem. The fact that you are using a specific CI/CD system is what dooms you to eventually need another one to solve the problem that you current one cannot solve.
It's ridiculous that here we are in 2023 and nobody has completed an open source project that replaces general use CI & CD (for modern best practices & use cases; Jenkins is a curse from the pit of hell). Both CI & CD are just loops of jobs and logic that then decides what next job to run. What those jobs do are all different, but so what? Make an open source library of jobs and logic.
(for what it's worth, Drone.io is the best system I've ever used, and WoodpeckerCI is the open source equivalent, but it doesn't have a large community so work is very slow)
We work in an industry where senior developers don't know the difference between git and Github. The same seniors don't know the difference between Github actions, Travis CI and have never heard of Jenkins. Hardly anyone cares.
Are you going to sell this to a startup? I doubt it. Are you going to sell it to a smb or enterprise company? Maybe.
Is there a market to make cash there? Sure. Is it what I would want to make and try selling? Hell no.
Enterprises with strong, opinionated, tech org leadership, maybe.
The rest are shuffling around between solutions that haven’t been state-of-the-art in 15 years or more, and usually purchased or bundled from some giant vendor or another, as usual.
They’re ahead of the curve if they have any standard widely-implemented CI solution that’s fully in use and not one of three (or more) in various stages of being adopted, maintained, and phased out, in fact.
Why should senior devs know that? Someone needs to know those differences, but on a large enterprise project you have a team that keeps the CI system working. The rest need to know how to checkin a file and how to check the CI output. They shouldn't care about those details.
I expect any senior developer CAN learn the above things. However that doesn't mean they are worth knowing.
Just to be clear, your question is: why should a senior developer -- whose job ultimately involves managing a complex set of text files -- understand basic concepts behind the industry standard tooling for managing ... text files?
Complete waste of time, they should be out playing golf with a board member instead, right..
They need as much as a junior to get the job done. Then they should focus on the hard problems many of which are not tooling related.
However if a different team has made a good ci system they don't need to know how it works, just enough to diagnose a build error they didn't see locally. Likewise, someone needs to know the details of git, but there is no need for everyone to know the details of that mess (git is a particularly bad version control, I wish the world had settled on something else - there are several better choices) so long as you can do the basic commits, merges and checkouts. Most senior developers need to know how to add a text file to cmake (or whatever your build system is), but there is a lot more you can do with a build system that only a few need to know.
Senior devs need to be great at the problem domain they are in. That is normally the subsystem they write code in, and good architecture therein. They need to be able to figure out all the tools you list and more, but becoming experts in them all is a waste of time if enough others already are.
You get customer when you solve a real problem for them. CI-Systems exists, slow or fast doesn't be a problem for most of the devs, cause you run normally everything local and only on publish in a remote CI-system. So CI-Systems exists and solve their problem.
I think only for a really small market the CI-performance is a real pain. Sure, everbody wants it fast, but that is not a reason to change a huge complex setup.
I’m a huge fan of open source, however, if your business model includes an open source model - you need a differentiator. Beyond blazingly fast(tm). You need a reason for people to offer up their credit cards, or better, their accounting PO process. You need a feature you withhold or a service you provide from the open source offering to make it appealing.
GitLab restricts CI/CD to paying customers. Travis/CircleCI limit on build time/credits unless also open source. Azure DevOps is the devil. ArgoCD is complicated. GitHub actions are nice if you have hardware to run the runners. Swarm’s of Jenkins is what most are used to in the enterprise.
As a former Director of DevOps, my immediate question will be what features make me want to buy vs host myself. I’m technically capable. Why should I buy with you when I can run my cloud and my DevOps pipelines for my workforce the way that makes the most sense for my business? Help me make the most sense.
*EDIT* I'm not suggesting withholding features that are critical for the software to operate, but withholding business features that provide support or enterprise-level integration. Even making it tiered so the power users pay less because they support it more themselves?