The issue isn't tooling, it's hardware resources and in some cases licencing.
While the problem isn't inherently trivial, the issues the tooling can solve is, which is the order of startup and is usually solved with a waitforit startup script, as these services generally talk over network with each other.
The real challenges, like determining seed data etc is too project specific to be abstracted away.
Not to take away from your problem domain, it would be nice to have framework to use so the developers don't have to wire the plumbing for these automations manually anymore - it's just not going to solve the issue that developers will have to wait for the ci pipeline to get their results
> The issue isn't tooling, it's hardware resources and in some cases licencing.
Hardware resources are definitely an issue. That's why we generally recommend using remote development environments, which aren't as resource-constrained as the local dev machine. Making that comparably smooth to the local dev experience (e.g. for live reloading of services without rebuilding containers) needs some clever tooling (which is partly the reason we're building our product).
With production-like remote dev environments, you get the same capabilities as your CI environment, but can run test suites ad-hoc (and without having to spin them up and tearing them down for every test run).
There's no fundamental reason why CI environments should have capabilities that individual dev environments can't have—it's all a matter of automation in the end.
> The real challenges, like determining seed data etc is too project specific to be abstracted away.
Very much agree with that! The generic stuff (dependencies, parallel processing, waiting for things to spin up etc.) should be taken care of by the tooling, but without constraining the project-specific stuff (which is highly individual).
> Hardware resources are definitely an issue. That's why we generally recommend using remote development environments, which aren't as resource-constrained
On the contrary, I almost always advocate for having a rack full of machines in a/the office, running some kind of workload management (kubernetes, vmware/proxmox or a combination of the two).
Hardware is dirty cheap, and plenty fast these days.
If you have an office, chances are you already have a server room anyways (enterprise grade network switches require cooling anyway) you might as well throw a bunch of physical machines in there.
The only real issue I see is that most developers have literally no idea of the runtime resources needed by their code, for a number of reasons (like runtimes hiding that kind of information and the general mindset that pushing out new releases/features is more important than tuning existing ones) so in the cloud developers will just provision bigger and bigger environments. It's all fun and games until two things happens: (A) environment provisionin in the cloud will take long times just like on developer machines andb (B) company earnings will be eroded by cloud infrastructure bills (on-prem infrastructure OTOH provide tax shielding)
I would expand your point to add that another missing key part of dev on big distributed platform is being able to run parts of the system locally.
For some shop it is a lot harder than a simple docker-compose (think of envs with 10 or 100 of micro-services), any laptop cannot handle such load and it is critical for devs to work on the machine, otherwise you lose a lot of time with shared dev envs, SSH tunnels or rsync...
I agree that CI pipelines are a pain in a distributed system but I heard more complains from devs not being able to test locally some new feature in serviceA that depends on 10 other services + DBs to work.
Running locally is great but I would already be happy if I could step through a CICD pipeline with a debugger. This includes stepping though the services the pipeline calls. Also include breakpoints.
While the problem isn't inherently trivial, the issues the tooling can solve is, which is the order of startup and is usually solved with a waitforit startup script, as these services generally talk over network with each other.
The real challenges, like determining seed data etc is too project specific to be abstracted away.
Not to take away from your problem domain, it would be nice to have framework to use so the developers don't have to wire the plumbing for these automations manually anymore - it's just not going to solve the issue that developers will have to wait for the ci pipeline to get their results