Am I missing something obvious or do these kinds of systems lack a good story for testing?
With Travis and now GH Actions I find the fastest way to develop is to just push countless commits and incrementally develop my work flow in a live-ish way while asking peers to ignore the email spam on failures.
I've tried to get `act` to work several times over the past few months, and never managed to get even a basic workflow to run locally. It kept giving cryptic errors that I had to dig into their code to figure out (e.g. requiring a GitHub Personal Access Token despite not mentioning it in the docs). It gave me the impression that it was far from functional, for now.
I eventually gave up and have been running Actions on GitHub from a branch, merging into the main one once it works.
I use this action* to effectively put a 'breakpoint' in a job that lets you SSH in and inspect the env vars, files, refine your shell script code, etc. Cuts way down on the number of CI run/tweak/repeat cycles.
- if ${{ github.event_name == 'why do you think this is acceptable?" }}
run:|
echo "I need programmatic CI no matter how much you think I don't"
Just let us write our entire pipelines in typescript. People are already using Actions for bitcoin mining, hosting porn, or whatever - if I can do arbitrary shell stuff you're not making my life easier by putting it behind a yaml DSL pretending to be "no code."
If it were just typescript with a reasonable library we could run and debug it locally without jumping through hoops.
I write Python for years (including messing code) and never feel the pain I found when I edit an even a slightly complicated YAML file.
The issue is not really the significant indentation of YAML, it is more of the ambiguity of the syntax plus the fact that YAML is a serialization format so it doesn't have many ways to reduce repetition. Now those issues coupled with the significant indentation makes YAML a mess, but those issues doesn't exist in Python.
I've been making a custom build image in large part because of this. Local testing with Docker isn't the most fun, I have to manually manage the environment variables I pass on the command line, but it's way better than commit then wait for Docker to schedule.
Once it's working locally I push up and it's usually only a commit or two away from working.
if I remember correctly CircleCI is the only CI-provider that would let you run their pipelines/actions locally in a Docker container. That worked really well for me, quick feedback loop without creating commits.
I agree with the other comments that CircleCI is my high-bar for "how can a sane person run the build locally?" because I also agree with the other comments that "act" is handy, but only for the most trivial of hello-world builds, and `gitlab-runner exec` is a cruel joke
Haven't found a better way than `./ci.sh` and do as much work as possible to keep github actions/travis/jenkins/whatever from requiring something different than running locally.
In our stack we use a custom CI runner and that's the image we give to gitlab-ci, so it's easy to just test locally. Not sure what the story is with GHA, but generally I try to make sure everything fits into one script so I can just test that. A more fun/hacky option is opening a reverse shell on the CI box. I was even able to get it working through ngrok to my local machine.
The biggest problem with Actions that I see a lot of people struggle with is that you can’t easily debug things when they aren’t going well.
I think they need to add some way for users to be able to ssh into the worker. Maybe each time an action step fails you have 5 mins to ssh into the container and inspect what’s going on.
I use tmate[1] to get a shell on the worker when I need to debug things interactively. Also act[2] lets you run a decent approximation of your actions locally. Still agree that it should be easier and not require third-party tools/actions, though.
Yeah, the only downside of tmate is that you need to add/uncomment the line with it, push, then remove the commit again.
It's much nicer in circleci, for example where you can simply rerun wish SSH in a button click. Even better would be if the VM was around for a bit after failing, so I could connect and inspect the state without rerunning at all.
Yep, definitely agree. I feel like there are a lot of areas that GitHub Actions needs improvements to catch up to other CI providers, really basic-seeming stuff like "allow failure" support and so on.
GitHub doesn't have exactly that, but it does have self-hosted workflow runners (which GitLab also has, as it happens.)
If you run a local self-hosted runner, point your workflow at it, and trigger it (push/etc), then the job bounces from GitHub's backend over to your local runner, where you can then poke / prod / trace its execution.
It's not quite as "safe" — you don't get to test the workflow in advance of deploying it, but rather have to test it by deploying it. But it's still pretty useful for debugging. You can e.g. set environment variables on the runner itself, that then propagate into the job, to see how they affect the job. Or change stuff in the runner's user's home directory until the job works, to figure out what the job needs there, rather than deploying over and over with different "tweak this file or that" steps.
CircleCI lets you rerun a pipeline with ssh enabled so you can debug. It's helpful unless your failed step is at the end of a complex 20 minute deployment. I'll usually run my docker image locally, copy over the source, and run the pipeline by hand to debug, but then I have to set up all the environment variables that I get from CircleCI/Bitbucket Pipelines, and even then there's no guarantee that my local tests are identical to what I'm seeing in the pipeline (just today I hit a memory limit in Bitbucket pipelines that I couldn't recreate locally).
Oh and their interface is a `curl | sh` that execs whatever happens to be lying around in `/tmp/tunshell/client`. I guess* that is probably fine on a single-user system but I prefer not to run binaries that just about every process on my system can write to.
> I guess that is probably fine on a single-user system
Most of these services run your workload inside a container, so it's not even a single-user system; it's an ephemeral system. It's like anything's going to be there other than the base image + what you put there.
I'm not that familiar with GH actions, but Sourcehut has a couple of pretty nice solutions for troubleshooting: (as you mentioned) time-limited SSH into the build server and a page to edit the manifest online and re-run without having to push anything.
You could create a self-hosted runner to test things out on. I'm not sure how hard it is to match Github's VM configuration but I don't think they veer far from the stock ubuntu config for instance.
Yea setting up Actions is a pretty awful experience. I had to constantly git add, commit --amend, force push and retrigger the run and look at print statements.
I've been playing with the idea of maintaining a repository of common dockerfiles, ci/cd pipelines, and kubernetes manifests with tooling for answering basic question about your app and then generating the skaffolding automatically. It would be kind of like backstage.io in this regard, but aimed at smaller companies that don't have dedicated platform teams. If anyone would use this, I would love to talk.
This is especially troublesome considering that some actions only take effect once you have merged them into master.
So in order to develop an action that is using the on "delete" hook you need to disable CI on master, push your workflow to master, renable CI on master, switch to your dev branch and push a bunch of commits while iterating on your workflow.
With Travis and now GH Actions I find the fastest way to develop is to just push countless commits and incrementally develop my work flow in a live-ish way while asking peers to ignore the email spam on failures.