Im surprised there's still no "trace" view of pipeline execution, especially with how prevalent DAG pipelines have become. Somewhere on the internet I found this handy jq oneliner that will convert a pipeline into a format you can drag and drop into chrome://tracing/ to figure out where your bottlenecks are
FWIW, they had said that they didn't write it originally and had found it "somewhere on the Internet". Searching for "pid: .pipeline.id, tid: .id, ts" has turned up this, which might be the original source (if it is in fact the same script as I am on a phone and didn't 100% check):
Oh, thanks, I read it wrong. Thanks for digging up the source, I want to be sure to give attribution where due.
Fantastic insights in the issue, next to the scripts. One could write a script that generates Mermaid charts in Markdown, and document the CI/CD infrastructure automatically - with CI/CD pipelines itself. Hmmm :)
Ah yup, looks like @saurik pin-pointed my original source (iirc I slightly tweaked the jq, but you get the idea). Please do spread the idea- I would love capability native like this in gitlab.
This (and GitLabs UI in general) does not compare well to GitHub. The most obvious difference is GitHubs better sense of visual hierarchy due to smart usage of shadows, shaded backgrounds, and colors.
But GitHub also understands which information to emphasise, and what to hide. In contrast, GitLabs UI often feels like they scattered as much information as they could, with no thought to placement and grouping. For example, in the before, why is the number of jobs displayed next to the MR, and then the MR is displayed again further down?
They're also often unnecessarily explicit: spelling out "3 minutes, 12 seconds" instead of "3m 12s" and using full sentences instead of definition lists.
From what I see, it highlights a difference in design cultures between the organizations. GitHub has an amazing Product Design organization who have really evolved the product from the early days to what it is today. When I look at GitLab's UI, I get a feeling that it's a UI that feels designed by an engineer without much thought about these design principles you mentioned. It would take a huge overhaul of their design team, culture, and code for GitLab to bring their UI up to par.
GitHub has gotten so much worse over the last few years. Actions feels like it was thrown together with little to no thought as to the user or product journey.
I'm not quite sure how you can say GitHub has a superior information hierarchy when it's flat from first principles. GitLab allows nested groups as you'd expect from any modern filesystem but there's just no equivalent in GitHub.
I've found that philosophy has followed to actions output. I can pretty easily follow the DAG shown in any GitLab pipeline view, but GitHub just gives me a sequential list of jobs on the left and I'm left to just remember whether any of them were dependent on previous jobs else go read the code to figure that out.
Maybe it's just familiarity, as I've self hosted GitLab and its CI for years before actions were a thing, to the tune of nearly 8 years now. Personally, I was really disappointed in the implementation and the UX/DX of Actions as their own brand new construct rather than a DSL for gluing together the shell commands you already run on your dev machine.
GitHub has been getting far more clunky IMO. It takes a very long time to load almost every time I try to go to any GitHub page. Why does it take 2 mins to show what amounts to an 'ls' command? My guess is the ever increasing feature creep of the web.
I find the GitLab UI far better than the GitHub UI. Not that they can't be improve, but it just makes sense for me. Especially the code review approval, where its mostly hidden on GitHub but easy to find on GitLab.
Using GitLab EE at work and GitHub for side projects, the difference is genuinely shocking. I love tinkering with CI usually but using GitLab I actively try to avoid it.
I mean, CI is probably not where you want people to be getting their dopamine hits. I've found that the absolute simplest CI pipeline is the best CI pipeline. Sometimes just a single make command with the right tools preinstalled into a container image is the best. But the more you feel like you've learned something new or accomplished something complex, the more challenging it's going to be to work with your project in local dev and in CICD, and the more work it'll be to maintain in the future as dependency additions necessitate build changes.
Dear Gitlab... please focus on finishing out your parent/child pipeline concepts. I'm very disappointed to see basic CI capabilities missing in 2023. None of the shiny AI things matter if CI isn't solid.
Gitlab CI is so weird man, depending on where you are workflow/rules and so on seems to have very different effects. The obsession it is has making jobs === VMs is also mind bending, why so many virtual machines for one pipeline! You can't consolidate child pipeline reporting up to a higher level - all the data is there in the UI but hidden. There were loads of other things.
Github Actions which is largely free (if you use your own runners) is much better.
Not sure what you're getting at, as I've always found GitLab CI substantially more obvious than actions. CI definitely shouldn't be a snowflake, and it almost certainly shouldn't be something you can't run locally with very similar steps.
GitHub makes it nearly impossible to replicate a CI build locally unless you're super familiar with the actions being used and how they were implemented, because actions are highly abstracted from what is actually happening.
GitLab CI is just a series of shell commands with a container image for context (bring your own tools).
It's usually pretty easy to figure out a GitLab build unless you get fancy with includes or child jobs which imo, for the aforementioned reasons, are substantial antipatterns.
The other things I really like about GitLab's CI is that it's easy to use different images/machines for different tasks. For example, the build of a CUDA-using job can be done on any hardware, but the testing needs CUDA access. Instead of tying up our CUDA machines on boring compiles, the jobs are split on build/test lines so that the CUDA hardware has a better shot of actually being utilized well. Same thing with giving only testing access to things like X11 setups or whatnot.
One can also build once and test multiple ways with this method (e.g., build a ABI3-using Python wheel once and test it with all newer versions of Python with distinct jobs to spread the work and to allow the pipeline to show better things at a glance.
I have no idea how to do such "tee" bits in GitHub Actions (though I haven't looked in a while, the artifacts usage seem…convoluted compared to GitLab's "here's a list of globs"). For that matter, "join"s would be nice too (e.g., to collect the 20 or so wheels built across various jobs into one `twine` upload).
I've been following that Epic and the issue it is based on since 2021 (which is how I found your comment). It's been a roller-coaster which grew from small issues to a giant epic. A few times a week, I get an email about it hoping it's finally a patch for the artifact type I'm using. In the end, I manually roll up all artifacts to the parent pipeline and have a MRs ready in case the feature is released.
There are quite a few issues I follow, sometimes they are very small, sometimes they are huge epics. While I love GitLab and GitLab CI, I find myself hitting some of these frustrating issues nearly every time I do something uncommon.
They usually end up being delivered and well done, which is to the credit of the awesome folks at GitLab, but they rarely take less than a few years.
Have you considered sending a MR for the "very small" things that bugs you? GitLab team member here... I have a personal list of things I wish will be prioritized, but there is only much you can do with the scope the application has and the amount of coworkers. So when things really piss me off, I reserve some time and send a MR to fix that.
That's what I've done before joining the company, so it works ;)
Yeah I finally just stopped watching that issue. I would get a few emails a week that is just some gitlab sales person saying "another customer interested in this" over and over. It's become pretty obvious that it has lost its internal champion, whoever that was. Parent/child was very close to a very cool feature.
Trying to treat CICD as a function interface, that is calling other jobs with inputs that return outputs, is so sketchy to me. I've been tasked with unraveling these types of constructions before and after that experience, I'd personally prefer anything of the sort be deprecated entirely unless it's given dramatically better dev tooling.
At that point you're probably not creating builds, you're creating apps, and you should just use an actual programming language that supports tests and debugging, and pull that into your pipelines as a proper tool with proper controls and its own build.
Gitlab seem to be a company obsessed with feature parity with others rather than making their software make sense and actually work. I wish they would stop following others and parroting features, instead a few conversations with your customers and they would happily tell you where things are not working well!
A problem I've run into with GitLab is that it's quite annoying to view the variables of multiple projects and environments in one place. Projects inherit the variables of their parents, which means that getting a view of which variables apply to a particular project in an environment requires spelunking around the UI and working it out.
A benefit it has over a native GitLab UI is that it downloads and caches the project/group hierarchy in local storage (it's entirely client side), and can be refreshed on demand. It groups variables by environment mask and gives you a view of the entire variable hierarchy for a project.
Do you have an example screenshot somewhere? I'm wondering what exactly this does that GitLab doesn't, because inherited variables are shown within GitLab UI directly – fairly barebones, though, so I'm curious to see your UI in action. (I work at GitLab, but not on this – just a big CI fan myself.)
Unfortunately environment scopes are only available on premium so I can't demonstrate that, but it solves one of the major issues with the existing UI in that it doesn't give you an environment-centric variable view.
Also, it looks like it's only possible to view inherited variables when viewing the vars for a project. Group's don't show the inherited variables for their parents.
I also much prefer the project tree view to clicking around the gitlab UI so much, and when you're configuring a pipeline its quite tedious.
Thanks, sorry for the late response – busy week. Good point, wasn't even aware that the inherited variable view is only shown in projects but not subgroups. On a quick glance I couldn't find an existing feature proposal to add it at the subgroup level as well… but that might be because the search terms aren't terribly unique.
I'll show your demo to the team handling CI secrets, thanks again for sharing! :)
I really wish I was working on Gitlab every day rather than GitHub / Actions. Gitlab’s UI / UX is so much less hostile to the user.
It’s a real shame that Gitlab’s pricing went up so much. I have no doubt that many of the clients I work with that are wasting countless hours trying to add core functionality and reliability to GitHub Actions would be on Gitlab.
Could you elaborate a bit on what you don't like about GH Actions? Just curious as I have a lot of experience with Gitlab CI and relatively less with GH Actions.
Actually, I was recently writing up a list of issues that the teams I work with have with it the other day. I'll share it here however I'll have to post an update to later as it's missing one of my sources.
Github is aware of (and somewhat accepts) every one of these items.
-There are frequent outages (several times a week, sometimes daily) to Github Actions APIs that result in people having to retry workflows that are stuck, slow or fail due to some upstream Github issue.
- There's no way to manage the settings of repos in code (e.g. protected branches, status checks, etc) - this is a huge pain point for us as we have to manually set these things up for each repo and branch.
- If someone makes a comment on a PR and the author pushes new changes to the PR the conversation doesn't show in the PRs diff, nor do other peoples unresolved comments - but they are all still required to be resolved/closed before a merge can occur - this is not at all obvious to the reviewer or the author.
- There's no deployments dashboard to see what is deployed where and what the status of those deployments are.
- There's no concept of application health as part of a repo, the code is very disconnected from what is actually deployed.
- Notifications are a mess - you can't easily see what notifications you have set up and for what repos. You can't manage them in code. There's so much noise in the notifications that it's hard to see what's important - almost all of them end up getting missed.
- There's no way to manage the secrets in code - you have to manually set them up for each repo.
- Linting Github Actions workflows is unreliable even using Github/Microsoft's own VSCode and Github Actions plugins.
- There is no easy to use way of testing Actions locally or on a branch.
- Scheduled workflows only ever run from the default branch.
- There's no internal or private Actions marketplace.
- There's no way to manage the status checks that must pass across a given set of repos for a team, you have to do them all one by one for each branch that is protected....
- There's no concept of grouping repositories for a project, team or product.
- "The console log (which is a critical part of any CI system) is incredibly buggy - often sitting there blank for 10 mins with no output.
- Not easy to audit who has made changes or version control settings in the UI.
- Retrying a job that failed on an upstream action/workflow after it has been fixed - results in the previous version being used again silently without warning and no option to use latest version.
- Custom roles locked down to org admins.
- No ability for engineers to update / replace secrets without granting them full blown admin access.
- You can't set per-workflow timeouts.
- If you're investigating an issue in the console output and the run ends the output is collapsed and you lose what you're looking at and have to find it again.
Is it just coincidence that all the builds in the "after" screenshot for the pipeline lists are MR builds, or have the details of whether a build is for a tag, branch or MR been removed? That is the main thing I look at when skimming that list to determine whether I care about a failed build.
GitLab team member here. Apologies, that is a bug in the screenshots, I checked the latest pipeline tag working in https://news.ycombinator.com/item?id=36605418 and let our teams know to update the blog post.
I would love to know how dev resources are being spent there. Feels like every blog article these days is touting some small enhancement as some big feature. Meanwhile GitHub is pushing out actual new features.
Hey, GitLab team member here. We have a high level roadmap for FY24 in our handbook[1] and our OKRs are listed publicly[2] so you can see our vision for the product and what we're working on.
Appreciate the comment, but just genuinely curious. You guys seem to be clamping down on freemium, but not really landing any valuable "hooks" (product-wise). Would something like a better overall project management experience not be more urgent than a command pallete? Not to be nitpicky, but GH now has a scheduler component on top of custom fields - not amazing, but adds a ton of flexibility.
GitHub and GitLab have so many features, it's kind of unfortunate they you have to pick one or the other. I would prefer a world where these tools were built with the Unix mindset--one tool for each task, that does that task well. That way I could pick, say, GitLab's PRs with GitHub's issues, etc.
In the meantime, I am more than thankful for an alternative to the GitHubiverse, and I am progressively trying to wean myself off it onto GitLabia. I just wish I could have both.
I struggle to see why they’d focus on things like this when missing features like this [1] go unresolved for years and prevent big enterprise customers from migrating…
FWIW if you use Datadog, the "CI Visibility" product allows to query GitLab jobs by name, status, duration etc. (as well as set up alerting on all this).
(One current limitation though is that only finished pipelines show up, no ongoing ones).
At $WORK, someone upstairs decided to ditch Bitbucket to go for Gitlab. Boy do we curse every single day! In comparison, Gitlab UI is way too cluttered, and focuses on being fancy rather than productive. It's even buggy here and there: lines don't line up with line numbers, source lines displayed wrongly indented when they're not, fails to save comments, to list a few.
One thing we particuarly miss from the Bitbuket days is the ability to link to multiple ranges of lines. Gitlab can only do a single range, as if it's doing charity.
If you ask me, they'd have to pay me pretty heavy sum to use their product.
In general it seems like they removed information and made the UI more dense. While the "before" pipeline header did have a lot of wasted space, the "after" makes it harder to visually differentiate things, imo.
Agreed, I hate when UX is ruined just for the sake of design. It blows my mind how these decisions are made. I mean, Gitlab has to have a product team, right? They must not be technical or have experience using the product. Either that or they need to find a new UX researcher to conduct better user interviews earlier in the development process.
The whole blog post is written as though a major engineering effort has taken place, which a) might have happened, but that would be odd from what is shown in the screenshots, or b) hasn't happened, but is the sort of major transformational change that UX consultancies churn 'em out to speak about. And, as you say, almost all of seem almost entirely focused on UI design.
The picture in the blog post needs an update. I just checked a running pipeline that shows the green `latest` label. Added a screenshot of the pipeline, and the update suggestions for our teams into https://gitlab.com/gitlab-org/gitlab/-/issues/414756#note_14...
