> I'd love to hear from someone at GitHub (anonymously or not) what they've done to be satisfied with action Heroku have taken that would allow the integration to be turned back on. My confidence in Heroku to give me accurate information on this is low.
As far as I can tell from Heroku's communications they:
- Have no idea how the attacker gained access
- Have no idea if the attacker still has access
If they do know these things then I've not seen them say so.
> Currently, when you authenticate with GitHub using OAuth, we request repo scope… As GitHub OAuth integration is designed, it provides us with greater access than we need to get the integration working.
> In an effort to improve the security model of the integration, we are exploring additional enhancements in partnership with GitHub…
Github permissions possibilities continually confuse me, but integrations are always asking for more github permissions than I really want to give them, more than it seems like they should need for the integration; I'm never clear in an individual case if this is because they are doing it wrong, or because github doesn't offer granular enough permissions. Some vendors with integrations in the past, when I've complained, have _claimed_ it's because github does not offer any more granular permission that includes what they need.
This announcement still leaves it unclear which it was in this case.
I wonder if the fallout of this thing will result in github fixing whatever it is about their permissions system that is leading to integrations asking for and getting more permissions than should be required?
I have seen most blame over this kerfuffle focused on heroku, but I suspect github's too blunt integration permissions could use some ire, which might help motivate Microsoft/github to improve things.
Having integrated with Github before - for providing OAuth and pulling private repositories - I will say that they've never really had fine-grained permissions. The scopes are here[1] and from what I can tell, I can't ask for private repo access to a _specific_ repository for a given OAuth token. Maybe this is different for a Github App, but just quickly browsing through their docs, I don't think this is the case either.
Any idea how Netlify does it? There I can only select from the repos I have granted access to, and if I want to add a new one I click "Configure the Netlify app on GitHub", which opens a window where I can choose which repos to allow access to. Always wondered how that works.
They actually give you fine-grained permissions, down to the single repo access level – but only if you build a Github app. OAuth app don't offer that unfortunately and I assume are considered a bit "legacy".
Apps let a user specify the specific repos that one can have access to. That’s what we use for our company, tasker.sh.
Furthermore, we basically only ask for the one “mandatory” permission - there are scores of perms you could request when authorizing an app - and that’s just read only access to the code.
I've written a few open source github apps and I've always had to ask for more permissions than I really want simply because Github does not have good enough controls.
Absolutely. I moved several projects from Heroku on to more modern alternatives and was surprised that many were actually just totally free. For example, I had some Vue apps that could be rendered as static sites and hosted on Cloudflare Pages at no cost.
Unless you live in the Salesforce world. Heroku has a couple of really nice sfdx products that make life much easier. Heroku Connect has been a godsend for our company.
As someone who works at a company that used Heroku (I shut down our relatively dormant account) and Travis CI, this was a fun exercise.
This is not exactly related to the topic but in the course of this wider fiasco, we actually uncovered a bug in Github's audit logging.
The Travis CI app was removed at an org level by a Travis employee in the Middle East and to my knowledge, this wasn't publicised in advance so at first glance, it seemed kind of concerned.
Anyway, that org level event didn't actually propogate up to the enterprise/umbrella level. That is, you can have an umbrella consisting of multiple Github orgs and the audit logs are supposed to roll up into the umbrella audit log.
Anyway, we got confirmation a couple of days ago that it should be fixed now but worth a note if you used Github audit logs to respond to the Heroku incident or the Travis CI one
I'm actually impressed that Heroku despite so much backlash refused to enable it until they were certain it was secure. Even if it took forever and no doubt probably lost them significant customers.
My armchair guess is whatever method someone used to gain access more than likely took an architectural change to fix.
My anecdotal understanding is that it has been GitHub who has been apprehensive to allow Heroku to reenable and not something Heroku could be lauded for
The way they handled this was atrocious. It token them a month to identify and recommend secret rotation - after having communicated they didn't feel it was necessary. That can't happen from a PaaS provider.
We were already moving off of Heroku anyways and this just accelerated it. Sad to see what was once a giant (really two giants, if you consider the Heroku + GH integration) fall.
I wonder what the real root cause is, organizationally? Is it corporate apathy as Heroku was swallowed up by Salesforce, and GitHub by Microsoft, or something else? I wish I had a bird's eye view inside the org. For now though, I guess all I can do is move the workload to AWS.
Yeah, just last week I was thinking 'this is pretty amateur hour' as I spent almost a whole morning bringing us back up..
(We are.. a tiny fraction of Heroku, in terms of anything you like - it's excusable IMO that it was an untested procedure not smooth etc., small team with MVPs to ship.)
In 957h I would think you can start to think about bringing on a specialist on contract (or implement the new GH App based still-future version mentioned) if the permanent team can't figure it out / don't have capacity! It's not good for reputation, surely, I have to imagine it was considered low priority rather than something they actively tried but failed to fix for so long, but I don't think that's a good look, even if metrics show it's little-used or only by free tier or whatever.
Sure, but that means deploying was broken for a large number of people, and that they had to spend a non-trivial amount of time setting up an alternative deployment method. For us it wasn't too bad as we already had external CI pipelines (but it still took me most of a day to sort it out). But if you were using heroku for CI then it would have been a much bigger issue.
Is there a service that provides something like Heroku’s PR deployments in a free tier (for open source development)?
While Heroku was out I took a look at Fly.io and Render.com but neither one quite fits the same use case.
Fly does not have automated PR builds and Render does have them but they all use the same database instance and you can’t have more than one active free database (so multiple PRs affecting the database are not possible).
I actually switched to Render before realizing that limitation. It was easy to do and works well but now that Heroku has their integration working again I’m thinking I’m just going to switch back…
Nobody that I’m aware of offers this as a turn-key solution at the same level that Heroku does. It’s a major selling point.
Setting up a well organized PR to preview environment deployment pipeline, that can be automated by CI passing is time consuming and full of yak shaving opportunities.
Just clicking “enable” for a small team is a huge win.
Thanks. I’ll check that out. I liked Fly better overall than Render but I bailed on it pretty quickly when I didn’t see a PR environment option in the settings.
> I'd love to hear from someone at GitHub (anonymously or not) what they've done to be satisfied with action Heroku have taken that would allow the integration to be turned back on. My confidence in Heroku to give me accurate information on this is low.
As far as I can tell from Heroku's communications they:
- Have no idea how the attacker gained access
- Have no idea if the attacker still has access
If they do know these things then I've not seen them say so.