If you’re wondering if GitHub has been down more since the Microsoft acquisition, I actually just recently did some analysis on that using their status page data (the answer is yes):
Or it could be some of the older employees decided to cash out after the acquisition, and they've lost some institutional knowledge about how to keep things running smoothly?
The addition of interns and new members typically cause more outages, as they make a lot of changes and discover how things work or not.
Retaining older employees has little effect to prevent that, because they typically don't/can't review much of what's happening from a new team, sometimes a whole department, who joined in on the project and has its own agenda.
I was just thinking that Github stayed fairly consistent and Bitbucket didn't really challenge it on features, and then Gitlab came in, shipped loads of popular features and lots of CI functionality, and Github has been delivering a lot of new features a lot faster ever since really, so I think Microsoft acquisition is quite possibly not a root cause of downtime changes.
It's also worth noting that downtime and issues caused during updates in self-hosted Github are significant. I have experienced them, and I guess the main difference is actually that you can schedule the risk, rather than accept it arbitrarily which has positives but comes at significant cost too.
If you’re wondering if this^ website is hugged to death right now, I checked it on downforeveryoneorjustme.com at 11:08 UTC/04:08 PDT/07:08 EDT(the answer is also yes)
How much downtime could it get? They might redesign the homepage a few times and break it accidentally, then it's done. They're not gonna redesign it again and again every week forever.
You under-appreciate how enterprise software development tends to go.. there's always new "features" to be had, KPIs to measure and no time to tackle tech debt...
Hosting your own does not guarantee you 100% uptime. It just means someone less expert than the GitHub folks will be responsible for bringing things back online. It also costs you time and effort, which is a problem if you're on a deadline.
Same goes for the question of where to host it physically. It seems unlikely your physical server will have better uptime than a virtual server in the cloud.
> Why pay someone to mess it up, when I can mess it up for free?
Because it isn't free. Your time is a huge cost.
A senior dev who spends even 10 hrs on standing up a git server has blown through years' worth of GitHub costs, and that's assuming you're even actually using the paid service.
Factor in the extreme security requirements of a code server, including needing to update dependencies daily, and you're spending far more time self-hosting with riskier results.
To add to this, and at the risk of restating my earlier points: even if you get it working, it still probably won't be as reliable as a provider like GitHub.
Keeping it secure is no small thing, especially if you want to permit access from arbitrary IPs on the Internet (rather than using a VPN, say). GitHub does this, and presumably they have solutions in place for everything from intrusion-detection to DDoS protection.
GitHub employs people to take care of server failover and data backups. You could spend your own time building your own solutions here, but they're unlikely to be as good as GitHub's. Your solution is guaranteed to be less well tested.
And that's assuming you even have a server room in the first place. You could run your own Git in the cloud, of course, but you're not really 'running your own' if you do that. GitHub take care of the server question (apparently they use a physical-server provider called Carpathia [0]), and because git always needs to be available but is only used rarely, the amount they charge you is probably less than the cost of running a dedicated server for the purpose.
And all that is assuming that a self-hosted GitLab is just as good as GitHub from the developer's point of view. It may or may not really matter, but GitHub is probably the more polished and feature-rich service.
Building a competitor to GitHub is possible, but not trivial, see SourceHut. (We've been talking about GitHub, but of course they're not the only Git provider.)
I can see only a few situations where it makes good sense to run your own Git/GitLab:
1. Your Internet connection is slow and/or unreliable
2. There are extraordinary safety/security concerns associated with your source-code (military avionics code, say) so you want to run Git in an isolated network (no Internet connectivity at all)
3. Related to point 2: You don't want your organisation's data to reside in the USA. (To my knowledge GitHub don't offer any choice about this, but I could be mistaken.)
For the average developer though, I don't see much upside. Having more control isn't a compelling advantage, it's another way of saying you have more obligations.
I set up a self hosted Gitlab where on-prem was a requirement and it's surprisingly easy. Obviously another service that needs to be maintained and updated but as code repositories and connected services are a crucial part of infrastructure it makes sense to have control over that.
Not the biggest fan of Gitlabs UI but I got used to it quickly.
I find it a bit surprising that it is still a headline, sure it is a huge service used by many. Though as you write its not the first time this year. But apparently still people use a single service as their 'origin'.
So many services rely on github that it has become a single point of failure for online infrastructure as we’ve come to know it. I remember seeing this shared many years ago [1] and not much seems to have changed in the interim. If anything the MS acquisition has only exacerbated matters.
If you let Github become a single point of failure, maybe you are using git wrong? It's named "Distributed version-control" for a reason. I can really recommend this talk by Linus Torvalds about how git is more of a way of working than a piece of software: https://www.youtube.com/watch?v=4XpnKHJAok8
I may use git wrong, but didn't use GitHub wrong. First, Github provides more functions than a git repo. Second, I haven't seen anyone or any company using Git in the way as Linus said in the video. I believe most companies are not either.
I'm sure its 'less bad' now -- but here's an example of an absolutely horrendous security-bug-as-a consequence-of-design-deeply-baked-into-rails ... https://github.com/rails/rails/issues/13420
Interestingly -- the rails developers decided to put in a really horrendous hack to mitigate the common paths through which this design-flaw might lead to unexpected security outcomes ...
In a way, one could argue that the willingness to put in a horrendous hack to 'mitigate' a security flaw provides an example which demonstrates some amount of 'security reasonableness' in rails ...
In reality tho -- I think that this example serves as evidence more for the fact that rails is deeply flawed and very unlikely to be secure in practice -- for reasons of design complexity alone.
There were multiple high profile rails vulnerabilities in the past, such as the infamous yaml vulnerability in 2013 which allows executing arbitrary code in all rails app.
Kallithea[0] has support for both. You might want to test it out and see if it's a good fit for your setup. I enjoyed working with it when when I used it.
99.95% SLA means less than 66 minutes of downtime is acceptable per quarter. I guess they will be offering credits to enterprise customers this time for the first time.
Edit: Correct a mistake but the time calculated is correct.
After Facebook bought Whatsapp it also started having outages at a scale never seen before. Whatsapp used to be a service that was practically never down, after the acquisitions it sometimes had outages of up to a day. Rumours were that it was related to features that FB wanted to add (e.g. video chat). Maybe there's something similar at play here, that Github integrates MSFT services without proper testing?
As few people seem to use Git without those features it would probably be a good idea. Git isn't really decentralized if large parts of the workflow are vendor specific and not cloned.
I'm not sure how active Linus still is in git development, but at least historically, the issue there is that the Linux kernel dev workflow doesn't use any of those features.
That's not really true. Kernel development users PRs, but they're implemented as mailing list threads. They also do code reviews which are implemented as... mailing list threads. It's the most decentralised and vendor-agnostic implementation we can have right now.
So an ideal solution would use mailing list threads as the protocol with a custom UI on top. Same as we use git as the protocol with GitHub eye candy on top. Imagine github as we have now, but all data for issues and PRs are saved as e-mail. Of course, that doesn't benefit GitHub in any way, so we'll never have that.
Browsing through the status page makes me wonder how they manage to have issues like this so frequently. Are there publicly available post-mortems for previous incidents? From a first glance I can't find any on the status page.
Having an electronics engineering background, my personal pet theory is that the convoluted layers upon layers of automagic container management, load balancing and scaling mechanisms act like nested control loops in respect to each other and sudden load increases (e.g. Monday morning load spike) cause the system to essentially produce a step response and it starts overshooting/oscillating. Just a thought tough.
Earlier this year, they were blaming several of the outages on "the database". Not sure if that's still their scapegoat or if they've come up with a new one.
It feels like this is happening a lot more(since quarantine) . It's really weird for how big this company is, how important, and have microsoft's backing. It is a little concerning given how much my work depends on its stability.
It drives me insane that they don’t localize the times on their status page, is there a real reason they are displayed as UTC? If you are looking at a status page chances are anxiety is already high, to throw time zone conversion into the mix too is just obnoxious.
Since turning on resist fingerprinting in Firefox, I am totally used to this problem. I have now memorized and automatically do the timezone math in my head.
Sadly I don't have a real solution for you but at least I have an option that means it's all consistent! :-D
First time I've ever seen it down, I guess there's a first for everything. Now I understand why many people are very paranoid when it comes to taking backups.
Is it worth backing up my GitHub repos somewhere else? What do other people use as an alternative source of truth for their code?
https://downdetector.com/status/github/archive/2020/06, This verifies your data as well, but if you look at the data from Jan '20 to July '19(massive issues in july), there have been more outages. Keep in mind that the stats don't include small outages that last upwards of 2 mins or are region specific. My team and I have been encountering a lot of two min or less kind(usually fixes on a refresh 10 seconds latter). It gets annoying after a while. Its weird given that github hasn't adjusted to quarantine effect.
I think this implied that it never was down for him before when it was reported to be down. Which is exactly my experience, too. I've seen a lot of GH down posts here - it never was for me. Now it's the same.
The code is not the problem; you and other collaborators have a copy of the code locally. It's the fact that I can't submit or update PRs anymore and collaborate with my colleagues that's annoying, and no backup can fix that.
You can always set up a «git fetch» cronjon on your dev laptop/rpi/vps/toaster. I had one on all repos in my ~/git directory to ensure I have up-to-date repos without manual intervention.
https://nimbleindustries.io/2020/06/04/has-github-been-down-...