This kind of logic I would expect only from a bean-counter, and a bad one to be honest... Why would anyone pay $80k/month to solve a problem that could be solved with 0.5 FTE?
Not just that... if I'm working at the IT department of this company (surely they have one) and hear about such deal, I'll have three thoughts:
1. Can I do it myself? Give me a raise and I'll take the extra responsibility. Everyone wins.
2. If they are throwing this much money out of the window, I'll go knock on my managers door and ask for a raise.
3. If they say they can't give me a raise for any bullshit reason, I'll immediately lose trust in upper management and I'll start looking for a job the next minute.
And that’s the sort of thing I’d expect from a programmer! The operational complexity of anything operating at scale extends far beyond the immediate cost. For example, play your scenario out: they give you a raise, you start managing something, and then you decide to leave? Then what? Or, you turn out to be incompetent and make a mistake that takes 24 hours to fix: that’s thousands of people unable to work! That’s going to cost the business a lot more than 80k.
Ownership is a burden, ownership at scale is a nightmare. Paying a third-party to own something for you is fantastic value in the default case, until you have a strong business case for taking on that ownership burden. “One of our nerds says he’ll own it for 50% of the cost” is a terrible option, it’s nightmare waiting to happen.
If I had 10k people, and I could pay $50k to offload ownership of some critical infrastructure to a third party, I wouldn’t even blink. That’s great value.
Well, this is still called Hacker News. Am I in the wrong place?
Anyway... you've created many strawmen here, where should I start?
> you decide to leave? Then what?
My "hiring" for it would imply defining a proper budget for it and a set of conditions negotiated a priori. It's not really "I will just do the job myself", I'm talking about "ok, you are willing to pay $80k/month to have this solved. Here are the 5 other different plans and solutions that we can implement and that will cost less than that, which one gets the go-ahead from upper management?"
> make a mistake that takes 24 hours to fix: that’s thousands of people unable to work
It's still coming out ahead of Github that took 11 days to solve?
Also, what is that joke about HackerNews overloading with traffic whenever there is a github outage? Or that one of how half of the internet GDP being tied to AWS?
Seriously though, the answer would be "you don't migrate everyone at once". You'd start with these migrations on a project-by-project basis, starting with the less critical projects on the new system and slowly weaning off on your dependency of the big vendor.
Bonus: by migrating your systems you will have some kind of redundancy. If GH goes down, the teams could use the opportunity to move to the new system. It it works, the teams gain confidence and can accelerate the migration. It it doesn't, it becomes an opportunity to learn something out of a sunk cost.
> If I had 10k people, and I could pay $50k to offload ownership of some critical infrastructure to a third party
Paying $50k is not giving you any guarantee that your business is robust. You are just paying for CYA.
it was tongue in cheek because you insulted the bean counters :)
There’s not much more I can say because you’ve just outlined an operationally expensive strategy without appreciating the costs. I recommend spending some time in a big organisation, it’s hard to appreciate the enormity of the challenge in orchestrating people until you’ve witnessed it first hand.
Amazon, for example, has thousands of employees dedicated just to running systems to support the other employees! And that’s out of necessity, because third-parties cannot meet their needs — if a third-party could, Amazon would absolutely use them in a heartbeat (as they do already in many cases).
After my time at Amazon, I gleefully pay for 1Password for my org because the thought of what you outlined, in a growing org, would keep me up at night.
> I recommend spending some time in a big organisation
My very first comment started with "I'm really not cut out to work for a big company". I am well aware that I do not want to do this.
> After my time at Amazon, I gleefully pay for 1Password
But that's exactly the type of cultural issue that I am talking about. So many people going to work at FAANGs and when they leave they think that mentality should be applied everywhere.
The first issue in this thinking is straightforward: I know that in your mind your company is the greatest thing in the world, but it is not Amazon. YAGNI applies beyond software development.
The second problem: FAANGs can operate like this because they are making so much money per employee that it simply does not matter to them. But this mentality when applied to a smaller company, can be the difference between 6 months or 2 years of runway. And every time that a company outsources this is a missed opportunity to learn how to do it more effectively. Instead of thinking "this would keep me up at night", I'd rather think "we are doing it the hard way, but that makes us more resilient and increases our chance of survival".
At $15k/month per employee on salary alone, 1Password costs <0.1%. I’d argue the complete opposite of your conclusion: the smaller you are, the more important it is to focus exclusively on the things that move the needle. Reducing my employee cost by 0.1% does not move the needle. If I spend my time debating the merits of self hosting a password manager, I’m not spending my time growing my business.
The reason businesses like Amazon can be successful is because they focus on what matters, and that’s what startups need to embrace too.
It's 1Password, it's Jira, it's Github, it's Salesforce, it's Tableau, it's Google Docs, it's Dropbox, it's Figma. It's all the services that have viable alternatives, but you just don't want to try out because... it's easier to think it's not worth it?
> The reason businesses like Amazon can be successful is because they focus on what matters, and that’s what startups need to embrace too.
That says more about our different views of what constitutes "success" than anything.
I’ve been that IT guy that thinks like you do many years ago. I get it.
But then I became the boss. And we pay for many of the services you listed and more (google docs, gitlab, tailscale, 1password, JIRA, zendesk, Okta, gusto, quickbooks, etc, common tools used by startups). We have 11 employees.
I’ve spent more time typing this comment than the time I’ve ever spent wondering whether they’re worth the cost.
And they all total probably $200 a month per employee or more. And they still pale in comparison to our payroll.
They’re all no brainers especially given how small we are.
Those tools allow us to perform at the same level security and compliance-wise as much larger companies. And they liberate our time so we can focus on adding value to our customers rather than futzing around with unstable internal systems.
Again, I get it, the 80k number seems like a lot. What we’re all trying to tell you in different ways is that 80k is _nothing_ to a company with 10k employees. They probably spend that much on stationary and cleaning supplies.
We have been talking about two separate things, and I reckon they are getting conflated. To be more clear, one is about the "Closed SaaS" vs "Free Software SaaS", the other is "outsourcing vs doing in-house".
My argument is more against the former than the latter. And it's not just about cost. It's about lock-in. It's about shitty customer support that takes 11 days to respond to a ticket. It's about building your systems on the "no-brainer cloud provider" that leaves you empty handed on the semi-yearly outage, and all you can do is pray to be resolved it quickly and console yourself that your competitors are offline just like you.
The point on the second issue was not just "they could do it in-house and cheaper". It is also "they are paying this much money and they are still in a weak position, where they have no control about some critical piece of the organization. I would understand if someone takes this route as a temporary measure while better processes are being developed and putting in place, but if this paying $1M/year is accepted as the "natural way of things" of things, it seems like management is saying "we are incapable of doing it ourselves, and we are too lazy to even care. Let's just hope that the money keeps coming, and if doesn't we just lay some people off".
I realized about a decade ago that I don’t actually care about the code being open source as much. It’s the data and file formats that need to be open.
That way I have a path to migrate to a different product. That’s the true lock in for me.
Never noticed any difference or cared between gitlab being “open source” vs GitHub being closed source. As long as it’s an unaltered git repo and there are apis for my data.
If you use Github only for its repo hosting capabilities, sure. But there is CI, issue tracking, discussion history, the OAuth server and everything else that is built around the repo hosting and is used to lock people into the platform.
>Why would anyone pay $80k/month to solve a problem that could be solved with 0.5 FTE?
I'm not sure who's the bean counter here. Maintaining a service, with the same availability as Github, is 3 FTE minimum, unless you are expecting your 0.5 FTE to also be on call. "I can hire an intern to replace $X" seems like the bean counter attitude. Do you have an SLA? Can management claw your raise back when your bespoke system has more than a day of downtime? If you go on vacation and the system goes down, can management expect you in the office on the next day? Remember they are paying you an $80k/yr "bonus" for three nines of availability. Or are you just going to tell the other 10k employees "hey it won't go down while I'm in Tahoe, just trust me."
> Maintaining a service, with the same availability as Github, is 3 FTE minimum.
Keeping Github is a complex issue, because they are trying to offer the same service to millions of people. It is a totally different beast than having to manage a service to thousands of people. Github needs multiple datacenters in multiple regions. A company self-hosting needs one server and can handle thousands of users. A lot less complexity, a lot less moving parts.
> when your bespoke system has more than a day of downtime?
What is "bespoke" about Gitea? Or Gitlab if you want an all-stop-shop solution? They can also do all the enterprisey things that one expects from Github.
Even if some catastrophe hits the server, your downtime will be measured by the time that it takes to provision a new server (hopefully automated), run some (hopefully automated) install scripts and restore the backup. All this work is a one-time cost, and any decent sysadmin should be able to handle it in minutes.
Here's a bean counter comment: Your self hosted site goes down. What's the cost per hour of my 10,000 users not being able to use it?
The same as the cost per hour of GitHub going down, I agree, but GitHub will have 20 people working on fixing it. We'll just have to wait for you?
What do I tell my 10k users?
How about security? Are you going to manage the provisioning of those users, the access control? How about audits? How about scaling? Your magical .5 FTE will do all this on top of your daily duties?
The calculus is never "just give me a raise" – it's not paying 1Password 80k vs paying some IT person that much. It's just that that money supports a core function for 10k people.
A former-boss once told me when he became a CTO of a very large company – he said, "at our scale, most of our conversations revolve around risk rather than cost. we even start to wonder, will this vendor be able to cope with our level of demands? what if they go out of business? what if they want to quit? what is the cost to our organization in time and money to have to switch to a different vendor/product/etc?"
Who knows, a company of that size might save half that money in cybersecurity insurance premiums just by adopting 1Password. (guessing here...)
Already answered in another place. Github is infinitely more complex to keep up than an internal company service. If Github goes down, can be for a few hours. Or worse, it fails in a way that affects only a handful of users and it takes them 11 days to respond your support ticket.
If an internal service fails, the sysadmin (not the "guy who was brought in to setup gitea") can run a troubleshoot playbook and in the worst case can have a whole new server from the latest backup running in some minutes. Even better, your internal monitoring service detects that the service went down and it can be reprovisioned automatically.
> Are you going to manage the provisioning of those users, the access control?
Sigh...
A company at this size would have already a SAML/LDAP/AD/SSO solution for their other services. You just integrate your service with it.
> most of our conversations revolve around risk rather than cost
I don't disagree. I just think that is just a way to say "I want to pay to buy some CYA".
I fully believe that someone managing a larger organization has other things to look at in terms of priority, but at the same time I think accepting this blindly leads to some complacency.
It's like software developers who are so used to their top-of-the-line workstations hooked to their 3-monitor and fiber internet who forgets that their users might be connecting to their website through a $50 phone on a 3G connection.
There are other ways to reduce your risks. Prioritizing open source is one of them, as that gives some sort of "built-in" protection against vendor lock-in. Giving more autonomy to your departments so they can independently choose their IT solutions to eliminate the chance of systemic risks. Adopting a solution that let's you both outsource or bring it in-house without expensive switching costs. And all of those could be applied no matter the size of the organization, yet it seems that everyone wants to think they are some Silicon Valley unicorn and feel justified in spending ~$1M/year in a password manager.
>A company self-hosting needs one server and can handle thousands of users. A lot less complexity, a lot less moving parts.
Just because you have one server doesn't make you immune to downtime. If the server goes down at 8am EST (5am PST), who is on call to fix it? Or is the server just down for 3 hours until you get into work (assuming you live on the West Coast)? That's what I mean by 3 FTE. If this is core to your company and you require 24/7 uptime, then either one person is on call 24/7 or 3 people doing 8 hour shifts.
>Even if some catastrophe hits the server, your downtime will be measured by the time that it takes to provision a new server (hopefully automated), run some (hopefully automated) install scripts and restore the backup. All this work is a one-time cost, and any decent sysadmin should be able to handle it in minutes.
And if it happens at 1:30am? Or if it happens while that Sysadmin is on vacation? I don't think you are considering that your solution has a bus-factor of 1 for a service that is depended on for a 1000 other people. The idea that an $80k/year cost is preposterous for a core service only applies to people who think humans are fungible, cheap robots who don't sleep, get sick or take vacations.
I'm not saying Gitea or Gitlab are bad products. Plenty of companies self-host Gitlab with their own teams. But the idea that it costs _much_ less than Github Enterprise once you get company sizes of 1,000+ is absurd. We haven't even considered what happens when Gitea or Gitlab starts serving you 500s because your company hits some use case that the OSS developers hasn't thought of. Who fixes that? Now you are looking at sponsoring development of that. Or do you fix that? Can you guarantee any SLA on fixing any bug in Gitea? That company is moving to Ubuntu 420.69 LTS. What's the timeline on getting Ubuntu 420.69 in CI? Is that .5FTE engineering hours?
Sorry, I was abusing the terminology. When I mean "0.5 FTE", I don't mean (necessarily) that you'd be hiring someone to work part-time. What I am saying is that if you have an IT team, the time that should get dedicated to this particular service should be at most 20h/week.
Presumably, your IT team will have many other projects to work on, some of them will be on-call, some of them will be on vacation, etc, etc. But when allocating the resources, I'd guess that 20h/week is plenty to such a service, even for a corporation of thousands of people.
>This kind of logic I would expect only from a bean-counter
I think you're not doing a very good job of updating your expectations based on what other folks are saying. Across multiple threads you have a whole variety of folks trying to explain their reasoning, and you're dismissing them all out of hand. You don't seem to be giving much consideration to the idea that you could be wrong in at least some reasonable organizations (rather than all the folks you're replying to).
> This kind of logic I would expect only from a bean-counter, and a bad one to be honest... Why would anyone pay $80k/month to solve a problem that could be solved with 0.5 FTE?
I think you're underestimating what it would take to solve the problem (even one as "simple" as this one). But, let's assume there is an off the shelf open source solution that is perfect for your environment and requires no changes and integrates easily in your corporate processes without any work (this is already very unrealistic...).
Then, you need to run it and provide 24/7 uptime for your 10k users, which are presumably spread across multiple timezones. Your 0.5 FTE is going to be on call 24/7? Good luck filling that position.
Finally, how are you going to support the 10k users? Is that 0.5 FTE going to answer all the support requests?
I'm not saying it's bad to do things in house - and there are really good reasons to do so. You just have to be realistic. It's going to take a whole lot more than 0.5 FTE to run even a "simple" service like this with the kind of reliability and support your 10k users will demand.
Suddenly that $80k/mo to make it someone else's problem doesn't sound so bad :)
> Finally, how are you going to support the 10k users? Is that 0.5 FTE going to answer all the support requests?
I know I'm cherry picking but the outsourced solution is taking 11 days to say "try again" so maybe whatever that employee takes is an improvement, heh
> Why would anyone pay $80k/month to solve a problem that could be solved with 0.5 FTE?
Depends on the structure of your org. Is the 0.5 FTE being able to maintain addons for all the browsers, dealing with on and off boarding, multiple apps for multiple platforms, write end-user documentation at the same level as 1Password does?
Pay $80k/month to 1Password for an year, one year later you (and all 1Password customers) are still dependent on them.
Contribute $800/month to the vaultwarden developers, one year later you (with the help from all the other companies that understand the benefits of open source) will end up with a FOSS product that can be as good or better than 1Password: https://news.ycombinator.com/item?id=31582369
Not just that... if I'm working at the IT department of this company (surely they have one) and hear about such deal, I'll have three thoughts:
1. Can I do it myself? Give me a raise and I'll take the extra responsibility. Everyone wins.
2. If they are throwing this much money out of the window, I'll go knock on my managers door and ask for a raise.
3. If they say they can't give me a raise for any bullshit reason, I'll immediately lose trust in upper management and I'll start looking for a job the next minute.