We just consolidated all our remote offices into a global corporate account, we all had pagerduty setups for our alarming systems, solarwinds, nagios, email, etc.
I've been using pagerduty for years, and I've been getting other admins to switch from private emails to it, so no more single point of failures. They just had to be shown how simple it was to configure and use.
I won't wake up in the middle of the night to a text, but a phone call, wakes me up. I don't even answer, I refuse the call then open the pagerduty app. Love it.
Same here. I need a phone call to pay any attention. I use UptimeRobot and their RSS feed to pipe into a Zapier Zap which uses Twilio to call me and read me the site that is down in a British voice haha. Easy to setup, works and really cost effective!
I don't think so for my usage and I shared it because it's cost effective and would be acceptable for a lot of devs' projects. I still get UptimeRobots notifications if Zapier or Twilio hiccups. For the phone notifications, if the system fails on a given night it's not the end of the world if I don't wake up. I can fix in the morning.
But I agree for e-commerce or mission critical applications - you need redundancy and it's worth ponying up the extra money for an advanced system.
Yeah, this kind of sound a like a UX bug. Ideally the behaviour could be changed to match the way you’re using it, which is more of a wake up call than a call based notification system.
I feel like the core product is pretty easily replicated with Freeswitch and tailored with a little configuration. Eg: Ring a phone for 30 seconds (so as not to leave a message accidentally), if answered read back the issue (using text to speech), otherwise send the phone a text and <insert whatever else you want to do>.
Having set systems like this up for numerous clients, the latter bit is usually a case of waiting a few minutes for the on call person to either accept the alert, or failing that move onto whoever the fallback is for that person.
The core feature of Pagerduty here is resilience imo. As in, i expect whatever’s issue I’m having that they will not be having. I imagine if an AWS zone is down, large scale network issue, etc, any home rolled solution is likely to be impacted too.
While I get the point, two servers running Freeswitch combined with a bit of python and a regular rsyncing of the user data would give one pretty good uptime for monitoring.
plz stop.. PD should be a cash cow, stop 'innovating' and 'growing', just deliver what it says on the tin. Over the past few years I've been using PD it is becoming worse and worse, it seems to me because they have to keep their developers busy.
They could of course just make the core product better, but thats not cool enough i guess.
I mean, we have no idea what they’re planning to do with the money, it could be used for acquisitions to branch out from being a “1 product” company. Or it could be pumped into sales and marketing. There’s no guarantee that it’ll be used for new features.
Also, I’m a pretty light user of PagerDuty, but what has been getting worse and worse over time? From my usage it’s been reliable software, that lets me set and modify on-call schedules, and notifies the right people when things go wrong. I’ve been at a company that’s been using it for the past 6 years, and the core functionality has been strong/reliable the whole time, as far as I can tell - though again, I’m personally a light user, just on-call a few weeks a year and doing minimal administration. What are the big regressions they’ve had?
I built https://www.pagerline.com/ to be the “core product” that Pager Duty use to be. It doesn’t have an app; you just create a team, invite people, they add their phone number, and when an alert triggered it txts a link to the alert and rings your phone. That’s it. No app. No ticketing systems. No VC pressure to bloat the thing.
Would love feedback from folks on here about it or what they’d like to get out of a “core pager duty” product.
I was responding to the parent comment. The value of tools are diminished as they try and do too many things for too many people. Single purpose applications, that are compose-able and interchangeable are better, then big software suites that try to provide for everything under the sun. The issue with SASS over open source, is cloud companies are constantly trying to attract new investors, and justify new funding but building out new feature sets. It is antithetical to the premise of singular purpose, simple applications.
I remember these guys from the first TwilioCon in 2011. It was some sort of group fireside chat IIRC and they were describing how their initial product was all on Amazon with no redundancy. Of course many of their clients were on EC2 as well so when EC2 had an outage they all had to man the phones and call their clients and say "your site is down" haha. Happy for their growth and success.
I think the huge valuation has more to do with how absurd startup valuations really are[1], and less to do with showing how big the market for a niche product is.
Investors paid to be a part of what Pagerduty can eventually become. Not to have a piece of what Pagerduty is today.
What Pagerduty can become, is much more than just alerting for IT. That's what the $90mm is for.
They hint a little in the article at what some of that future can include: "PagerDuty has shifted some of its focus to doing more with the data it tracks for customers in the future, an area in which it plans to invest."
It depends entirely on your growth rate. In my experience, a lifestyle business growing 20% YoY with 5% churn and 10% operating margin will sell for around 2.5-3x ARR.
20% YoY is very low relative to most of these large SaaS startups that you see on here. The S-1 for Elastic had it at 75% YoY or so. Most of these companies also have absurdly high negative revenue churn. Gross margins approach 75% which become net margins when they stop growing so fast. See the 40% SaaS rule.
Capital market has strong influx which they don't know what to do with. On the other hand they have liability to create money from money. We usually don't realise but investors needs as much opportunity as startups does.
Youre looking at the wrong market. Pagerduty is working “bottom up” in to the IT support game. The trending and business reporting tools are leading indicators of this. Im genuinely curious how long until they have pro services & billable hours. See ServiceNow as an example, $1.9B revenue & $30B market cap.
Uhm, ServiceNow seems to be in a nearly entirely different game than PagerDuty.
ServiceNow is a pretty generic business process and service management solution, with modules for ERP, software development, HR planning etc. And maybe also a module for alerting and incident management.
I haven't used PagerDuty, but it seems it's a pretty focused and specialized application.
Is there any indication that PagerDuty is trying to be the generic business workflow modeling software that ServiceNow is?
Agree servicenow is bigger than just incident & problem management. But they are very big in that area, as a subset of ITSM. As mentioned upthread I definitely see Pager Dutys move away from alerts & messaging as preliminary steps. A lot of their investments in the last few years are around business metrics, process, and visibility. Theyre explicitely talking about problem management, outside the traditional event engagemnt actions folks in here are focusing on. And if youre doing both problem & event management how long until billable hours come in to “reach the enterprise.” And then its a (shorter) jump to generalized ITSM.
So no, I dont think pager duty is going after hr & payroll functions. But the value/market theyre after is much more than just event engagement.
> Amazing how a relatively niche product like this can have such a huge valuation. Shows how big the market is.
Is PagerDuty exactly "niche"? The core product - providing a third-party system that handles the mechanics of paging people during a production issue - is something that virtually all software companies could use, in theory.
It's hard to think of a SaaS company that's less niche than Pagerduty (again, focusing on the market size of the core product, not the actual market penetration or the current state of the product).
> The core product - providing a third-party system that handles the mechanics of paging people during a production issue - is something that virtually all software companies could use, in theory.
I think "virtually" is a bit strong. There are still a lot of software companies that build software that doesn't rely on company-controlled production systems.
Also, in practice companies that have large-scale production systems tend to have dedicated 24-hour human monitoring.
Thus, I do think PD is hitting a niche within a niche: small-medium sized companies that rely on always-up servers.
It's because they can charge a huge amount for their service, where there is relatively low costs of operations compared to other enterprise products. It's amazing how they are able to charge as much as a product like slack.
They passed $100M in revenue. Assuming their growth and other numbers are good, this isn't crazy at all. The only thing that is different is that this was a private funding rather than an IPO. Used to be that companies would go public by the time they got here.
In the first tech bubble, yahoo was making tons of money from ads placed by other startups, who pumped VC money into ads, and VCs spent money liberally since they saw huge returns on yahoo. The whole thing was cyclical.
If Pagerduty makes their money from startup VC money, could a similar thing be happening here?
Thanks. I had genuinely misread that em dash as, well, an em dash. I hadn't seen that you could it like a single word, which is obviously how it was meant. Edited.
Still, the difference is enormous. Your explanation is so much better. Eg you used words like "servers" and "something bad" and "notify the right person". I still maintain that their landing page scares startup founders away and that we can safely deduct they don't make the majority of their money from VC funded companies.
Article says that it's for potential acquisitions. I'm not a banker, but I assume they don't make loans for those kinds of operations. Also, they charge interest. Ultimately, it's a question of what the most effective (cheapest) way to add capital to the balance sheet is. They almost certainly could have raised venture debt (they probably already have an agreement in place), but this was probably deemed a more cost effective way to add capital.
They likely are doing both, raising $ for % and debt additionally. It's not either or, just that debt rounds usually don't get reported/announced. They often happen hand in hand with investment rounds.
The same thing that happens when PagerDuty's AWS availability zone goes down or loses connectivity. Exactly like it did in June. PagerDuty is not magical.
I don't believe I said, or implied, it was. I also don't think it's particularly apples-to-apples to compare a box with a couple NIH Perl scripts and a service with $100mm+ in annual revenues for whom that's their sole job. I promise you, they aren't just in a single AZ.
No, what you did was infer "a couple perl scripts" meant "a box", as though it's not possible to geodistribute perl scripts. The entire point of the post you replied to was expressing amazement that this task could even warrant someone's entire job, because in many, many IT departments around the world, monitoring is both taken seriously and not nearly as expensive as PagerDuty.
It's a valid opinion, and your faux-Socratic response came off as needlessly condescending, that's all.
And if you read June's outage report, they are in multiple AZs... and it didn't help.
You had a couple of perl scripts that integrate with 3rd parties (Slack, Pingdom, etc), manages complex on-call schedules, make phone calls, send SMSes, handles retries, and is fault tolerant across providers? That's some amazing perl skills. :-)
My teammate and I got really excited about pagerduty to replace our phone calling, slack messaging, spreadsheet reading slackbot. But at $29/user/month there's no way in hell management will sign off on it. We are at that point of growth where we need this service but can't justify the risk when we have something ths clunky but basically free.
I wish there were more creative pricing models to get us in the door somehow.
The problem is that our support structure, being a smaller newer company is that everyone grabs a paddle and rows. So for a 50 person company 30 are support on-call staff all with small windows of operation.
Penny-wise and pound foolish. I can't imagine you are so big that $1/day/user is meaningful and less cost effective than spending an hour on your internal system a week.
Unrelated to pagerduty directly but related to the pricing aspect. I couldn't convince the owner of a company that he should spend 50 bucks a month on a service that would literally save him 40 minutes every day. I guess his time was worth less than a buck an hour. The particular software was also amazingly easy to integrate into our ecosystem too.
As an engineer I agree. But I try hard to keep grounded in the irrational real world that I occupy. So I have to respect the reality that I have to work with the irrationality, not against it.
I'll second the comments about pricing. I attempted to negotiate with PagerDuty (after the sticker shock wore off), and their sales team said, "We're Number 1, so we don't really negotiate. Just pick a plan with less features or designate some users as read-only."
Using a very naive calculation of valuation = 5x yearly income, is PagerDuty really pulling in $260,000,000/year? My company is currently paying them about $90/month. or $1,080/year. We are most likely one of the smallest clients they have, but that is still quite a leap.
Well, they're good at alerting. They're not great at incident revie/. The interface makes it pretty difficult to look at, say, "what are all the incidents I got paged for related to this schedule".
In general, the interface is pretty painful to use for large organizations. This is a fairly common theme, though....
> Using a very naive calculation of valuation = 5x yearly income
How did you come up with that metric? Pretty much every company at that stage has negative income. If you mean revenue, that isn't accurate either. Even the largest tech giants are valued at ~8x annual revenue or more, and that number is higher for the SaaS sector. It is common for a smaller, fast-growing company to be valued at 15-20x ARR.
That's not income, that's revenue/run rate. Also, your 5x revenue (while reasonable historically) is out of line with current valuations for peers. A couple of similiar public ones: Atlassian is almost 25 x ttm sales and and Zendesk is 15.
I was a PD customer for 3 years at FarmLogs. There was literally zero innovation or development on that product during the entire time there. It was abandonware. I am shocked that they're dominating this space with such lackluster enthusiasm about their own product.
I'm not just referring to a lack of new features... it’s a lack of improving their core product. It's buggy and painful to use. I can’t tell you how many times acknowledging an issue (during the crisis around the issue) failed and caused cascading problems with other engineers being paged. That’s the one thing this product should do without fail.
I was also a customer for quite a while, and also didn't notice any new features or anything. But I also didn't need or desire any new features, and I can honestly say that of all third-party services I've used as a part of infrastructure across every company I've worked at, PagerDuty is the only one for which I've never experienced downtime.
To me, keeping the product stable and making sure that I never miss a page when I'm having downtime is by far the most important thing. I commend them for having designed and developed a system that appears to be one of the most robust services you could use. It means I'll continue to use them in the future without hesitation or lengthy debates on cost.
The logic/brains around the actual nucleus of the business (calling people when shit hits the fan) is bug ridden. At this valuation and after this amount of time I would expect that to be the most bulletproof and pleasant part of their entire program. It is not.
I agree with your assessment of zero innovation or development.
I help manage a fairly large on-call rotation and we've actively been finding gaps in their product where we think the product should be of assistance. Through support tickets or even hopping on a call with them they didn't seem all that interested in hearing about it or finding a solution for it. I mostly just got a sales pitch about upgrading our plan for new reporting features.
Some of the problem we've frequently encountered:
1. Anyone who is offboarded from the company will just get removed during the next LDAP sync. This just moves everyone up a day for their next shift, no notifications so the visibility is hard. If you try to remove someone manually there is a pre-requisite that they get removed from any schedules they belong to manually.
2. Overrides do not coincide with a specific shift, but a point in time. While I understand why overrides work the way they do, when combined with the problem outlined above, can be a real pain to fix the rotation if someone is taken out since there is no Audit Log.
Here is a scenario that happens:
- Person A is on-call on 9/29
- Person B is on-call on 9/30
- Person C is on-call on 10/1
- Person D has an override for Person B since they couldn't make their shift. Person B now does not have a shift until it goes through the rotation again.
- Person A is offboarded after the override was configured.
- Person B's next on-call shift was moved up one day to 9/29. Person B now is on-call again, with no notification.
- Person C's next is now on-call on 9/30, which has an override from Person D and now is not on-call.
As you can see, it would be beneficial if there was at minimum a notification of a gap in the rotation through automated means, or at least allowing some overrides be tied to a specific persons shift rather than point in time.
We built a tool internally that every few minutes keeps a copy of the rotation and the order of its participants and detects any changes. If it does, it will open a GitHub issue with the user that was removed and the position in the rotation they were at before being offboarded. I often put myself in that spot to preserve the rotation from any breaking changes until the rotation goes through at which time I remove myself.
3. Assistance to find someone to take your shift. Say you get scheduled for 9/30 but find out you can't make it, with one click of the button PagerDuty could email 4-5 people about your shift time and ask if they can take it. Someone can accept it and it'd notify the person their shift has been covered, and others that the request was sent to it has been taken care of. It could factor in any fatigue or length of time someone has been on-call before it includes them in the pool of engineers to cover the shift.
4. Per schedule notification policies. Anyone can change their notification settings or how they're notified. For one particular rotation, we'd like to enforce certain minimums to make sure the push notification is sent immediately, and the engineer is called if not acknowledged in 5 minutes. Currently we cannot enforce that.
5. Audit Log. Who added X to the rotation? Who removed X from the rotation? Who changed the length of shifts? I could add more. For Enterprise level software an Audit Log would be great. They mention they have one internally but don't have plans to expose it for customers.
While their API allows you to build all these tools, having it a first-class part of their product would be wonderful.
I was a PD customer for about the same length of time at my last job, but I do recall a couple of 'innovations'. Incident timelines and incident tracking were pushed reasonably hard, and not at all anything I cared about.
Here's what I'm looking for as a user:
- primary / secondary escalations pulled from a single roster without overlaps
- add / remove people from roster without fucking up who's on call next
But it's pretty obvious they're more focused on listening to fence-sitters than paying customers. Which may be a viable strategy, considering their new valuation.
Honestly I think that's just a result of the fact that enterprise software is a sales driven business much more than it is a product oriented one. Additionally, especially in backend tech products, migrating can require a sizable number of man hours for limited results.
Aside from the lack of innovation, it's got lots of bugs. Sometimes it will continue to call you over and over when things have been acknowledged, or it wont acknowledge things, or you will try to acknowledge something that has already been acknowledged and it will error out.
Because the things we monitor are heavily interrelated, and an outage or degradation in one service will often affect others, it's not uncommon for us to see multiple alerts at once.
The typical scenario goes like this:
1. Pile of alerts goes out.
2. I acknowledge all of them in the course of acknowledging the first.
3. Some of the alerts will decide, despite having state-changed to yellow in the app, that they somehow haven't been acknowledged, so the app will issue the notification tone — for, again, previously acknowledged alerts.
4. Some more alerts will decide they still haven't been acknowledged, and will play the notification tone again, to remind me that I have outstanding, "unacknowledged" alerts.
All of this while I'm trying to fix the broken thing and am being interrupted to attend to the feels my alerting tool has about having alerts.
That is to say: PagerDuty is getting in the way of my fixing the outage. No, I can't just ignore them. We have a global team, and whoever's secondary is probably asleep, because we've taken care in crafting our on-call rotations such that people shouldn't get alerted at 3am. If they sleep through the alerts, they will escalate up my management chain. Then I'm having to explain to my director or VP why there are "unacknowledged" alerts — while I should be fixing the fucking fire.
It's an abhorrently crap user-experience, in a way that is antithetical to the tool's very purpose, and I'm not the only person on my team who has either wanted to throw, or actually has thrown, their phone through the nearest wall because of it.
What exactly does this mean? A "lack of innovation" doesn't tell me anything. It does what it says on the tin. That's usually what I'm looking for from a product like this.
Also true. Basic things like being able to cascade a delete or manage dependencies during a delete is a nightmare.
It truly feels like 6 months were invested into a fairly basic CRUD rails application with a Twilio gem installed and then forgotten.
Good for them to have generated so much revenue with such a weak product. We paid them for years even after we removed everyone from rotation. I am sure lots of companies do this because it is the only thing out there.
Well, this announcement https://techcrunch.com/2018/09/04/atlassian-launches-jira-op... by Jira, 2 days ago, regarding Jira Ops and their acquisition of OpsGenie, might have helped PagerDuty (don't want to downplay PagerDuty's value here) to get that valuation.
I embarrassingly have read the main and about page a couple times and still do not really get it. What are events? Like a critical failure on your server?
Events are whatever you send them. So we have some internal monitoring tools and SignalFX set to trigger pager duty when certain metric thresholds are tripped
I've been using pagerduty for years, and I've been getting other admins to switch from private emails to it, so no more single point of failures. They just had to be shown how simple it was to configure and use.
I won't wake up in the middle of the night to a text, but a phone call, wakes me up. I don't even answer, I refuse the call then open the pagerduty app. Love it.