We're hoping to do a sort-of AMA in the future about rebuilding Sentry (and the open source approach), but if anyone has any questions about the new version I'm happy to answer them.
Sentry looks cool. I have previously tested Rollbar and Airbrake, but was deeply unsatisfied with both.
The big problem I have with all these services is that you need set up each application individually: Each one is a silo, with its own notification settings, API key, and so on. We have dozens of apps (several products + lots of microservices), so this is just unacceptable. I looked into whether I could use a single API key with Rollbar or Airbrake, and simply tag each event with the name of the sender, but it turns out this is not feasible. They don't even have APIs that let you programmatically configure apps remotely.
How does Sentry compare?
(Many other SaaS services suffer from the same issue; for example, Semaphore, which we use for CI, requires a build script for each project. We worked around this by letting the build script be a single curl command that pipes a remote script to bash, shared across all projects. But hardly ideal.)
We're pretty similar in a lot of ways. It's definitely a pain point, but from a UX perspective its tough to solve. In theory you could manage a lot of things with the API (most of our UI runs off the API these days).
Even with the limitations, we have many people with hundreds of projects, and the general problems we see are more around notification settings than anything else. The usual suspect is that people are getting too many emails when they either want none, or they want digests.
We are giving this a lot of thought as part of our upcoming infrastructure pass. From my point of view it'd make a lot of sense if you could just have a massive stream of events and sort through them however you want. That causes a lot of challenges so it may not be feasible.
Either way its a problem we're aware of and looking to address. The simplest solution from our point of view is to allow you to configure some kind of organization-wide defaults. That won't resolve the API key constraints, but we don't feel that specifically is a pain point.
I see. What I don't understand is why settings need to be so granular.
For example, our company uses Slack. All notifications go to Slack. There's no need for per-project settings because it's all just one room. At some point we will probably want some projects to go to other rooms, but that's a coarse-grained exception, not something that warrants a per-project setting.
Also, if you need per-person notification settings, then surely that's part of the user's account/profile, not the project. That would be just a single dashboard page with (say) on/off sliders for each project.
In my mind, the data model used by exception trackers (including, it seems, Sentry) is wrong. There needs to be just one stream of events, tagged by things like application name that you can filter on. How those events map to things like notifications and aggregations and rules and source files and teams and so on is completely orthogonal to the stream. Those things then apply to a subset that you can modify dynamically without having to reconfigure the sender.
This is basically how Datadog works. Unfortunately they do service monitoring rather than exception tracking, but certainly lessons can be learned from them.
Hands down the best designed operations product I've used.
Brian from Rollbar here. You actually can do what you describe in Rollbar: set the name of the app as the 'environment' (configured via application code), and then you can just use a single API key. That's not really what it's designed for, but I think it would work rather well.
I'm not sure when you last looked at our APIs, but we do now have methods for creating and managing projects (though not yet notification settings): https://rollbar.com/docs/api/provisioning/
Stackify works with a single API key for all apps and I think we do a good job with notification settings. Love to see what you think. Thanks Matt with Stackify.
Looks decent. Slack integration? Does it intelligently group messages with similar messages together? Is there a sandbox account I can look at without having to sign up and log test events?
Your logging service seems overpriced, by the way. I'd love to have a central, cloud-based log viewer, but our system is producing many, many gigs worth per day. At the moment our expenditures are about $80/mo for a small rsyslog box, but with your system we'd be paying thousands. For that price I can whip up a bunch of Graylog boxes and get approximately what you're offering, at the expense of having to admin ElasticSearch myself, which we're already doing anyway.
We integrate with Slack, TFS and JIRA. Yes we intelligent group things. From our home page you can click sandbox on the top menu and play around with most things.
We do some unique things like when looking at error show you all the log statements related to the same transaction.
Our pricing is similar to our competitors. But there are self hosted open source options as well, like you mentioned.
A lot of our customers like our product because we combine monitoring, errors, logs, metrics and even apm on one easy to use platform. It's very unique.
David, I've been a Sentry customer for years and I rely on it. But one feature I miss dearly is having a 1 day graph that shows me the health of my servers based on a single count.
A few months ago the top-right dashboard lost its total count and I had to rely on hovering over the graph bars to see what the numbers looked like. I generate a hundred exceptions in a day (on a good day), and I like to use that count to gauge if anything is wrong. A bad day can generate hundreds of exceptions and I like to have an easy way to spot those days.
The dashboard view kind of gives that, but there is no single number of exceptions that I can use. And I greatly miss being able to have that count and graph on the same list of exceptions.
Can you bring back that 1 day graph, complete with a total exception count?
We're definitely going to be improving things like the dashboard (and organization dash). I do agree regarding better visibility into what the graphs represent, and we've opened an issue to track this:
How do you see yourself stacking up against Rollbar https://rollbar.com which has a free tier and unlimited users for paid plans? What makes your product more valuable?
We've been around for the better part of a decade and are completely open source (and free). Everything from our clients (which a lot of the community started) to every piece of the server. The only code you don't get access to is our billing infrastructure.
We're also fortunate to call many of the biggest names in Silicon Valley (as well as outside) our customers.
What you'll quickly find is anything that looks like Sentry, acts a lot like Sentry, and usually was inspired by Sentry. What you'll also discover is that even with this in mind, they are far from the same level of maturity. For example, we've spent a lot of time tuning our grouping algorithms, expanding support for power users, and generally solving problems that aren't obvious until you've invested a lot of time into using the tool.
This is a long winded way of me saying that Sentry is the best in the business.
Rollbar actually wasn't inspired by Sentry -- it was inspired by our founding team's efforts to solve exception tracking internally at Lolapps.
Several of the new features in Sentry (like release management / deploy tracking) actually look a lot like things that have been in Rollbar for a long while, though I expect that is the result of two smart teams attacking the same problem, rather than one inspiring the other.
I already implemented releases a few weeks ago. They're great, thanks for that!
I'm wondering what the reasoning is behind suggesting I upload source artifacts to you as part of a release. The blog post even specifically calls out Go, which is what the majority of our application is written in. So from my understanding, uploading artifacts would allow the Sentry UI to show me exactly where we did a Sentry Dump?
On the same topic, our current deploy flow makes it a bit hard to upload specific code changes per release. It would be great if Sentry could get the code from Github directly (assuming that I login and give you access to the repo)
Thanks for Sentry, we use it daily and would have a much harder time tracing errors and monitoring overall application health without it.
> I'm wondering what the reasoning is behind suggesting I upload source artifacts to you as part of a release. The blog post even specifically calls out Go, which is what the majority of our application is written in. So from my understanding, uploading artifacts would allow the Sentry UI to show me exactly where we did a Sentry Dump?
For Go, you ultimately compile your application into a binary. This binary doesn't have the source code, but it does retain references to the filenames and line number in the binary's metadata. What this means is, if you send along a stack trace to us, we can tell you that information, but we can't show you what the actual line of code is, or what the surrounding code is. By uploading the sources to us explicitly, then baking in the release version into your compiled binary, we can map those up and show you a stack trace that includes source code. This is relatively the same idea with any modern JavaScript. Typically a project is compiled down into a minified version, or run through a transpiler, etc, and the original sources aren't accessible to us. So it's then necessary to upload to us so we can map things up.
> On the same topic, our current deploy flow makes it a bit hard to upload specific code changes per release. It would be great if Sentry could get the code from Github directly (assuming that I login and give you access to the repo)
We have plans for VCS integrations in the future (GitHub included) that would likely work with releases so we can just use the working directory at a current sha.
So right now when we have a need to send a stacktrace (error or panic) we use Raven-Go's NewStacktrace to create a stacktrace and attach it to the packet we send up. In the UI I see the exception with surrounding lines.
What makes releases different from what we already do? Or, do they do the same thing, but the difference is crafting the stack trace in our code vs just upload the code and letting you figure it out?
> So right now when we have a need to send a stacktrace (error or panic) we use Raven-Go's NewStacktrace to create a stacktrace and attach it to the packet we send up. In the UI I see the exception with surrounding lines.
Yeah, so raven-go will check for the source files on disk first. You must keep your source files along with the binary. :)
There really should be a requirement that SaaS services supported some kind of load/dump configuration API, where you can deal with raw config. The admin UI should be completely optional.
Most SaaS vendors don't have APIs that cover all parts of the service. Some do have decent APIs -- for example, we store our DNS records in files and use Gandi's APIs to update our zones, using a custom script (with some nice diffing capabilities), and we do some automatic configuring of Mailgun -- but APIs are always stateful and never declarative, which means that a client will always have to issue half a dozen calls to diff and converge its state. It's painful.
That's an interesting idea. We're Datadog users and one behavior I like is that, while they have a fancy way to build the charts, you can simply click an advanced mode and grab the full JSON configuration (thus being able to copy paste it).
We'll give some thought into this, as it's not too far from being a possibility.
Do you have any numbers about what percentage of your customers running Sentry on their own nodes instead of just using cloud version that you provide?
We do have a beacon that reports anonymous statistics to us as well as giving us an estimate on unique installations. That said, it's an optional feature, and its only in newer versions. One thing that is generally true is many companies don't update frequently so we end up not getting a large chunk of data due to them running on older versions.
If take only our data (ignoring anyone who isnt actively reporting to us), It's a fairly close 50% split of on-premise companies to hosted customers.
That said, our estimate is its more around 30% of users run off of getsentry.com, and the other chunk runs on-premise.
We wanted to but nothing really handled tokenization well. In the end we wrote our own (which is why there are various caveats).
For other things we're using select2. At one point we had tried to switch to selectize but we found limitations in its API and frankly didn't want to spend the time to find ways to work around them.
We do have some support for Java, but its unfortunately heavily tied into Logging right now. Our short term goal is to at the very least ensure we have proper documentation (and support) for getting things up and running in Clojure and Scala.
Our goal is the next 60 days, but the realist in me is saying "sometime this year". We have an ever changing list of things we want to accomplish still as part of the 8.0 push:
Regarding future capabilities of Sentry, we're not quite ready to talk about most of it publicly yet, but we'll making some drastic improvements in the very near future.
> Awesome, I've just upgraded our instance to 7.7.0 - when will 8.0 be available on Pip?
"Soon", but you can install it from the github master already. Since we generally deploy right out of master the upgrades are fine to and from there as well.
Of note, we're actively hiring someone to come help us expand our mobile offering. If you're in SF (or interested in being) and like what we're doing, take a peek:
@David, after upgrading from 7.7.0 to 8.0.0.dev0, sentry health api /_health/?full=1 reports
{"healthy":{"CeleryAppVersionCheck":false,"CeleryAliveCheck":false},"problems":["Background workers haven't checked in recently. This can mean an issue with your configuration or a serious backlog in tasks.","Celery workers are referencing a different version of Sentry (8.0.0.dev0 vs 7.7.0)"]}
and the website has two red bars for these warning, but the reporting mechanism seems work well, and there are no warning/error records in logs.
Do these warnings have serious problem? how can I fix these warnings? (I've totally removed old sentry files in /usr/local/lib/python2.7/site-packages/sentry* and made a fresh installation)
Same goals, different approach. Our JS support is extremely mature at this point and you'll find that out of the box we'll do a great job at catching errors in your application. In many situations you won't even need to do anything beyond toss in a couple script tags.
That said we also support a lot of advanced scenarios:
Sentry's sourcemap support is one of my favorite features. After spending years poring over cryptic minified exception messages, I finally set up Sentry for my latest JS frontend. It's glorious to get a nicely formatted traceback containing actual source code lines.
I'd really love to be able to use Sentry to collect errors in a closed .NET web app that just spams a log file. Is there a collector/raven for just tailing files (on Windows), configurable via RegEx or other parser definition? Right now we use a custom grok filter for Logstash and dump it all into Elasticsearch, but the Sentry UI sure would make life easier when something goes wrong and we have to figure out why.
There's not. The primary reason is that Sentry doesnt really match up well to traditional logging. A lot of we do involves highly structured data, and simply wouldn't be usable without that.
It helps to think of Sentry more like classic crash reporting more so than modern logging. We want (and need) to know precisely what a stacktrace is, which piece is the function name, the line number, etc.
Well that's what the app is logging. Full stack traces. ;) I can think of lots of apps that do just that. If a formatted stack trace is truly what you need to ingest, maybe I should look into contributing a "Raven" for generic systems that log stack traces.
We're hoping to do a sort-of AMA in the future about rebuilding Sentry (and the open source approach), but if anyone has any questions about the new version I'm happy to answer them.