Hacker News new | past | comments | ask | show | jobs | submit login
Show HN: We’re open-sourcing our session replay tool (github.com/highlight)
408 points by podoman on Feb 22, 2023 | hide | past | favorite | 71 comments
Hey HN! We’re open-sourcing highlight.io (https://github.com/highlight/highlight), a session replay and error monitoring tool. Highlight.io gives you a high-precision video-like replay of what users are doing when an error or exception occurs in your web app, along with a full-fledged error monitoring experience (similar to bugsnag, rollbar, etc..).

The main value prop of highlight.io is that we help you understand the full context surrounding an error and allow you to drill down to the code path that a user invoked (i.e user clicked button X, sent network request Y, and backend code Z was executed). Some of our customers compare this to a “web debugger” of sorts. A picture of what this looks like in our app is here [1].

For some background, when we worked at our previous companies as engineers, we encountered hard-to-reproduce issues spanning across both the frontend and backend. The main issues were (1) if a customer complained about a problem, it was hard to reproduce the issue without asking for a screen-share or jumping on a video call; and (2) when viewing errors caught by tools like BugSnag or Rollbar, understanding the triggered code path required stitching together logs, errors, and trace; all from different sources.

Highlight.io is completely open source and written in Go and Typescript. To build the replay capability, we use an open source project called rrweb [2] and have worked closely with their team to add support for features like canvas recording, shadow dom recording, and more [3]. Beyond that, we use the OpenTelemetry spec for our SDKs [4], which has made it pretty straight forward to support several languages, even with our small 4-person engineering team!

Our product is completely self-serve at app.highlight.io. Installing it is as easy as a npm/yarn import and installing the backend sdk of your choosing. In addition, given the privacy-centric nature of session replay, we also offer the option to self-host [5]. Highlight.io currently makes money off of our hosted offering, and our self-hosted deployment is completely free. We’re also toying with the idea of an “enterprise” self-hosted deployment, similar to gitlab’s billing model, and thoughts from the community on this front would be appreciated!

And as far as what’s next for us: Our customers are asking to render logs and traces on a highlight.io session (and vice versa), and we’re excited to be going deeper into a developer’s debugging stack. The long term goal is to build a platform that connects replay, errors, logs and more so that engineers can “playback” the full state of a web application.

Overall, we’re quite new to the open source scene and would love the HN community to share their feedback on what we’re building. If anyone has opinions on where we’re going, or what they’d like to see in an open source monitoring product, we’re all ears. Check us out at highlight.io and at github.com/highlight/highlight to give us a shot.

[1]: https://www.highlight.io/docs/getting-started/frontend-backe...

[2]: https://github.com/rrweb-io/rrweb

[3]: https://highlight.io/docs/general/product-features/session-r...

[4]: https://opentelemetry.io/docs

[5]: https://www.highlight.io/docs/general/company/open-source/se...




Interesting, I hadn't heard of Highlight.io. Thanks for sharing your code!

How does your session replay feature compare to LogRocket (which I've always found quite impressive)? (Side note: I'm missing proper screenshots and/or videos on your website.)

Regarding your pricing: Am I seeing this correctly that, in contrast to LogRocket, you're not charging ridiculous amounts for team seats? This is what drove us (web shop, 30M user sessions / month, ~150 developers) away: For 100k sessions/month and 100 user seats they wanted $30k/year of which $21k were supposed to be just for the user seats. ( ._.)

Now we didn't expect 100 of our developers to use the tool regularly (as in every day) but since LogRocket told us accounts could not be shared or rotated and we expected many devs to access LogRocket at least every now and then, that price was beyond unacceptable.


> How does your session replay feature compare to LogRocket

Forgot to reply to this. We've actually had quite a few customers switch off Logrocket because of our support for things like Canvas recording (https://www.highlight.io/docs/general/product-features/sessi...). From a session replay perspective, we have feature parity, i.e. we report everything in the devtools and even report network request/responses. We also support backend error monitoring and pairing those errors with network requests, for example, which is something logrocket doesn't do.

But overall, logrocket is very comparable tool, but as we mature, we're looking to become a more generalized monitoring tool (w/ logging, traces, etc..).


Thanks so much for your detailed response! This does sound very interesting!

If I may ask a few more questions, out of pure curiosity: What are, in your experience, the biggest challenges when it comes to frontend error monitoring from the point of view of a company / a developer using it? How can these be addressed? And what are good criteria to evaluate frontend monitoring services by these days?

(Obviously, you'll be biased here but that's alright. :))

EDIT: I should add: Obviously I've got some experience when it comes to frontend monitoring (though not a ton) and could – in theory – partly answer these questions myself. However, sometimes I wonder whether the problems we face in our day-to-day are the "correct" problems. (I.e. is everyone else struggling, too? How do other people solve their problems? How do other people choose their frontend monitoring solution?) Hence my curiosity.


I think we see "frontend monitoring" in two different buckets: "metrics-oriented" and "incident-oriented". Metrics are about things like lighthouse scores, optimizing your frontend bundle, etc.. which tend to only become relevant for larger companies as only at that point do these things affect conversions to a significant level. Incidents, however, are more about one-off issues happening in your web app, and that' what we're focusing on at highlight.io (at least right now). Examples include customer support issues, bugs that affect a user's experience, etc..

Both are important for larger companies, but for smaller companies, the only thing that we see as relevant are frontend "incidents".

So to answer your question, I'd ask yourself what types of issues are you trying to track down? Does that make sense?


My question was more about the error/incident monitoring part. For us the biggest challenges in this area have been

1) grouping errors correctly. Often, identical errors are not logged because the browser messes up the stack trace, because error messages are not completely identical (browser-dependent), or because the stack trace (line numbers etc.) have slightly changed from one version to another. So we often need to group errors by hand.

2) identifying what the impact of an error is. Is it critical? Is it enough if we look into it next week? Is it relevant at all? Add to that that our web shop sees a significant number of automated interactions by resellers, so errors due to broken browser extensions, their bots doing crazy things etc. have been quite common and often need to be ignored by hand.

I really hope session replays are going to help us with 2).


In my experience, the biggest problems we have is that it’s hard to figure out what exactly to log. There’s sensitive information in the system that we are not supposed to store outside designated locations.

Then there is session storage itself. Do we log every request? Do we log even very large responses?

We’re using https://openreplay.com


Yep, we don't charge for seats at all. Just usage. In fact, we're likely going to be adding a lower tier so that individual developers can use the tool (at about $50/month). With the direction we're going with the product, we see ourselves having several "products" long term (error monitoring, session replay, logging, etc..) so there's a lot of value in making smaller teams happy w/ pricing, even as they grow.

And if you ever give us a try, definitely send feedback our way; we have a discord community up at https://highlight.io/community.


I wanted to say thank you for picking a business friendly license!

Watch out for the (presumably old) GitHub organization in .gitmodules: https://github.com/highlight/highlight/blob/d818969acddd7c74...

OT1H, GH does a 301 to the new name, but OTOH unknown how long that lasts and is one more thing to go off the rails for fresh clones. It has been my experience that using the https URL makes clones easier for users which may not have set up ssh auth, but it depends on the target audience and whether you have some internal tooling that depends on that submodule coming down over ssh


That's a great point! Just opened a PR to fix this: https://github.com/highlight/highlight/pull/4362

Switching this to https too since the our rrweb fork is now public (ssh reference was previously necessary for cloning a private fork).


> I wanted to say thank you for picking a business friendly license!

Is not AGPL business friendly, if not why? Only restriction that I know is you cannot provide it as a service. Databases like ScyllaDB are AGPL and they are doing fine.


What do you say to potential customers who are concerned with the legality and privacy challenges associated with session replay?

I recognize how tools like this could level up product/marketing-level decision-making. But with litigation as recent as October 2022, I'm wary of incorporating something like this into any software I control.


In the past, for customers in healthcare and fintech, we've had to actually pass on signing more restrictive contracts (BAA's, etc..) due to the liability. For such a small company, it was hard to be comfortable with situations like that.

Since then, we've done a couple things that will hopefully open up more opportunities to work with these sorts of companies. First of all, we have a "privacy by default" mode that obfuscates all text by default (see https://www.highlight.io/docs/getting-started/client-sdk/rep...). And secondly, we're hoping that open sourcing the tool will help these companies control the data they're collecting so they don't run any risks, and long-term, these companies pay us a license of sorts.


I think this is a great question, especially with the continued shift towards data residency and customer protection in recent years (GDPR, CCPA).

When we did this at Sentry - which is also built on rrweb (we help fund rrweb btw, everyone else should too!) - we realized privacy was really hard to get right (let alone data security), so we just block all elements that are at risk by default and it turns out the experience is still pretty usable. Ben wrote a bit more about this here: https://blog.sentry.io/2023/02/16/introducing-session-replay...

Not commenting on highlight, but most of the mainstream players in this space take an opt-in protection approach, and often its "you can block this specific HTML element".. which glhf getting right


Highlight also has a private by default mode. Totally agree that this is the right way to do it. Also, congrats on the recent launch :)


I see[0] your session replay feature also includes network requests. Do you fully capture request & response there and, if so, how do you handle the arising privacy issue? Does the customer have to set up filters by hand?

[0]: https://sentry.io/for/session-replay/


With the highlight.io client, we use an opt-in strategy to be privacy-first (since network request bodies + headers are often sensitive). We're not capturing network requests/responses unless you enable it and allow deeply configuring what should be captured [0].

[0]: https://www.highlight.io/docs/getting-started/client-sdk/rep...


We have a ton of filtering, but great question. I will ask someone from the team to respond as I'm not sure how well we do on this particular thing and/or what we are capturing today.


Right now we just capture network metadata like URL, connection type (fetch, xhr, websocket, etc), status code, filesize, etc. We additionally filter this data at the edge (before anything gets stored) for any known unsafe values like credit card numbers or SSNs, in case they somehow get into the URL.

As of right now we do not capture actual network request headers or bodies.


In addition to the network call data we already attached, that does go through data PII scrubbing on the edges as you mentioned, we're discussing capturing request/response body. We even talked about it today: https://github.com/getsentry/sentry-javascript/issues/7103

Avoiding any PII from getting into the system is always top of mind. And we're considering this as an opt-in regardless of scrubbing on the edge ingestion service.


GitLab team member here.

Thanks for sharing!

> We’re also toying with the idea of an “enterprise” self-hosted deployment, similar to gitlab’s billing model, and thoughts from the community on this front would be appreciated!

Transparency and an open stewardship of the project can help: https://about.gitlab.com/company/stewardship/ Make it clear what features are paid, and define that for example free features are never moved to the paid tier.


That's a good point; we should be clear about what will always be free. Right now, literally everything is free, but what we'd consider putting behind an enterprise tier that are top of mind are: - multi-tenancy - enterprise authentication features (sso, saml, etc..)

Re: the gitlab link you sent, we really like the transparency around not making a free feature paid, and the fact that all tests are OSS.

Overall, we'll work on something like this. Thanks for pointing it out!


Please don’t put basic features behind enterprise licences.

We are a small start up. We use SAML / SSO. We care about security. We care about privacy.

None of this is “enterprise”.

We stopped using competitors to you because of data privacy issues.

We would happily pay for a self host version, where we control all the data recorded.

These type of tools are extremely useful, and it hurts not having it. But they are just way too much of a privacy issue.

Self hosted and paid - is the future of data privacy.

I would love to see you (and others) off these options, it’s really a game changer and makes the switch easy from any product that is not self hosted and collects our customers data.


> Please don’t put basic features behind enterprise licences.

> Self hosted and paid - is the future of data privacy.

So, you want a paid, self-hosted version - not named "enterprise"?

Much as I dislike the gitlab decision to gateway sso - there's no problem paying for and self-hosting it?

Or are you making two arguments - keep features like sso foss - we will happily buy paid self-host version anyway?


What do you think of uxwizz.com ? The only option is self-hosted, because focusing on the hosted version would mean competing directly against the self-hosted one, so making self-hosting harder would drive more revenue, which seems wrong.


That's great feedback; understood. What features would you consider "enterprise" in that case? Right now, nothing is paywalled in our self-hosted version, and we're still very much in the brainstorming phase here.


how companies should make money, if AWS can package it any day and sell as a service?

regarding self hosting, how can contributors make sure that you are self-hosting and paying? If code is open and already in your infra there is no way to enforce anything.


> regarding self hosting, how can contributors make sure that you are self-hosting and paying? If code is open and already in your infra there is no way to enforce anything.

The way we're thinking of addressing this is not including the enterprise features in the output binary when you install highlight.io. This makes it a bit harder to actually build the project with all the enterprise features. It it something that folks can take advantage of, though.


I would encourage you to consider not even trying to cripple the freely available self-hosted version. Personally I'm not a fan of open core or the GitLab business model, and Sentry has shown you can build a business without removing needed features from the self-hosted version. While folks may not agree Sentry is "open source" these days, our license effectively achieves the same from a customer pov as yours (which is ideal for customers). More people should take the risk IMO.

Ultimately people do not want to run all their own services if they can pay someone else to run it for them.


Makes a lot of sense. We're going to try to take a bit of a different approach, but no doubt that a successful business (like sentry) can be built w/o the gitlab model!


Keep https://sso.tax in mind


Will do. Sounds like other folks on the thread feel the same way, and we’ll keep this in mind.


SSO and SAML are not enterprise features. They are table stakes security wise, and intentionally crippling your security model is a good way of driving off customers that take security seriously, no matter their size.


Absolutely, makes sense. We'll make sure to make all security features we ship a first-class citizen of the open source version and available to all customers.


Please don't put SSO/SAML behind a paywall. Security should be a free priority.


At the end of the day security costs money. Should security consultants offer their services for free? Should security guards work for free? Should DDoS protection be free even though it uses expensive firewalls and bandwidth? Should security audits be free? Should badge scanners be free?


Adding a "login by google" button costs a few hours of developer time. I've done it myself, it isn't a whole lot of work.

On the other hand, I'm dealing right now with two services (NetSuite and Certify). Both "support" SAML, but they offer no direction on how to set it up or any sort of support for the setup. The documentation that is out there, is out of date and/or incorrect. Even though they offer these as paid additions, they don't really support them.

Anyway, that's where I'm coming from. I'm not trying to get into some deeper meaning debate here.


Super interesting project! I'm working on a related/similar project that I'm curious about the technical aspect. Would you mind sharing how are you simulating the reconstructed screen? Do you construct the DOM, together with the CSSOM, on every browser events, and send the result to the backend for replaying?

And do you monkey patch the fetch API to intercept request? Apology for not looking thorough enough on github!

How would you compare yourself with tool like openreplay? Disclaimer, I am not related to them.

An HN comment is not completed without a unsolicited feature request: a demo site, maybe using highlight.io itself, and let user replay their interaction.

Good luck building!


Hey! Sure thing. Our frontend session replay happens in two parts:

1. The HTML DOM Recording using rrweb (https://github.com/rrweb-io/rrweb) 2. The monkey patching of fetch, xhr, console methods, combined with data from `window.performance`

The rrweb repo has a good in-depth explanation of how the DOM recording works. In short, it captures the full HTML of a page when a user first loads it, then tracks DOM changes via the `MutationObserver` api. From our client, these are all shipped to our backend as serialized events. We process the events per 'user session', storing them temporarily in redis, then compressing a permanent payload once the session no longer is sending new events in the local filesystem or S3.

Monkey patching network and console methods allows us to capture request/response payloads, headers, status code, etc. We then combine that data with the `window.performance` api's notion of network requests to ensure we capture all requests (even ones that happened before our monkey-patch had time to apply), as well as to get precision timing data.

Happy to give more detail about parts of our stack if you're interested!


Following up regarding the other points I missed. Compared to OpenReplay, we have similar functionality in our session replay, but we've focused a lot of effort on making a cohesive error monitoring (backend and frontend) experience. We closely link sessions with errors (stacktraces and associated metadata) and vice versa to make it easy to get to the root cause of a bug.

To demo your own sessions on highlight, you can easily spin up our next.js example app that has highlight installed: https://github.com/highlight/nextjs-13-sample. All you need to do is make an account on highlight.io and copy your project ID over into the sample app!


Thanks for the follow up. Definitely checking up the demo. All the best to your endeavours!


> We’re also toying with the idea of an “enterprise” self-hosted deployment, similar to gitlab’s billing model, and thoughts from the community on this front would be appreciated!

First, Thanks for sharing - there might be a few interesting applications in my (non competitive to you) world I wouldn't have considered for the sheer amount of work to begin with.

My experience with B2B on prem has been that maintaining first Class On-Premise support will have an outsized growth vector for you. It's easier to do from the start than to try to shim in. If you really want it to be an unfair advantage and moat, being present as a first class verified citizen in the marketplaces of multiple clouds is good too for those groups who might want it.

For example, being ready to roll (and license for enterprise) int he Microsoft Marketplace is a bit of a superpower, because they already have a good chunk of the Fortune 500 already as customers, so you don't have to go get them and be close to being a one click install where an on-prem m365 environment might allow things from Azure directly or indirectly.

The harder part is making sure enough of the useful and even valuable bits aren't locked away from the spirit of open-source, or critical ones committed moving forward. It's hard but the hope is the magic of creating beginners for your platform remains the chief priority for growth and adoption for your system - people readily available and skilled in it.

One nice way to do this is to perhaps consider some creative licensing around headcount of an organization, sector, or yearly revenues. Appgyver had an interesting approach to this prior to being acquired by SAP and seem to have been able to maintain it. Still, some folks do well with an AGPL type of license to minimize competitors which I respect a groups right to do for their own sustainability.


Could you tell me how this compares with other open-source session replay projects such as: 1) OpenReplay (https://openreplay.com/) 2) PostHog (https://posthog.com/)


Having used the Posthog product (which is really good btw!), its an "analytics" tool, so their replay product is very good at mapping user funnels to sessions.

On the OpenReplay end, its seems we have feature parity for session replay, but they don't have support for error monitoring and therefore linking errors to corresponding sessions (as far as I can see).

Nonetheless, happy to see other oss tools in this space!


PostHog started as a product analytics tool but we now have a fully fledged session recordings product including handling console logging and network requests for debugging. Mobile support is coming shortly too

Disclaimer: I'm Head of Product at PostHog


Very cool. Didn’t know y’all were investing in that. Always been a fan of posthog.


Thanks! And congrats on the move to open source


Thank you sir!


I didn't think postHog was a good comparison either,

I knew of rrweb https://www.rrweb.io/

Great to see more open source contendants in the space


Thanks. Yeah, we actually use rrweb under the hood and have met with a few of the core maintainers. We're proudly powered by that project.


Posthog also seems to be using rrweb.


Congrats on the open-source move! It's always good to see more players in the OSS space. We do track JS errors at OpenReplay and of course link them to recordings. Errors come with stacktraces, are also fully searchable and can be used as cards to build custom dashboards. In terms of backend errors, we took a different approach at OpenReplay. We pull them from third-party systems (i.e. Sentry, Elastic, CloudWatch, Datadog, etc.) and sync them with the corresponding replays.


On debugging side, sentry (recently?) rolled out replay too https://docs.sentry.io/product/session-replay/


We launched this last week! Right now it's only available on SaaS (sentry.io), but we've been developing it in public in our main server repository (github.com/getsentry/sentry) and plan to have proper instructions for self-hosted users soon.


Hi Ben, thanks for responding. I wonder what approach are you taking in serializing DOM changes, is it similar to rrweb [1]?

[1] https://github.com/rrweb-io/rrweb/blob/master/docs/serializa...


Sentry also uses rrweb!


PostHog, Highlight and Sentry are all using rrweb. At OpenReplay, we use a full proprietary solution that we built from the ground up and covers all session replay aspects (tracker, protocol and replayer).


> We’re also toying with the idea of an “enterprise” self-hosted deployment, similar to gitlab’s billing model, and thoughts from the community on this front would be appreciated!

First, Thanks for sharing - there might be a few interesting applications in my (non competitive to you) world I wouldn't have considered for the sheer amount of work to begin with.

My experience with B2B on prem has been that maintaining first class on-prem support will have an outsized growth vector for you. The cloud after all is someone else's computer under the control and ownership of someone else, and that won't work for everyone. It's easier to do from the start than to try to shim in. If you really want it to be an unfair advantage and moat, being present as a first class verified citizen in the marketplaces of multiple clouds is good too for those groups who might want it.

For example, being ready to roll (and license for enterprise) int he Microsoft Marketplace is a bit of a superpower, because they already have a good chunk of the Fortune 500 already as customers, so you don't have to go get them and be close to being a one click install where an on-prem m365 environment might allow things from Azure directly or indirectly.

The harder part is making sure enough of the useful and even valuable bits aren't locked away from the spirit of open-source, or critical ones committed moving forward. It's hard but the hope is the magic of creating beginners for your platform remains the chief priority for growth and adoption for your system - people readily available and skilled in it.

One nice way to do this is to perhaps consider some creative licensing around headcount of an organization, sector, or yearly revenues. Appgyver had an interesting approach to this prior to being acquired by SAP and seem to have been able to maintain it.

, some folks do well with an AGPL type of license to minimize competitors which I respect a groups right to do for their own sustainabilty.


How the hell did you get the GitHub username highlight?


Ha, we were very lucky. We reached out to the owner of the username (a developer from france) and he was kind enough to give it up because it was inactive. He didn't ask for any money, but we ended up paying him a bit since he was so generous.


hey, look, another analytics SaaS based on ClickHouse ! :)


Performance and ease-of-use of ClickHouse have been great so far! Looking forward to be leaning more in on ClickHouse for the logging product (https://www.highlight.io/docs/general/product-features/loggi...) we're building, and potentially consolidating other infra with ClickHouse.


Yes, I know it's awesome. I predicted 4 o 5 years ago that all analytics SaaS with certain scale will migrate to CH. Sentry,plausible and many others has already switched. In the other hand, I've been trying for the same yeras to incorporate it at (Corporate Work) to replace ElasticSearch with not success (because Managers...)


Thanks for sharing this! I'm curious why you chose the Apache 2.0 license over another possible license that would help mitigate cloud providers (read: AWS) simply taking this code and offering their own version of your service?


What applications are good candidate for highlight to be used? is there any breakdown of existing client/partner base to learn more about in which domain it gets used most and how.


Any web-based application could be a good fit! We also support a lot of the gnarly session-replay features that a lot of other vendor don't: namely canvas recording, shadow dom etc.. (see https://www.highlight.io/docs/general/product-features/sessi...).

In terms of audience, most of our customers are startups, and the users at those companies are often engineers or technical product people!


Looks very interesting! Any plans for supporting ruby/rails? Is it a lot of work to make an sdk for a new language?


Yup, planning to have a Ruby/Rails SDK shipped next week. We're also working on docs on our opentelemetry usage to make it easy for folks to contribute to future ones.

Here's the github issue if you'd like to add any considerations: https://github.com/highlight/highlight/issues/3999


Tools like this are super creepy when used without informed consent. Please don't.


We use the cloud hosted version and love it. Hope the OSS version is easy to set up.


We've put a lot of effort already to make the OSS setup a single `docker compose` command despite the extensive infra we use under the hood. Longer term, we're planning to support other deployment strategies such as kubernetes, cloud templates, and openshift templates. Appreciate any feedback on the self-hosting end here or on our GitHub issues!


Super




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: