I worked for a personalization vendor a few years ago and was on an engineering team that built a feature like this. It was ... a major headache, and I don't think anyone was ever completely comfortable with the security implications.
So I wish the team good luck, and especially with issues like:
* A/B testing. How can you ensure that what your clients are seeing is the same thing you are seeing? Maybe you've been assigned to different A/B groups.
* Extensions. What if your clients are using Chrome extensions that affect how the page is rendered? How can you replicate that?
* Networking/auth issues. What do you render if you're unable to connect or log in?
We appreciate the good luck! We're definitely going to need it. I spoke to the security side in a previous comment, but here's some thoughts on the other points you mentioned:
* A/B testing: We're grabbing the browser session data, which includes things like cookies and local storage. In our experience, this tends to be where A/B testing group identifiers are stored. But, if it was on something like the IP, then, yeah, it is pretty hard for us to get around.
* Extensions: Yeah, this is a really tricky one. We've resorted to asking some of our more active customers, who've had these issues with content blockers, to use a separate browser profile. One thing we do for this (which also can help solve the first point) is what we call "smart capture", in which we capture based on a DOM selector, instead of x/y coordinates. That way, no matter the page layout, we get it right. The hard part of this, though, is this is not reliable on every site, and we've only enabled it on sites that we know it works well on.
* Networking/auth issues: This is a really fun one! I mentioned it in another comment, but we're doing some logic around detecting logged out state, with an image classification model, and checking the HTTP status code for resource requests. When we can detect it, we send the user an email, and give them a flow where they log back in, and allow our extension to capture the state data again. On networking, we can't get around things like corporate intranets of course, but these seem to be rare for our customers. We also will just retry the capture if there's a network error, and we notify people when there are errors, so they can take action accordingly.
There is a payment provider in Europe that has become rather popular, who implement "instant" payments by asking for your online banking credentials... Security best practices always go out the window when they interfere with the business case.
Correct me if I’m wrong but this is disallowed by many end sites eg Google and circumvents oauth/any real security. If your very new business was hacked in any way you’d be solely on the hook for the massive damage a bad party could do with such an elevated session with no real boundaries.
Cool idea, but I fail to see how this is 1) legal under many ToS’s and 2) not just a dangerous XSS attack waiting to happen.
Tangentially related question, what are the most successful products built around browser extensions? I sometime have ideas that would require browser extensions, but it feels like the target market must be computer literate for any chance or success (no mobile support for example).
It depends on your definition of success, but the list is bound to include Adblock, Grammarly, Honey and some password manager. And at least Grammarly and Honey are not specifically targeted at computer literate people.
There seem to be a lot of those chrome video recording/school ones, like Loom. Seems like a lot of schools use it to record lessons. Being on Chromebooks I would think they are used to installing extensions, if allowed! Usually that is on a whitelist basis.
Possibly Pinterest. I am not sure if the browser extension was 100% necessary for Pinterest to succeed, but I remember it being part of the new user sign-up flow, at least at one point.
So I wish the team good luck, and especially with issues like:
* A/B testing. How can you ensure that what your clients are seeing is the same thing you are seeing? Maybe you've been assigned to different A/B groups.
* Extensions. What if your clients are using Chrome extensions that affect how the page is rendered? How can you replicate that?
* Networking/auth issues. What do you render if you're unable to connect or log in?