I was a Mintlify customer 6 months ago and found their onboarding strange. Their Github app asked for permissions to access all my repos and there was no way to get around that. I just wanted to give them access to my docs repo not my app repo. Ended up having to dig through their settings page and revoking that permission after onboarding.
For customer OAuth tokens, I believe you should NEVER store the access token in the database. Only store the refresh token in the database. When you need api access, get a new access token with the refresh token and client secret.
This prevents incidents like the above as the attacker would also need your client secret, which is ideally not in your database.
Too generous permissions are something that often makes me not install software. Zoom apps for example. Slack apps another example. Mostly browser extensions.
This is pretty typical of GitHub, unfortunately. They have terrible scoping and don't give developers the option to even give users that flexibility. It's been like this for over a decade, I don't think GitHub will ever learn.
Wrong. “GitHub Apps” always (?) had fine-grained repo-scoped permissions, it’s the “OAuth apps” that are stuck with coarse-grained permissions. So developers do have the option, it’s even the preferred option.[1]
> In general, GitHub Apps are preferred over OAuth apps.
If half a decade is new, sure. Except I was replying to someone claiming “I don't think GitHub will ever learn”, commenting on an article from a company created in 2021.
With apps, you can certainly grant access to individual repos instead of all repos. Yes I’ve tried.
Here’s how this thread went:
brycelarkin: Mintlify’s GitHub app asked for permissions to access all my repos.
You: Developers don’t have the option to not access all repos.
Me: False. GitHub apps do have fine grained permissions, as in they don’t have to have to access all repos.
You: Even with apps you don’t have fine grained permissions “you’d think you’d get”.
So what are you talking about? What’s the thing I’d think I’d get but not getting and how is that related to “permissions to access all repos” as discussed here?
Exactly. You'd need to implement standard authentication mechanisms, preventing these vectors of attack from existing in the first place.
They are "in the process" and "looking" to do that.
As of now if I am not mistaken the main issue still persists and any other new security vulnerability will possibly give full access to tokens again.
Correct me if I am wrong please but I don't see anything in the comment or Blog post saying otherwise.
> and are also in the process of completely deprecating the admin tokens for a more secure internal authentication procedure. Not to mention, we're also looking to fully deprecate the need of the GitHub OAuth tokens entirely in the coming weeks.
Mintlify sponsors one of my open source projects, express-rate-limit, and we use their service to host the docs for it. So I'm probably biased, but I've honestly been pretty happy with it overall - I'm thinking about using it for some of my other projects.
I think the best thing I can say about it is this: Mintlify helps us write better docs. Before Mintlify, we had a humongous readme plus a few one-off articles in the wiki. After switching the Mintlify, we started organizing things better, beginning with a less overwhelming readme.
I found out about this hack when they sent me an email yesterday. I looked through the git history (edit: and github activity), but nothing seems awry.
It sucks, but they're a pretty small business, and I think they're handling it reasonably well.
I reported security issues to Mintlify back when they used Discord for support. Gave recreation steps. Never got any feedback that looked to me like they were looking into it.
I reported similar issues to what's described in May 2023.
The vulnerability issue that you reported, if I recall correctly, was relating to server-forwarding attacks via our API playground. We've addressed that issue around 8 months ago, and unfortunately, it was not the root of this incident.
Oof not a fun incident, this is my nightmare as someone who works on this type of stuff.
As an aside, GitHub’s security model for apps/integrations is extremely puzzling to reason about and enables a lot of foot guns. Add the fact that it’s very obtuse to audit integrations (especially within an organization) makes them pretty scary to use sometimes.
This is Han, I'm one of the co-founders from Mintlify.
For the security of our customers from the publicity we were expecting from the announcement, we've decided to leave out the technical details of the breach in the blog post.
The source of this security incident was due to an uncaught error response in one of our APIs that didn't properly format the response before sending it back to the client. The response contained our internal admin tokens, which can then be used to access internal endpoints, which unveiled sensitive user information.
Our initial patch upon discovering the incident fixed the response of the vulnerable endpoint, but we have since also implemented a sweeping number of security provisions and are also in the process of completely deprecating the admin tokens and GitHub OAuth entirely to prevent an incident like this from ever happening again.
>"we've decided to leave out the technical details of the breach in the blog post"
>"Our dedication to transparency, security, and the trust you place in us remains unwavering."
You are contradicting yourself here.
>"The source of this security incident was due to an uncaught error response in one of our APIs that didn't properly format the response before sending it back to the client. The response contained our internal admin tokens, which can then be used to access internal endpoints, which unveiled sensitive user information."
Why would you leave that out? Seems like it is vital information.
> For the security of our customers from the publicity we were expecting from the announcement, we've decided to leave out the technical details of the breach in the blog post.
This doesn't exactly inspire confidence that your service is now secure.
It's most likely not. Not sure if this was intentional but they pretty much confirmed it in a reddit thread:
> ... and are also in the process of completely deprecating the admin tokens for a more secure internal authentication procedure. Not to mention, we're also looking to fully deprecate the need of the GitHub OAuth tokens entirely in the coming weeks.
Yea it does. It most likely is related to MDX documents beeing susceptible to XSS attacks. Having worked at mintlify last year I can tell you that this is not surprising at all and I've been warning them extensively of sth like this happening.
"we cannot confirm that no other such instances occurred". I'd like to understand this more. Do they 1. not have a fingerprint of what the exploit looked like because it was indistinguishable from customer behavior? 2. Not have logging necessary to audit this? Regardless, with the added monitoring to detect suspicious activity, do they also have the logging to see everything a bad actor did if there is a new exploit?
So a user (or white hat?) alerts them to security issue on their endpoint. Email comes in at 4:55 PDT on a Friday.
[first red flag] No internal monitoring to check for unauthorized access
Poor engineer(s) tasked with issue on a Friday at end of work day manually rakes logs in their app. After ~1 hr of searching, discovers unauthorized device using leaked credentials
[second red flag] allows any device to hook into their critical infrastructure and access apis that have potential to expose PII and possibly move laterally within the org or customers.
Then poor engineers tasked with painstaking task to rotate all of the tokens throughout the night. Bye bye family time, any planned events (game tickets?). I hate this company already and I have no clue wtf they do.
[third red flag] why the fuck are they storing user tokens in a database. This is apparently a “SOC 2” certified application/company. Shows how much that is worth here.
[edit: fourth red flag] no indication of how long the unauthorized credentials have been used and what customers have been impacted. Very very piss poor logging or purposely omitting due to potential litigation issues
They are claming that they resolved the vulnerability that caused the token leak but don't mention it. Doesn't exactly seem transparent to me or like handling it well.
I was contracting for them last year and tried, among other things to build an actual engineering culture that prevents and fixes issues that accumulate to catastrophic incidents like this.
They generally prefer to "ship fast".
I informed them very thoroughly again on January 13th (3+ months after they terminated me for "cultural differences"), because I was worried of this exact nightmare scenario happening very soon.
The reason for this was that they open sourced a package that let's an attacker easily practice and test locally in like a minute.
MDX exposes to Cross site Scripting easily.
I assume this is the "fixed vulnerability" they are talking about, just to be transparent.
If they're a GitHub App, they receive a token to authenticate into your account/org when you grant them access/enable the app.
Everyone should audit their GitHub Apps periodically/avoid using them if at all possible IMO. Most of these integrations are just a convenience for adding webhooks, which you can do yourself without compromising security. Always prefer "outbound" integrations.
Broadly speaking, being SOC 2 "compliant" isn't really a thing. They just had a SOC 2 audit done, you have to read the report to discover how "compliant" they are.
SOC 2 is useful as setting a baseline for how a business and IT org should be ran, assuming it's followed...
All SOC tells you is that there is a process being (mostly) followed. Always review the reports for your vendors and follow up on any findings or gaps that show up.
It's surprisingly common for places to be SOC compliant, yet their latest report has half a dozen or more gaps/findings.
For customer OAuth tokens, I believe you should NEVER store the access token in the database. Only store the refresh token in the database. When you need api access, get a new access token with the refresh token and client secret.
This prevents incidents like the above as the attacker would also need your client secret, which is ideally not in your database.