This is great for internal distributed systems where having ordered keys is useful, however, it should probably be noted that these probably shouldn't be used as public identifiers (even though this will probably be the defacto standard and used publicly without thought).
Having any information, specifically time information, leaking from your systems may or may not have unanticipated security or business implications. (e.g. knowing when session tokens or accounts are created).
Given that a UUID identifier fits in a single cipher block, and the whole point is that these are unique by construction (no IV needed so long as that holds true), it seems like a single round of ECB-mode AES-128 would enable quickly converting between internal/external identifiers.
I like the idea, but I think it's not possible to rotate the key with that approach, without introducing a breaking change. Eternal secrets are usually a very bad idea, because at some point they are going to be leaked.
If you were using something like UUIDv4, you wouldn't be exposing that information at all though, neither in cleartext or ciphertext. It seems weird to say, "the user ID contains secret information so we encrypt it with an eternal fixed pre-shared key then share the ciphertext with the world", when you could've just said "the user ID contains no secret information".
It feels like the right solution here is to pick between: use UUIDv7 and treat the account creation time as public information, or use an identifier scheme which doesn't contain the account creation time.
That doesn't fix the info leak though, you're still leaking approximately when the account was created. Knowing whether an account was created in 2003 or 2023 is a pretty significant amount of information even if you just know that it was created some time between June 2003 and August 2003.
I mean it's certainly an improvement over telling everyone the millisecond the account was created, but if account creation times are to be considered non-public info, I would probably just not include any version of it in public facing user IDs. And if you do consider approximate account creation times to be public (such as HN, where anyone can see that my account was created July 1, 2015), then adding some fuzz to the timestamp seems to be a good way to avoid certain cryptographic issues.
With massive amount of data this would negate the performance advantage completely. Related data is no longer stored close to each other, every time-sequential access is a cache miss ...
An incrementing version number would once again leak time information. Even a not incrementing version number would leak that kind of information, because if you know the timestamp of another ID with the same version.
I think there is no good alternative to random external identifiers.
This is why I’ll probably just always use a UUIDv7 primary key and a secondary UUIDv4 indexed external identifier… which is extremely close to how I tend to do things today (I’ve been using ULID and UUIDv4)
Lookup speed. Direct mappings are faster. If something needs an external identifier that can be looked up for URL/API queries and such things…
Internal sorts and index on write with ULID/UUIDv7 which reveal potentially sensitive timestamp information, and so when that’s not appropriate a separate column can be used for…
External opaque identifiers which are UUIDv4, and if indexing speeds become an issue, can be switched to deferred indexing…
It’s a good balance and everything I’ve ever used supports this (I only needed a pretty strong representation wrapper for ULIDs since PG doesn’t validate UUID bit field structures so i can just miss-use UUID columns)
It looks like Snowflake has the same information exposure issues that using UUIDv7 or ULID for a publicly visible identifier.
Purely random identifiers are the recommended primary key for plenty of databases - eg. spanner.
Random identifiers spread work evenly between shards of something sharded by keyspace. They also don't get 'hotspotting' on more recent records - recent records frequently get more than their fair share of updates and changes, and database query planners have no knowledge of that.
For large distributed systems like Spanner you want to avoid a hotspot on a single node as that limits throughout.
However for a single-node system like PostgreSQL you want hot spots because they are cache friendly. (Locks aside)
Basically you want a hotspot that is as hot as a single node can manage, but no hotter. Even for things like Spanner you still want good cache hits (which is why they support table interleaving and other methods of clustering related data) but in general avoiding overloading single nodes is more important (as it hurts latency and efficiency but doesn't limit throughput like a hotspot would).
Spanner is also bespoke, and was probably designed with that in mind.
Anything with a clustering index - MySQL (with InnoDB), SQL Server - will absolutely hate the page splits from a random PK.
Postgres also doesn’t like it that much, for a variety of reasons. Tuples are in a heap, but the PK is still a B+tree (ish), so it still suffers from splits. They also use half the default page size as MySQL, AND their MVCC implementation doesn’t lend itself to easy updates without rewriting the entire page.
Go run benchmarks with some decent scale (tens of millions of rows or more) on any DB you’d like between UUIDv4 and UUIDv7 as the PK, and see how it goes.
Right. In a way, these new sequential UUIDs approach the fact that recent records are often accessed more frequently as something that can be exploited, not as an issue that needs to be ironed out by randomness.
For tables this is not such a problem because of reasonably good locality (rows inserted close in time will end up close in the file too), but for indexes that's very difference. In Postgres this is particularly painful for writes, because of the write amplification.
None of this really breaks sharding (IDs are still unique and can be hashed), so no new hotspots. It can't fix the "many rows with the same ID" issue, but neither can any other ID.
I’m afraid you won’t be able to ever rotate that key, would you? Since it’s result is externally used as an identifier, you would have to rotate the external identifiers, too.
Assuming you have a table where the identifiers are stored you'd have the internal one (UUIDv7) and the encrypted version from it (external id).
You could rotate encryption keys whenever you want for new external id calculation, so that older external ids won't change (as they are external, they need to stay immutable).
If you're storing the identifiers then you don't need encryption, you just generate a UUIDv4 and use that as the external identifier. And then we're back at where the blog post started.
That's an interesting idea, how would you deal with the bits in the UUID that are used for the version? Setting them to random bits may cause issues for clients that try to use the identifier in their own database or application, as mentioned in the article.
For your information, yes, you can [1]. For example if you have a good enough 128-bit block cipher (e.g. AES-128-ECB), start with a block of 128 bits where specific 6 bits are filled out and others are filled with the plain text. Repeatedly encrypt the block until those specific 6 bits are reached again (and do the same thing in reverse for decryption). This is possible because a good block cipher is also a good pseudorandom permutation, so it should have a small number of extremely long cycles (ideally just one) with a 2^-6 probability of allowed values in average.
Cycle-walking FPE has a (nearly) unbounded upper-bound on latency.
Breaking the 122 bits into two 61-bit halves and using AES as the round function for a Feistel cipher gives you a constant 3-encryption latency instead of the expected 64-encryption average latency of the cycle-walking format preserving encryption.
If there's any way for the client to influence the input, it may be prone to DoS attacks: By my calculations, with a million random attempts, you would expect to find a cycle of length at least 435, which is over 13x the average. (Mind you, multiplying the number of attempts by 10 only adds about 72.5 to the expected cycle length, and probably no one has the patience to try more than 100 billion or so attempts.)
The properties of the permutation are dependent upon the encryption key, so a client being able to select malicious inputs to get long cycles implies either that the client knows the AES key, or that the client has broken AES.
In any case, as I mentioned in a sibling comment, with 3 AES encryptions one can construct a 122-bit balanced Feistel cipher with a constant amount of work.
I'm actually working on encrypting database keys like this, and I opted for a string ID with "url safe" base64. It avoids the ambiguity of looking like a UUID when it's not, and I prefer "o3Ru98R3Qw-_x2MdiEEdSQ" in a URL over "a3746ef7-c477-430f-bfc7-631d88411d49". (Not that either one is very beautiful)
And you have no problem with the same data being encrypted being identifyable (no salt). i see now why it might be useful in this case, though I still don’t like the idea, it feels bad somehow. (Yeah I get that you save storage for some computation)
Encrypting an internal id with ECB into an external id continues to allow the comparison for equality of 2 ids, to determine whether they are the same or not, but except for this it removes all the information contained in the structure of an UUID.
Yep have used an approach just like that, worked quite well if you have a strong pattern to easily translate from one to the other. Gives you an id with the right properties for internal use, efficient indexing etc, and in its encrypted form gives you the properties you want from an external identifier being unpredictable etc, all from one source id.
It is true that now your encryption key is now very long lived and effectively part of your public interface, but depending on your situation that could be an acceptable tradeoff, and there are quite a few pragmatic reasons why that might be true as has been described by other comments.
Edit: you can even do 64bit snowflakes internally to 128bit AES encrypted externally, doesn’t have to be 128-128 obvs
True you could rotate by persisting the old value and complicate your lookup/join process, not my idea of an acceptable solution but yep totally possible and worth it for some set of tradeoffs.
One key per token would require a table matching internal tokens to their key for forward conversion, and another table matching external keys to their key for reversing.
Might as well just use randomly generated external keys and have one table if you were doing that.
Thinking that harder-to-guess IDs will mitigate attacks is an example of security by obscurity. It's better to think of any IDs in your database as being public knowledge, because they will leak anyway. Assuming that no one can guess another ID leads to shoddy practices. I generally keep IDs sequential and build security around the basic assumption that IDs are not keys, passwords, sessions, or secrets - they're just the public matching identifier for those things.
To that end, I think it's neat to be able to improve indexing on UUIDs, but it's not a security solution.
Having sequential ID's is more than just a security risk, it's an information risk. Competitors can use them to estimate the size of your business, the number of customers you have, and all sorts of stuff.
This was used in the war to estimate the number of German tanks based on the sequential IDs
I’ve heard this argument many times, but I’ve never seen anyone actually post a reference to it happening (as in, a company finding and using this information; not the German tank problem).
To me, it reeks of solving imaginary problems while causing new ones.
Years ago I wrote a library that would exaggerate sequential IDs to make our SaaS platform appear more popular than it actually was to anyone trying to pay attention. Not sure if I’m proud of the hack or embarrassed. But of both I suppose.
but UUIDv7 isn’t sequential (unless I’m getting it mixed up). There’s just a time-based component which can make sorting really nice & some random bits at the end.
If you don’t let an attacker iterate your data, all they can tell is when the ID was created.
I vaguely remember somebody figuring out from photoshop activation IDs how many sales adobe was making and trading options around their earnings report with that information.
I don’t remember the details, so maybe it was something else and not photoshop/adobe.
Depends on how far you get, I suppose. Wozniak's apple employee badge has ID #1 on it, and that's cool as heck. I also remember ICQ users sorting themselves based on how many digits their identifier had.
I agree just knowing the number is bad, but it also makes it easier to discover far worse problems as well.
My second job was for a company that provided internet enabled phone conferencing solutions (this was years before VoIP became widespread).
The customer ids were sequential. Couple that with an outright idiotic security flaw (the login process set the customer ID in a cookie and the app trusted it on ever subsequent request. Just the ID. Nothing else), I was able to iterate over all the customer ids and hand over a complete list of users to my boss to illustrate the problem, starting with a list of the accounts of the complete upper management.
They could have been used to spin up huge numbers of 30-person long distance conference calls at high costs (this company was building out nodes with 20,000 line pstn switches before they had customers... it was crazy, and they failed but would've failed far faster if that had been abused and they were on the hook for costs from their carriers)
Trusting that cookie was still stupid, but had it been a long random key it'd at least been a bit harder to discover and exploit (their next attempt was to base64 encode it and I had to explain why that didn't help; they then finally blowfish encrypted it, but without any time component, so still subject to replay attacks... I jumped at the first opportunity I got to get out of there)
Not disagreeing with the general concept - these IDs leak information - but these are sequential IDs, not auto-incrementing IDs. The leak is the time the ID was generated, not the volume of IDs generated.
That's still a competitive risk -- it does things like reveal if a given list of customers from recent orders/posts are all new customers or long-term customers.
Or from a list of most recently added customers/users, you can figure out the rate of signups.
Revealing timestamps is bad because it can reveal way too much information about the health of your business that you prefer to keep private, if a sequential list of ID's ever gets exposed (which is hard to prevent).
They can still find out through LinkedIn, ZoomInfo, BuiltWith, and any number of tools.
The tank problem doesn't fit when the incrementing value is time since epoch. Integers yeah, UUIDs, KSUIDS, or any other semi-ordered thing to make your database indexes less fragmented I haven't seen a real leak issue with those.
Just friday I've had a discussion with a colleague about filenames.
We do a lot of computer vision and in his project, each processed object is assigned a UUID and he wanted to save images to files for each one.
So we took some time to go over various timestamp formats to be embedded into the filename to make the files sort chronologically. UUIDv7 is just spot-on solving our problem. In this use case, there are no real security considerations.
It may. Certainly, for instance, sequential invoice numbers do. If a business decides to take measures to obscure that, no problem. All I'm saying is that obscuring a numbering system for data artifacts shouldn't be considered any sort of security as far as keeping your endpoints from being hacked.
The point on invoice-numbers brings another issue to mind.
We model our domain(s) using DDD, and often "The ID" really is best left a thing with meaning. Customer-id, Bank-account-number, invoice-number, email, etc. At least within the domain, it is. The business (and laws etc) already ensure there can only ever be one invoice with this number.
Its terribly counterproductive to have two ID's for something. "Hey, can you have a look at invoice 20230233, because it seems the VAT was applied wrong. Hmm, do you have the UUID for that invoice and DM me that? You know, the long one with the hyphens".
I guess there isn't a one-size-fits all solution and that "it depends" very much on what e.g. "public" means.
You're absolutely right, this is also why you generally encrypt sessionized or "consistent view" pagination tokens for public apis (save for primitives like ddb or Kafka)
The end user should know no details about your internal key space.
> Moderate amount of well implemented obscurity is helpful.
You're getting that wrong: Everything else being equal, the more obscure system will always be the safer one. It's just that obscurity can easily be lost, so your system should, if in any way possible, still be secure even if fully known.
In the end, however, no system is 100% secure, but more obscurity will make it harder to find the inevitably existing issues.
I think the counter argument is, that all else is not equal when obscurity is a goal of security, because it adds a maintenance burden to some greater or lesser degree, and that maintenance burden becomes time taken away from proper security practices, or other value providing work.
I think the main argument is that security by obscurity can easily be circumvented, be it via sidechannel, secret leak, source code leak or a surprisingly small search space (for example the whole range of IPv4 being scanned by now). It's easy to assume something is secure and spend a lot of time on obscurity, which completely falls apart thanks to a small sidechannel attack. It's (usually) just a weak defense overall. Yes, it can also be a maintenance overhead and therefore risk via proxy, but it can actually be easier in other situations.
For a personal anecdote, I used to work in a small webshop and our software was horrible, to the point where minimal effort would have been able to compromise our servers, which were running software roughly as old as I was at the time (I want to note that I worked on improving the situation while I was there). Still, the only time we had a problem was when we took over a Joomla-hosted site, as we were small enough to not get any individual attention and your off-the-shelf WordPress or Joomla-scripts did not work on our home-brewed software.
In the end, I fully agree that security by obscurity is a weak concept and the usual wisdom of not relying on it is completely correct. Still, it's important to acknowledge that obscurity can and does help security and bring actual reasons on why you should not rely on it. Just saying "it's obviously bad" leads to an easily refuted argument and will not convince some developers, leading to worse software overall.
To me, the main reason to avoid obscurity in naming or numbering things, or even in code - rather than view it as a modest addendum to security - is to force yourself to do the mental exercise of what happens when that obscurity is lost.
Not doing that is how small companies seem to get away with terrible security holes for a long time, until suddenly they don't. I've seen too many cases of companies in a position where they built a small, insecure service that's now getting shared more widely than envisioned, who don't want to spend the money to make it right, because no one has compromised it yet (that they know of), and what are the chances of someone stumbling across it - where even pointing out that it's an attack vector can earn you trouble.
passwords and encryption keys are secrets, not obscurity.
Security by obscurity would be hiding your house key under a doormat for your friend to find - depending on the culture you live in you may be more or less safe but it is not security (just like hosting your ssh server on port 9384 will repel 99% of attackers but is not a security measure).
I keep SSH on Port 22. After years, I'm still amazed about the operational model of these attacking hosts.
They are completely dumb. I haven't kept record, but I have the feeling that some IPs in my fail2ban list are practically in there for month or even years now.
I assume they are just sweeping the whole IPv4 range? No state, no cache. Either they successfully attack a host or they go to the next IP. Repeat 2^32 times, start again.
I'm not sure where I wanted to go with this comment. Is it _that cheap_ to constantly sweep the IPv4 range or is it _that profitable_ to do it once you have a successful attack?
You should think of them as public, but that doesn't mean it isn't still helpful to obscure aspects of the information they carry.
Obscurity can be helpful as part of defence in depth, to reduce the impact when someone does something stupid, or to make it more difficult to extract information that might be helpful as a means to attack the system from another angle.
If you're already thinking about the implications, you can likely ensure people doesn't jump to the conclusion that the IDs can be trusted just because they look complex.
Security by obscurity is a working solution if implemented with other measures. It increases the cost of attack, which in the presence of unknown vulnerabilities gives you precious time to respond.
> But not too fast: If you can hash too quickly you can launch parallel attacks to find duplicates or break entropy-hiding. For unique ids, the fastest runner loses the security race.
> Cuid2 has been audited by security experts and artificial intelligence, and is considered safe to use for use-cases like secret sharing links.
I'm getting some snake oil vibes from this... There absolutely shouldn't be anything like a random ID that is 'too fast' to compute. You might need a rate limit to stay within your collision bounds, but CPU usage is a poor way to do it.
And there is currently no publicly available "artificial intelligence" that would be useful in a security audit, unless you want to call fuzzers "AI".
The comments on performance are utterly incorrect, modulo discussions on hotspots, but you shouldn’t be sharding randomly anyway. If you get to the point where you _need_ to shard for anything other than geolocality, doing so randomly will rapidly reveal your hotspots.
> One reason for using sequential keys is to avoid id fragmentation, which can require a large amount of disk space for databases with billions of records.
Disk is cheap but not free at higher tiers. But more importantly, record fragmentation means more pages (unless you take the time to do a full table lock and rewrite it, and who’s doing that?) which means more index bloat. I assure you, that adds up once you’re into the billions of records level.
> the ids will be generated in a sequential order, causing the tree to become unbalanced, which will lead to frequent rebalancing.
Given the width of B+trees used in DBs, I doubt they generally need to go more than one or at most two levels up. I’ll take the ability to rapidly follow the leaf nodes and have a good shot at sequential reads in cache from prefetch, thanks.
Reading their docs: No real benefits, just misconceptions.
1. Collision resistance / "weak" PRGNs used to generate UUIDv4. Firstly, these are properties of the implementation, not the spec. Secondly, the source for calling the browser `Crypto.getRandomValues()` insecure is an issue that has been fixed back in 2016. I would not trust the developers of this implementation to do a better job than current browsers.
2. "Not URL or name friendly": Fair, but not very strong argument.
3. "Horizontally scalable" and "offline-capable": No argument given for why UUIDv4 does not meet these requirements, apart from point 1 above.
4. "Too fast": No argument given for why having a slower algorithm to generate random ids is more secure. Both UUIDv4 and Cuid2 use a similar number of random bits (122-124). When using a secure PRNG, both are equally difficult to guess, the SHA3 hashing doesn't add anything. You don't have to try and guess the "input" of the Cuid2 - you can just try to guess the "output" and skip the SHA3 hashing. It would be impossible to actually guess a generated ID, but UUIDv4 is just as impossible. Also no argument given for why UUIDv7 is fine but UUIDv4 is not.
I've used UUIDv4 for genering unique IDs for over 10 years now. I have run into collisions, when I hand-rolled my own implementation for J2ME with major bugs many years ago - ended up with around 20 bits of entropy instead of 122. That's not a reason to not use UUIDv4, just a reason to not implement it yourself unless you really know what you're doing.
I once had UUID collisions in Linux Boot IDs when we started developing our own embedded systems at my company.
Found them because systemd-journald isn't very happy when Boot IDs repeat and (apparently, then) stops showing earlier boots once it hits a repeating boot ID. And I wanted to see an earlier boot. Then I started logging the Boot ID in a textfile myself and it took less than 10 reboots to have duplicate Boot IDs.
Long story short, some weeks earlier, I "optimized" the Kernel config for that system and some config flags that didn't sound like something I'd need. As it turns out, an ARCH_ZYNQ target apparently also needs ARCH_VEXPRESS set. Otherwise it works absolutely fine, but with a broken RNG that you will notice weeks later.
That was a valuable "don't take down a fence until you know the reason why it was put up" lesson. Don't unset kernel config flags until you know why they are set.
Aside from breaking RNGs, I've never experienced any UUID collisions either.
"A couple of years ago we decided to standardize the use of sequential integer IDs as primary keys, due to the significant performance issues of non-time-ordered UUIDs."
From the article. I'd like a lot more exposition on that, since it goes against some of what used to motivate UUID use in the first place. Sequential ordering across distributed nodes isn't a fun thing to do, and even if you navigate the coding, the network agreement makes it really slow.
Do they mean "sequential enough" but some locality of the node that generated it? And I guess that prng can sometimes have some performance bottlenecks, but compared to doing locks on a single incremented integer?
Yeah, I don't really get this, lots of usual "better" "faster" etc without actual numbers to back it up or detailed algorithmic discussions.
As you basically said, wake me up if it's good enough to get into the standard vetted libraries of UUID generation.
No argument given for why having a slower algorithm to generate random ids is more secure.
If the algorithm is too fast it means you can detect when some other part of the system is having a significant impact on how the key is returned. Eg checking a database to see if a user exists and returning their key versus getting null back and generating a new key. That difference can be used to determine if a user exists. You want your key gen process to be slow enough that it's a significant part of the process, which makes timing attacks hard.
This doesn’t make sense.
If you’re going to generate a new key if the user doesn’t exist then you’re creating a new user anyway, so there’s no hidden information.
Unless you mean that the system should return a bogus id instead of a status 40X when the user doesn’t exist, which makes even less sense.
ID generation should usually only happen when creating new assets, so it should be as fast as possible.
Some of the arguments mentioned are explained elsewhere in the README, others are assumed.
One argument standing out for me is the lack of collision-resistance for UUIDv4 which is surprising for me and I didn't spot any sources for that argument.
Another argument is the entropy source where they go about that Math.random is not reliable as a single entropy source but glimpsing at the source code, they sprinkle the CUID with Math.random data.
I am no expert in ID security, so I am not qualified to speak about the validity of their arguments, only that there's insufficient information to validate without prior knowledge about the problem domain.
>Having any information, specifically time information, leaking from your systems may or may not have unanticipated security or business implications. (e.g. knowing when session tokens or accounts are created).
I don't think this is really true? These are not serially incrementing, they just indicate the time it happened. If you have an ID that you know exists, having the ability to know _when_ it was created is very rarely meaningful.
What could present more of a risk is being able to predict a large part of IDs that will be created. Even then though, you shouldn't depend on your IDs for secrecy - best to ensure the IDs are never used as protection by themselves (ie treat them like they're just a simple autoincrementing number, even if they're not)
One real world security problem is the "elder account" problem: as the age of an account increases the likelihood increases that it uses an insecure old password and/or that the account owner isn't paying as much attention to the account in the present. Depending on what the account represents age may also imply more "value" in the account. (Including just "sentimentality" value in the case of ransom operations, not just financial value.) Being able to tell from an ID alone that an account is at least X years older than some other ID in the system can be a handy way to find "potentially high value/low security" accounts to focus on to social engineer.
There are certainly mitigations that can be made and not all things are equally valuable as they age. (Plus many public APIs include created/modified timestamps anyway. The information is often easy to discover even when not embedded in an ID.) I don't find it a strong reason to avoid timestamp-based IDs for the threat models of that many things beyond user accounts and other things susceptible for social engineering, but it is something to keep aware of.
You would be correct if the ID were an integer being serially increased. I can sign up to your website today and get an ID X and then sign up again in a week and get ID Y, I can then calculate the number of new users you've had by performing Y-X.
If this ID is a timestamp then there's no such information I can get out of it from a small sample. I sign up today and get todays timestamp, then I sign up next week and get next weeks timestamp..?
Don't create them each time a record is created, create a batch in advance in sufficient number, and do the same every time the previous batch has ran out. UUIDv7 is 128 bits, you can store a large number of them without major penalty.
But then you need to have the client communicate with the server to identify it's newly created object or complicate your logic to have incomplete objects in a pending state, which is one of the things people were using UUIDs to avoid.
You don't store the UUIDs in the database as incomplete records. You can put them in a unused_uuids table and store some of the values in memory to minimize the round-trip. You can even store them in a simple file, and remove each used UUID from that file. When the file is empty, you create a million more of them.
The incomplete objects refers to when someone clicks "new" in your UI. Until it's saved back to the server, that "new" object has no ID, since you need to communicate with the server somehow to get that UUID in this approach. So now the client creates objects without IDs, so now all your models need to assume IDs are optional, and you can't create your object references on unsaved objects.
Then we can get a random sample of UUIDs in sufficient number from the batch, send it to the client at the beginning of the session along with the other data, lock that batch until the client releases the session to avoid duplicate use, and have the client use these UUIDs until they run out, at which point it can request a new batch.
They’re also incredibly cheap to create & don’t need knowledge of each other. I mostly see batching of IDs like this when a lock is involved to prevent collisions & maintain performance.
With UUIDv7, you are reasonably sure that there won’t be collisions (check your use-case first), and can just generate them wherever on-demand (no locks required).
I’d argue batching IDs is actually more complicated than UUIDv7 for most use-cases.
This was a solution to the UUIDv7 problem of being time-dependend and therefore may leak information. Create the UUIDv7 in advance by batch of 10,000, use these randomly, and you fix that problem.
Fyi, the timestamp is already encoded inside uuid4.
But for having a good distribution of values, the low bits half of the timestamp is stored before the high bits half.
Here uuidv7 will just re-order that. So the content of the uuid in itself does not change.
Leaking a monotonic ID could allow outside observers to estimate e.g. number of accounts created or products sold over certain timeframe. Competitors (or traders, for a public company) could use this like a form of inside information on the company (e.g. sell the stock if the rate was falling).
This would really only be possible if you leaked a monotonic sequence; the monotonic clock only would potentially leak only event ordering or absolute time.
IMO it's not the job of the identifier itself to prevent information leakage vulnerabilities though; if thee is sensitivity to this, the solution should be explicit, such as employing a secondary key derived from the UUID using a secure KDF or similar.
The German tank production capacity was estimated by serial numbers of captured tanks. There are ways to read all kinds of information by observing energy usage. High resolution time and sequence data undoubtedly reveal more than you’d like.
Could be quite useful information to competing charger networks or in future M&A discussions.
I've seen a project for a trading firm that inferred all kinds of traffic and revenue numbers for companies before their quarterly earnings were made public. It wasn't perfect, but knowing with a certain confidence level whether the numbers were going to be better or worse than estimate was profitable for them.
Sure, for many places it doesn't really matter, but if your URLs or user ids can be seen/scraped by others, you might expose some commercially interesting information to competitors.
And the indexing argument isn't really compelling, is it? You lose very little by sticking to fully random UUIDs.
Here in Ukraine trying to estimate enemy's drone and missile production capacities by serial numbers of their parts is quite a mundane task these days.
I could imagine using the timestamp segment of publicly observable ids to estimate activity patterns in an organization. Probably not super crucial and there are probably easier ways in most cases but it could be a big deal at the right moment and for the right target. This could be like a more refined version of PIZZAINT (where you can detect impending policy/operational movements by the quantity of food deliveries to a government organization).
In ye olden days you had a minion sitting in a car watching the front gate with a notepad and enough cigarettes to last the night.
Pizza itself might be a bit of a joke but looking for non-operational behavioral changes is absolutely real. The Cuban missile crisis was started in part because soviets played soccer and Cubans played baseball and the presence of soccer fields helped confirm soviet presence (in enough numbers to bother making rec centers). A more advanced version might be the public Strava data leaking US base layouts and locations or Strava helping the Ukrainians kill a Russian submarine commander.
Edit to add: you could also just figure out where your target orders pizza from and pay one of the dudes working there to tell you when there’s a spike in deliveries to your target.
> soviets played soccer and Cubans played baseball
It's a Henri Kissinger's quote, but it's not accurate: Cubans do in fact play football. Also this quote wasn't from the 1962 Cuban missile crisis, but to another event in 1970. That being said, it is true that the US intelligence got warned by the construction of football fields (or maybe even more so by the lack of baseball grounds).
I know of people who used leaked customer ids in public facing chatbot solutions (like Intercom) to estimate how fast their competitors were growing and/or how many customers they had.
How would you do that with UUIDv7 though? I see how using sequential IDs would obviously leak that information, but if all you leak is the timestamp the ID was generated, how do you then infer anything about the rate of ID generation?
If it is the same database system, you may as well use a SERIAL or similar synchronized datatype. The point of using UUIDs is that any participant in a distributed system should be able to generate them, without having to rely on a centralized authority. If the use is purely local then UUIDs would be pointless.
If you have 74 bits of entropy, the birthday paradox says that after 2^37 keys you will hit a 50% risk of a key collision. Whether that is going to be a problem depends on the use case, and on the quality of your RNG.
But the uuid collision chance only matters if there is even a possibility of colliding writes and lookups.
So what if your microblogging platform's tweet uuid happens to collide with my cocktail recipe generator's ingredient uuid? The likelihood that any one device will ever be running our apps at the same time and trying to read data from one into the other is even smaller than the 1-in-380000000000000000000000000000000000000 (uuidv4) probability of having a collision in the first place.
Because incrementing integers require a singular central node to track the universal state and even so still have a risk of collision bucause they are not guaranteed to be unique in absence of an externally carefully managed master incrementer.
Applications use uuids to avoid colliding with themselves. Uuids exist so that web-apps can create objects client-side without waiting for a database CREATE, applications can be built with multi-node dbms, operating systems can name hardware compenents.
They can be bad for performance. It all depends on your access patterns. A common caching pattern is called "temporal locality" which means that theres a high likelihood that data created at the same time will be accessed at the same time. Therefore, if these pieces of information are on the same machine, they can be queried / returned much faster than if they were both on separate machines. This is doubly true if theres a data dependency between them. E.g. SELECT x + y or SELECT x WHERE y = 'foo'.
Yes but if that machine with sequential data receives 100x the traffic of other machines, it can be worse than splitting this traffic evenly across all available machines.
If your database simply shards keys sequentially, it's going to get hotspots in a lot of use cases, like plain old integer keys and timestamps, not just UUIDv7. In that case it would be fair to say that your database is doing it wrong.
Fortunately, there's no rule that says you should shard your keys using the sequential part up front.
One of the rules for generating randomness from environmental sources is to throw away the high bits and only use the low bits. Distributed databases should do the same if they want a good distribution.
What distributed databases shard on the low bits? How do they do something like a range query?
The closest I’ve ever heard of is sharding based on a hash (e.g. CockroachDB can do this on request[1]) but most distributed databases with strong consistency (Spanner descendants in particular) default to “doing it wrong”.
You're painting with way too broad of a brush. It is not always better and not always worse. E.g. "give me a list of users who made two posts where both posts were created within 1 second of each other" This query would likely blow up on a system which has all the data completely randomly sharded (because you'd have to aggregate all the data centrally, unless you had a complicated shuffle setup (which most dbs dont)) whereas would work fine on a system which has posts sharded by time.
We solved that with UUIDS and updated_at and created_at columns. The latter the default sort in all views and queries. So the btree/indexing issues were hardly an issue. Whenever you fetch a set of rows, they will be bounded by these timestamps.
We even sharded on these columns, because of this (our business case made it so that hardly ever did people need data over multiple months)
But we never encountered distribution issues. I don't think the locality issue will be solved, as postgres doesn't consider other columns when distributing data, only the primary key IIRC. I don't know why we never saw this, though.
If you're working on a multi-user system, particularly one with hundreds of requests per second, there is no locality of ids. Two of my actions are separated by a sea of actions by other users.
I know HN doesn't like jokes, but this is really funny. And the subcomment about mullets too.
(for folks who don't get it, mullets are a 1980s haircut (think MacGyver) with a short front but a long tail in the back. A funny description of them is "business in the front, party in the back")
Sitting on a tram in Sheffield, early noughties, a chap with a magnificent unreconstructed ROCK mullet gets off. I see two women chatting on the street, they are stunned into slack-jawed amazement as he struts past -- one of the women makes a scissor-snipping hand gesture behind him as he passes, the whole tram erupts in laughter.
If you're using a system which is built for distribution, random is great.
When you're leaning on a Postgres database which has powered your startup through scaling but expects right-leaning btree indexes, it's a bad time.
Rearchitecting to use a new data store is ideal, but often impractical as an immediate step. UUIDv7 is a great increment walking that road via sharding etc.
In all the distributed systems I’ve built I hashed the keys to ensure good distribution. A nice thing of ordered keys is you can use part of the ordering to distribute keys with a tunable amount of key locality in each node for efficiency.
Depends how you query it. In a lot of systems, recently added data is also the most queried data or data typically gets pulled out sorted by time. Having that data on disk in more or less the order it is going to be queried makes sorting it a bit easier. Even in a sharded system, each of the shards would have less work to do for sorting. Of course a lot of these systems would have an append only write model which would effectively sort things by time anyway, even with completely random ids.
Somebody posted an interesting article for the instagram ids, which do something similar. They use 41 bits for a time from a custom epoch followed two more groups of bits for a shard id and a sequential number. Each shard has an incrementing sequence for the sequential bit, which guarantees that things on a shard are sorted by time.
This UUIDv7 is slightly weaker than that but sorting things published in the same millisecond is mostly going to be very light work. The lack of a dedicated sharding group of bits is not that important as you could just take the n least significant bits at the end for that without too much effort. Those are random so you end up with nice consistent hashing. Having 48 instead of 41 bits for the time means we won't run out of time any time soon (nearly 9K years vs. 70 years).
Depends on the use case. If, for example, you store things on disk ordered by these IDs, and access patterns to your dataset are related to time (e.g. more recently created data is accessed more frequently), it will help a lot to have this data ordered by time.
This is especially useful when your underlying database stores data in large "chunks", such as LSM-trees you find with e.g. rocksdb.
It's great for performance, up until you reach the point where a single device becomes a bottleneck, at which point it's terrible for performance.
As a sibling comment says, you ideally want to shard on some other key to get "just enough" distribution that all your machines/disks have work to do, but you are still only hitting a limited number of hot sectors on each disk that can be effectively cached. But that requires active monitoring and rebalancing of your data as it grows. Totally random keys are a safe default that will scale with any kind of data distribution and access patterns.
UUIDs are good for data where I want either lots of different users being able to insert without collision, or lots of users who I want to keep their peepers off of other user's metadata (eg, how many X they add to the system per day).
In both cases I'm melding highly disjointed data into a single schema. There are no large consecutive sets of records.
If you're using UUIDs, there's probably a reason. And that reason invalidates the justifications for not using them.
As far as I understand, you want to have a random shard position, but once you have found a shard you want that index operation to be cache friendly. When choosing a shard, you can always use the last N bits or use some consistent hashing strategy[1]
They later note most of this traffic is going to a single postgres instance. Having all the keys go to the same range probably helps throughput because they can do a better job of grouping fsync. But that probably depends on the type of drives they are using (even fast NVMe benefit from locality).
Is there some reason new versions of UUID keep appearing? It seems like the desired properties are never quite achieved so new ones appear later. Is there a table with UUID version across the top and characteristics down the side, so I can see the differences and pick one that fits my needs? That might also help to explain why there are so many variants.
As someone who only cares about v4, I periodically wonder why don't I just use fully random 128-bit identifiers instead (without the version information).
That'd be perfectly fine. You need to take some care to avoid duplicates or patterns though. Databases might not come with crypto-random out of the box but usually do have UUID support. Similarly, you need to use the crypto-random routines in your favorite standard library.
Depending on the implementation you still might have to worry about seeding issues. That's probably moot though since the UUID library would probably be compromised by something like that under the hood too.
UUID v4 isn't large enough to prevent collisions, that is why segment.io created https://github.com/segmentio/ksuid which is 160bit vs the 128bit of a UUIDv4.
I think you vastly overestimate the likelihood of collisions. If all of the 2 billion computers in the world produced a new uuidv4 every millisecond, we still wouldn't expect a collision for the next 5 quintillion years.
Or if the same number of bits are used in a more structured manner, like uuidv1 which combines a 48 bit MAC address, 60 bit 100-nanosecond timestamp, and a 14 bit uniquifier with an effective resolution of 6.5 femtoseconds, you could have 300 quadrillion computers make 160 billion uuidv1s per second with guaranteed zero collisions until the year 3400 (plus an extra 500 or so years because the timestamp is referenced to 1542 for some silly reason), after which point any collisions that do happen will be completely irrelevant because they're guaranteed to be colliding with db entries created over 2000 years prior.
> If all of the 2 billion computers in the world produced a new uuidv4 every millisecond, we still wouldn't expect a collision for the next 5 quintillion years.
This would generate 2^127.8 UUIDs.
First, no, the collision would be expected in less than one year, approximately after exhausting square root of the available space (2^64 generated UUIDs): https://en.wikipedia.org/wiki/Birthday_problem
Sure, the presence of some collision somewhere is still likely to happen, but the chances that that collision will actually matter for anything is still vanishingly small.
In the real world, we do not spend 100% of our entire species's computing capacity generating uuids and doing nothing else. In the real world, we aren't burning through uuids at a rate of 2 billion per millisecond, and even if we were it wouldn't matter because the true denominator is the scope of the data system the uuid will be referenced in: if your hard drive partition and my webapp user entry happen to get the same uuid, we will never know, and if for some reason it does matter, then we can use uuidv1 or uuidv7 which guarantee no collisions for thousands of years by embedding a timestamp.
That doesn't help me. My code may generate a whole bunch of IDs in a microsecond, so using a 32bit time isn't going to keep them time sorted. I may as well just use a UUIDv4 at that point.
Looks like you got things a bit mixed up. You’ll not see a v4 collision in your lifetime, that is not why they built ksuid. At the time, UUIDv1 was the only standard alternative that includes a time component, but in way less bits, not sortable, and the fixed mac address takes a lot of space from the random bits.
ksuid is similar to Twitter Snowflake, the main goal is distributed generation of collision-free sortable IDs. The UUIDv7 proposal is meant to address the same use case. You don’t need to worry about collisions as much here, as the timestamp is monotonically increasing, there is a 42-bit counter for every millisecond + the random 32 bits at the end. You’d have to be generating trillions of IDs per second to have the chance of a collision.
Those are not helpful for database performance in a general sense. Last product I worked on used v4 uuids as primary keys without any issues - single master database, sorting by creation time only used in admin panels. You can index on created_at if needed. The IDs being sortable would be a non-feature.
Generating sortable IDs in very high volume, in a distributed architecture, is a problem very specific to systems like social networks, metrics collection etc. You won’t need that for your average e-commerce app or machine parts database.
> We use sequential primary keys for efficient indexing, and UUID secondary keys for external use. The upcoming UUIDv7 standard offers the best of both worlds
Unless you consider users being able to extract the generation time from the id to be an issue, of course.
I've been seeing a few different vendors do this already. MongoDB's ObjectIds are inherently timestamps (so you can actually generate generic MongoDB IDs to query based on time). There's also Discord's Snowflakes as well. I'm sure there's loads of others. All it tells you is when something was generated, not much else. I do love how MongoDB has it stored in such a way that it is easy to query against. I wonder if any RDBMS' will allow you to query these timestamps as well.
Can’t agree with that logic. Unless it’s specifically documented leaking timestamp data is going to get totally forgotten. So when you add (e.g.) the ability to change the sent timestamp on a message you’re going to inadvertently leak when a timestamp has been changed. Could cause embarrassment in a lot of scenarios.
This reminds me of that recent investigation regarding fudged data in those Harvard studies. The fields stored the "original" ID, solidifying their creation sequence, which differed to the displayed sequence - Implying that the fields were updated out-of-sequence, thereby they were tampered with.
Sure but what are cases where knowing when a random GUID was created is actually an issue of privacy? Every platform I've seen using Snowflakes that exact same data is publicly available.
It’s a case by case thing. I don’t disagree that in the vast majority of cases it won’t matter at all. But once it becomes the norm/default it’s inevitable that it’ll lead to inadvertent leaks.
And if you consider knowledge of the id sufficient for access.
Which, despite the fact that it really shouldn't be, still seems to occur every so often. Even in situations where the ids are very much not random.
Honestly if I have to read one more article about a 'hacker' who 'leaked' some secret government piece ahead of time because they thought to increment the date in the url of some yearly report, I'm going to lose my mind.
I don't understand. What part of this requires that one considers knowledge of the ID sufficient for access? And what kind of access are you talking about?
The performance benefits of index friendly user IDs seem like they would apply even if all user info is secret and requires a token to access... The application still has to look up the user by ID after all?
If I imagine a basic authenticated "get information about me" style endpoint, that would take a user ID and an authentication token. Checking if the token is valid is faster if the user ID is index friendly. Getting the requested information is faster if the user ID is index friendly. Yet a user of the API still needs both the user ID and a token to access anything.
And you can use it today with Postgres uuid type. Postgres doesn’t care what you store in it as long as it has the correct length. So you can generate a uuidv7 and store it natively
Stored in binary format, validation, more efficient due to non-cast, faster access due to non char*, being able to split the high-low, indexing and uniqueness at the byte level.
learned this last week that md5's are also 128 bits and so will fit perfectly in a uuid type in postgres, saving space in the physical table and indexes, and giving better index performance. So if youre storing md5 hashes, use UUID!
Similar to the old situation in the article, we are using sequential 64 bit primary keys, but we use an additional random 64 bit key for external usage (instead of 128 bit).
The external key is base64 encoded for use in URLs which results in an 11 byte string.
This hides any information about the size of the data, the creation date of customer accounts (which would be sort of visible with UUIDv7) and prevents anyone from attempting to enumerate data by changing the integer in URLs.
I thought about using UUIDs as external keys but the only compelling use case seems to be the ability to generate keys from many decoupled sources that have to be merged later.
It sounds like you basically just made your own 64 bit UUID. If you’re exposing this ID for manual use by a human (like URLs) then that sounds pretty helpful to be shorter!
Yeah, you can require them to generate a UUID if it helps for your app (e.g. bulk create objects with relationships between them) but then on the server you can generate new UUIDs and return a mapping from old->new to the client.
If UUIDv4 was all that ever existed, there would be no need to validate anything apart of the fact that it's supposed to contain 32 hexadecimal characters.
All other versions, including the new v7, attach meaning to certain bits of the identifier. That cat has been out of the bag for a long time, so now everyone needs to maintain code to ensure that some rogue node doesn't spew back-dated identifiers belonging to the wrong department.
I’d assume the validation is parsing the uuid with a uuid library (to decode it), and the library eagerly validates the version field, either to check for garbage or because it wants to yield a different subtype for each version.
- if the ID is intended to be opaque then the vendor shouldn't document it as a UUID, as this changing to a different format would be a breaking change
- if the customer isn't going to process the subcomponents of the UUID then they should process it as an opaque string
- if the UUID library encounters a version number in a UUID it doesn't understand, it shouldn't reject the UUID but present it as an unstructured string.
After this blog post it seems likely that even Kite more customer will parse the IDs to extract time, since this has been documented.
Probably because a typed UUID avoids treating it like a random string, and decoding the UUID means you have 16 bytes per in memory rather than 36 (assuming usual 8-4-4-4-12 representation over the wire).
As Lazare points out in this thread they're basically the same thing, except with ULIDs you get those 6 extra bits of randomness back that UUIDs have to use for metadata.
ULID isn't an "official" standard like UUID. Having a real standard usually promotes interoperability and makes it easier to use. Additionally as others have pointed out you can already use UUIDv7 with some databases since it's just 16 opaque bytes and the database doesn't care what's actually in the UUID field.
I'm not the author, but I work at the same company. ULIDs are nice, but we're a 10 year old company with many TBs of data across multiple logical databases and most rows had UUIDv4 ids.
Maybe if we were starting from scratch ULIDs would have been an option, but given where we were UUIDv7 was a much easier transition.
A useful/horrifying pattern on this topic: you can use UUIDv1 as a prefixed id, giving you a way to generate tagged IDs in a system that uses UUIDs.
You set the node field to a broadcast MAC address, and use that as a namespace/prefix. This inches close to the boundary of the RFC, but is arguably compliant.
As an example, you may generate demo or “canary” data items that are UUIDv1s with a well known node field, which then lets you do distributed “isDemoData()” checks by just looking at the UUID.
Can you take the first portion of the UUIDv7 string, and decode it to figure out the exact date and time that record was created? I'm wondering if there might be security/privacy concerns in some situations if the UUID codes are visible in your app?
Yeah, years from now we’re going to see some story about how a company fudged their timestamps in order to get away with X, only to be given away by the timestamp hidden in public UUIDs.
I just commented the same thing. I can't imagine most applications would want to leak time information in their identifiers but these undoubtedly will be used most placed out of convenience. In a year or so we'll read about an attack and everyone will migrate back to v4 or have to maintain a cryptographic identifier in addition to their temporal identifier.
Most application may not want it, but will it hurt them?
I mean this is a similar concern to sequential IDs: many apps do not want to leak them, and in some cases it might cause issues, but in general it doesn't matter.
UUIDv7 is a nice idea, and should probably be what people use by default instead of UUIDv4 for internal facing uses.
For the curious:
* UUIDv4 are 128 bits long, 122 bits of which are random, with 6 bits used for the version. Traditionally displayed as 32 hex characters with 4 dashes, so 36 alphanumeric characters, and compatible with anything that expects a UUID.
* UUIDv7 are 128 bits long, 48 bits encode a unix timestamp with millisecond precision, 6 bits are for the version, and 74 bits are random. You're expected to display them the same as other UUIDs, and should be compatible with basically anything that expects a UUID. (Would be a very odd system that parses a UUID and throws an error because it doesn't recognise v7, but I guess it could happen, in theory?)
* ULIDs (https://github.com/ulid/spec) are 128 bits long, 48 bits encode a unix timestamp with millisecond precision, 80 bits are random. You're expected to display them in Crockford's base32, so 26 alphanumeric characters. Compatible with almost everything that expects a UUID (since they're the right length). Spec has some dumb quirks if followed literally but thankfully they mostly don't hurt things.
* KSUIDs (https://github.com/segmentio/ksuid) are 160 bits long, 32 bits encode a timestamp with second precision and a custom epoch of May 13th, 2014, and 128 bits are random. You're expected to display them in base62, so 27 alphanumeric characters. Since they're a different length, they're not compatible with UUIDs.
I quite like KSUIDs; I think base62 is a smart choice. And while the timestamp portion is a trickier question, KSUIDs use 32 bits which, with second precision (more than good enough), means they won't overflow for well over a century. Whereas UUIDv7s use 48 bits, so even with millisecond precision (not needed) they won't overflow for something like 8000 years. We can argue whether 100 years is future proof enough (I'd argue it is), but 8000 years is just silly. Nobody will ever generate a compliant UUIDv7 with any of the first several bits aren't 0. The only downside to KSUIDs is the length isn't UUID compatible (and arguably, that they don't devote 6 bits to a compliant UUID version).
Still feels like there's room for improvement, but for now I think I'd always pick UUIDv7 over UUIDv4 unless there's an very specific reason not to. Which would be, mostly, if there's a concern over potentially leaking the time the UUID was generated. Although if you weren't worrying about leaking an integer sequence ID, you likely won't care here either.
100 years sounds short-sighted for something that's supposed to be "universally" unique. We're already having problems with the 32-bit Unix timestamp not being large enough. If you're willing to use 160-bit (or longer) identifiers, you might as well give a few more bits to the timestamp. Round it up to an even number of base-62 characters, too. That part of KSUID has always struck me as a weird decision.
I wish UUIDv7 pulled the version/variant bits up front, though, just to make sure that the identifiers don't all start with null bytes.
Apparently, humanity is damned to repeat it's mistakes over and over again.
"100 years should be enough" is what led us to a mountain of Y2K issues, because when would a two digit year ever be ambigious?
But I guess it's a psychological issue. Unless you're a megalomaniac, it's just natural to assume that your decisions won't matter much outside of your life and lifetime. And in that case, 100 years totally is enough because I probably won't live that long.
And even more, in a lot of cases, it's also the correct assumption and the project won't live longer than a few years.
So, thinking about it, unless you are developing a novel standard or something that you want the world to adopt, 100 years probably IS fine.
Unfortunately, KSUID wants to be a novel standard, so there's an issue.
If the version bits were up front, then switching to a hypothetical UUIDv8 in several years would be guaranteed to break the sortability. So I see that decision as a bit of future proofing.
How so? It seems like the only real use case for these timestamps is to get data from around the same time together. A second is fine for that. It's not about concurrency or avoiding collisions. A second can't handle that, but neither can a millisecond.
I’d think that the locality would only matter at the scale of your query. I’m sure someone has queries with a window less than a second and so much traffic, but it seems niche enough to not optimize the standard for it.
I could definitely be off. I work at a company that gets those levels of traffic but don’t deal with it directly.
For me the whole value prop for ULIDs is that they can be generated by any node in a distributed system without coordination, while roughly preserving time order. "Roughly" meaning: all IDs will be globally ordered at millisecond precision, subject to the accuracy of each node's system clock; and IDs from a specific node will be locally ordered, subject to the details of the monotonicity part of the ID generator. This is important for me, because most of the things I attach IDs to will happen many many many times per second.
If you need more than second precision then millisecond doesn't get you much further. The fact that the epoch ends in 120 years is a bit more worrying, but is also just about non-critical enough that it will be ignored for at least the next century.
Also, to all future historians of 2150, sorry about the mess, but yes we knew this was going to happen. Whatever it was.
Having any information, specifically time information, leaking from your systems may or may not have unanticipated security or business implications. (e.g. knowing when session tokens or accounts are created).