The example uses a different code per issue, for instance: "user/email_required". Most integrators will build their UI to highlight the input fields that contain an error. Making them parse the `code` field (or special-case each possible code) is pretty toilsome.
// from blog post
{
"code": "user/email_required",
"message": "The parameter [email] is required."
}
Wow, I had never seen an API with errors at this level of detail… I feel lucky when they at least use sane status codes instead of always giving back 200 and a maybe-json-maybe-plaintext-maybe-empty body…
I’d love to hear from anyone who has encountered APIs in the wild that actually implement this standard!
I used to work at Akamai and Problem Details is used in most of their APIs. It might have something to do with the fact that one of the RFC authors (Mark Nottingham / @mnot on HN) worked there for a while.
I've used both Problem Details and JSONAPI errors[0] which are basically the same idea (and I've used them plenty outside of JSONAPI-proper APIs). In both cases if you have a decent error-handling middleware there should be not much difference than outputting any other kind of errors.
One thing to keep in mind re. "maybe-json-maybe-plaintext-maybe-empty" responses is that the more complex your errors, the more likely the error handling encounters an error. If you're trying to send back some JSON or XML but it fails it's usually better to at least shove out a line of plain text and hope it reaches a human than to mask the real error with a second fallback in the "right format" but with unhelpful fixed content.
That is stupid. Most of our failed requests are logged and logs are only read by dashboards and alarms. Sure, you can have a friendly message too but formalizing the errors in a structured way simplifies things and also improves the performance when scanning through large amount of logs.
Such simple approach is limited only to errors without arguments.
For more complex use cases, where we would want an error message to indicate that field value was too long and in addition provide maximum field length, we would need to introduce new field in the error response.
While it is solvable by adding this information to client application side. It would create a situation where the logic is duplicated in two places (backend and client application)...
Also if we would want better UX, then we would need to display all errors at the same time in the form that is incorrectly filled. This would require changing error structure to return array of errors and it potentially create a breaking change in the API or would result in confusing structure that supports both, legacy and new format...
Some years ago, I wrote an article sharing the ideas on how REST API error structuring could be done depending on the complexity of application or service:
https://link.medium.com/ObW78jhDkob
From my experience it serves very well for validating inputs of large forms - e.g. loan application, international payments, etc.
If you need to validate some business logic like if sender's account has funds and receiver's account is not blocked, then this approach starts to look a bit strange. I would guess that the most of the time developers would implement checks so they would fail on first condition and would not check later one. This would result with single error in the array that kinda would look strange.
Validating business logic and returning several errors at the time requires good knowledge of the domain and in depth design of the system you are working. As this creates more complexity and slows down delivery, most of the time it is ditched and used only for particular use cases.
We could say that this would be more applicable to corporate solutions, but IMHO it really depends of the scale of the project, man power and the user experience you would want to create.
The Phoenix framework does this for forms. It requires a whole system built around a type called a Changeset that describes input parameters, their validity, and how to modify a struct to reflect the valid changes. In practice this ends up tightly coupled to the database layer for simplity.
It's only tightly coupled if you let it be. You can create changesets around any data shape, whether thats connected to your schema' table, a partial view of a table or entirely independent.
You need codes because the field isn't going to be 'email' for much longer than it takes for your management to realize that people outside of the US also have wallets.
Field ids are not (necessarily, especially when doing localization) something shown in the UI. The point made by the original commenter is that a field in the error should refer directly to which field has an issue. It does, via an id that happens to be "email". It's still up to the clients to decide how to represent that to the user, but they're given a distinct field rather than needing to infer which field an error code refers to.
(While the comment I replied to can be read differently, I assume we all know that changing actual field names (in APIs) depending on localization is nuts)
My view is that apis should simply return a number when an error occurs. The vendor should supply a list of error codes to its consumers, translated into as many languages as necessary. The developers who are consuming the api can then determine how best to present the error to its end users. A set of error numbers is tied to the version of the api that is being consumed so there should be no surprises.
In other words, if a set of people don't understand the message, make sure that no one understands the message (not even the people that normally just press the Google Translate button).
Is that what you're saying? Or did I misunderstand you?
It also doesn't hurt to repeat the HTTP status code in the JSON body - when you receive a response, the status code and entity body are coupled but even if the server logs the status code, they're often decoupled in the logging system - having both in one log entry is way easier!
Sometimes, I feel that we ought to have a simple protocol, on top of HTTP, to simply do remote procedure calls and throw out all this HTTP verbs crap. Every request is a http POST, with or without any body and the data transfer is in binary. So that objects can be passed back and forth between client and server.
Sure, there is gRPC, but it requires another API specification (the proto files).
There I said it. HTTP Verbs constrained REST APIS are the worst thing ever. I hate them.
They introduce un-necessary complexity, un-necessary granularity and they almost always stray away from the "REST principles". To hell with "Hypermedia" stuff.
I find it such a joy to program in server rendered pages. No cognitive overhead of thinking in "REST".
But, of course, all this is only where the client and server are developed by the same person / company.
For publishing data and creating API for third party use, we have no serious, better alternative to REST.
As someone who has spent a decade working with APIs, I 100% agree. The use cases that are a good fit for “RESTful” APIs pale in comparison to those that would benefit from RPC.
What is the point of having your client translate an action to some operation on a document (read or write), only to then have your server try to infer what action was intended by said document operation.
It pains me that this article doesn’t mention any of the trade offs of each suggestion (POST vs PUT vs PATCH and expandable objects, especially) or of using REST APIs generally.
+1, each time I return 404 for an object which is not found in the DB, the customer gets a red error message in their UI as if something failed more severely than an object being unavailable, and the metrics believe something is unavailable.
I bit my fingers every time I have mapped HTTP verbs and neither return codes, to REST verbs and codes.
Also, error codes at the API level often need a remapping when used in user context, for example if the OAuth token expires, we don’t say it the same way for an action of the user (then it’s mandatory) than when displaying data passively (in which case it shouldn’t be too red because the user may not care).
This answer is correct, but lacks context. REST wasn't conceived with APIs in mind. In fact, it's an awful fit for APIs, as many of the other comments point out. Rather, REST today is a buzzword that took on a life of its own, bearing only superficial resemblance to the original ideas.
HATEOAS is a generalization of how something like a website would let a client navigate resources (through hyperlinks). It requires an intelligent agent (the user) to make sense. Without HATEOAS, according to Roy Fielding, it's not real REST. Some poor misguided API designers thought this meant they should add URL indirections to their JSON responses, making everything more painful to use for those unintelligent clients (the code that is consuming the API). Don't do this.
If you must do REST at all - which should be up for debate - you should keep it simple and pragmatic. Your users will not applaud you for exhausting HTTP verbs and status codes. The designers of HTTP did not think of your API. You will likely end up adding extra information in the response body, which means I end up with two levels (status code and response) of matching your response to whatever I need to do.
If something doesn't quite fit and it looks ugly or out-of-place, that's normal, because REST wasn't conceived with APIs in mind. Don't go down the rabbit hole of attempting to do "real REST". There is no pot of gold waiting for you, just pointless debates and annoyed users.
> The central idea behind HATEOAS is that RESTful servers and clients shouldn’t rely on a hardcoded interface (that they agreed upon through a separate channel). Instead, the server is supposed to send the set of URIs representing possible state transitions with each response, from which the client can select the one it wants to transition to. This is exactly how web browsers work
The classic book on the subject is RESTful Web APIs[1], and it spends a while explaining HATEOAS by using the example of the web as we've come to expect it as the exemplar REST API using HATEOAS. I also have this essay[2] on HATEOAS in my open tabs, and it uses the example of a web browser fetching a web page.
This should come with a big warning for people looking to do real work. This is not what most REST APIs are like in practice, nor what they should be. The vast majority of REST APIs are RPC-like, because that's the pragmatic way to deal with the problem 99% of the time. The "REST" branding is just for buzzword compliance.
I used some API recently that returns URLs in the response body. It's really useful because they maintain those URLs and we don't have to rewrite our URL building code whenever the server side rules change. Actually we don't even have to write that code. It saves time, bugs, money.
I don't remember which API was that, I'll update the comment if I do.
Better yet, those URLs communicate what you may do.
Instead of building the logic to determine if, say, a Payment can be cancelled, based on its attributes, you simply check 'is the cancel link there'.
I find this a critical feature. Because between the backend, and various mobile clients, react, some admin, and several versions thereof, clients will implement such businesslogic wrong. Much better to make the backend responsible for communicating abilities. Because that backend has to do this anyway already.
I had a new hire on my team criticize the API we built for our product because we don't use put or patch, and we don't allow a GET and POST to share the same path. He said "it's not very RESTful"
I pointed him at HATEOAS and suggested if he wasn't familiar with it, he probably hasn't ever seen a truly RESTful API.
I don't think I convinced him that our approach is good (I'm not sure I am convinced either, but it works well enough for our purposes)
I do think I convinced him that "it doesn't correctly conform to a standard" isn't necessarily a useful critique , though. So that's a win.
I feel like the whole of the 1990s was devoted to this. How to serialize an object and then what network protocol should be used? But increasingly over time, between 2000 to 2005, developers found it was easier to simply tunnel over port 80/443. In 2006 Pete Lacey wrote a satire about SOAP, which is funny but also accurate, and look at how late people are to discover that you can tunnel over HTTP:
I was puzzled, at the time, why the industry was cluttering HTTP in this way. Why not establish a clean protocol for this?
But people kept getting distracted by something that seemed like maybe it would solve the problem.
Dave Winer used to be a very big deal, having created crucial technologies for Apple back in the 1980s and 1990s, and he was initially horrified by JSON. This post is somewhat infamous:
He was very angry that anyone would try to introduce a new serialization language, other than XML.
My point is, the need for a clear a clean RPG protocol, but the industry has failed, again and again, to figure out how to do this. Over and over again, when the industry gets serious about it, they come up with something too complex and too burdensome.
Partly, the goal was often too ambitious. In particular, the idea of having a universal process for serializing an object, and then deserializing it in any language, so you can serialize an object in C# and then deserialize it in Java and the whole process is invisible to you because it happens automatically -- this turned out to be beyond the ability of the tech industry, partly because the major tech players didn't want to cooperate, but also because it is a very difficult problem.
While I totally agree with the overkill that REST can be, I really do NOT agree with your statement:
> but it requires another API specification
This implies API-specs are part of the problem; and I think they are not.
Specs that have generators for client-libs (and sometimes even sever-stubs) are verrrrry important. They allow us to get some form of type-safety over the API barrier which greatly reduces bugs.
One big reason for me to go with REST is OpenAPIv3: it allows me to completely spec my API and generate clients-libs for sooo many languages, and server-stub for sooo many BE frameworks. This, to me, may weight up to the downsides of REST.
GraphQL is also picking up steam and has these generators.
JSON-RPC (while great in terms of less-overkill-than-REST) does not have so much of this.
My current company has settled on "We use GET to retrieve data from the server and POST to send data to the server, nothing else" because it was causing quite a lot of bikeshedding style discussions where people were fussing over "Should this be a post, put or patch"?
It all came to a head when someone wrote an endpoint using a PATCH verb that some people were adamant should have been a PUT.
It was among the most silly nonsense I have ever been a part of and these discussions have thankfully gone to zero since we decided on only GET and POST
Active use is really irrelevant if you only plan on using it inside a company, because you can implement a client and server within 30 mins to an hour, no external tools needed. The spec is clear and readable. It's excellent. We use it with a TypeScript codebase and just share the interfaces for the services in a monorepo. The spec is so simple it doesn't really need an update.
Arista Networking uses it in their eAPI protocol. It let's you have machine parsable outputs and avoid ye olde days of screen scraping network device outputs to view interface status and other details.
I believe most users make use of it via an open source json-rpc python lib. You can find a few examples online if you'd like to know more.
Agreed 100%. Slapping a REST api over a software is like reducing that software to a set of resources and attribute updates over those resources. And that never feels like the right way to talk with a software. That could be convenient for the majority of crud apps out there, but not everything we build is a crud system. For example how would you design operations on a cloud word processor as REST apis?
A better perspective would be, most softwares can be viewed as a set of domain specific objects and the set of operations (verbs) that can happen to those objects. These operations may not be a single attribute update, but a more complex dynamic set of updates over a variety of business objects. If you try to model this with a REST api, it either quickly becomes chatty or you end up compromising on REST principles.
GraphQL seems to make much more sense than REST, IMO.
I’m not sure what issue the verbs are creating, can someone help me get through my thick skull what this persons issue with them is? I don’t see how they add much complexity, just check the API docs and see what verb you need to use to perform a certain action.
In practice though, the tooling is cumbersome enough that you can't readily sub in some other protocol besides protobuf, json, and allegedly flatbuf. I've had little success finding ways to e.g. use msgpack as the serde. Maybe it's out there but I haven't found it.
>Sometimes, I feel that we ought to have a simple protocol, on top of HTTP, to simply do remote procedure calls and throw out all this HTTP verbs crap. Every request is a http POST, with or without any body and the data transfer is in binary. So that objects can be passed back and forth between client and server.
So the problem with “Data transfer is in binary” is that it really requires both the source and the recipients to be running the same executable, otherwise you run into some really weird problems. If you just embrace parsing you of course don't have those problems, but that's what you are saying not to do... Another great idea is for a binary blob to begin with the program necessary to interrogate it and get your values out, this has existed on CDs and DVDs and floppies and tape forever but the problem is that those media have a separate chain of trust, the internet does not, so webassembly (plus, say, a distributed hash table) really has a chance to shine here, as the language which allows the web to do this quickly and safely. But it hasn't been mature.
The basic reason you need binary identicality is the problem that a parser gives you an error state, by foregoing a parser you lose the ability to detect errors. And like you think you have the ability to detect those errors because you both depend on a shared library or something, and then you get hit by it anyway because you both depend on different versions of that shared library to interpret the thing. So you implement a version string or something, and that turns out to not play well with rollbacks, so the first time you roll back everything breaks... You finally solve this problem, then someone finds a way to route a Foo object to the Bar service via the Baz service, which (because Baz doesn't parse it) downgrades the version number but does not change the rest of the blob, due to library mismatches... Turns out when they do this they can get RCE in Bar service. There's just a lot of side cases. If you're not a fan of Whack-a-Mole it becomes easier to bundle all your services into one binary plus a flag, “I should operate as a Bar service,” to solve these problems once and for all.
> So the problem with “Data transfer is in binary” is that it really requires both the source and the recipients to be running the same executable, otherwise you run into some really weird problems.
I think you're misinterpreting "data transfer is in binary" with something like "a raw memory dump of an object in your program, without any serialisation or parsing step".
Some nice tips in here. However, tip 15, I strongly disagree with:
> 15. Allow expanding resources
I would suggest the opposite. A REST API should not return nested resources at all. Instead, and to stay with the example provided on the website, to obtain the "orders", the /users/:id/orders endpoint should be called.
It might be tempting to return nested resources, because clients would only have to make a single call.Technically this is true but once the domain of your API starts to grow, you will find that the interface will become increasingly muddled.
The suggestion provided (use query parameters) is basically a work-around. If you want to offer this to your clients, front your REST API with a GraphQL API instead. It is literally the problem that GraphQL solves. Keep your REST API clean and dumb and focused around CRUD.
Another two reasons to avoid nested resources, are performance and coupling.
A nested resource is hard to optimize. Simple, atomic, flat resources can be cached (by clients, proxies or server) much more efficient. Once you allow nested resources, there's no going back, as clients will depend on them. So you've effectively disabled lots of performance improvement options.
A nested resource implies data relations. Tightly coupled to a data model. One that will change over time, yet the api is hard to change. If you have a Project nested in your Users, and the business now needs multiple projects per user, this is hard to change with nested resources, but much easier with endpoints, the /users/:id/project can be kept and return e.g. the first project, next to a new /users/:id/projects.
And a third one is more architectural in nature. Your domain has bounded context (even if you haven't defined these explicitely). Within those bounded contexts, you'll have atomicity-constraints: transaction boundaries. Those should typically align, but are hard to find in a new project and will grow and shift over time.
To take that User/Project example: the Project and User have different transaction boundaries: it matters not that a User.username is updated while a Project.projectname is updated differently. But it does matter when a username and password get updated simultanous. And -for the sake of the xample- it matters that the user + users-roles must be within one transaction boundary.
By allowing nested resources on mutations you essentially break the bounded contexts and the transaction boundaries.
By not allowing nested resources on mutations, but allowing them on-read, you introduce inconsistencies: the resources on-insert differ from the resources on-read. This becomes even more problematic when you do allow certain nested resources on mutations but not on others (because, behind the scenes you've determined a bounded context/transaction boundary). To an API consumer this is entirely arbitrary. Why can I "PUT user.roles[1]" but not "PUT user.projects[1]", yet have both included at wish on-read?
With nested resources, here too, you paint yourself in corners. They take away a lot of choices you likely want to make in future. Being a bit more restraining in what you allow clients to do, keeps those options open for you. In this case, keeps the option to move transaction boundaries when business needs require (and they will).
I've consumed the kinds of APIs you're referencing and they are my least favorite. I would prefer a poorly documented API over one that returns me 15 IDs that I must look up in separate calls. I think there's a reason those APIs also tend to have rate limits that are way too low to be useful.
We're querying dozens, if not into hundreds, of GraphQL APIs for a couple of years now. Terrible DX, terrible performance, terrible uptimes. Everyone on the team, with no exception, hates them. Even a lot of those who produce them cobble together a bad REST implementation for parts of their own products.
Agreed even at rate limits comment - frequently, probably in moments of desperation, they rate limit according to Host header to keep their own products up and working.
Can you elaborate what's terrible about the developer experience? If anything it's much better than REST, even if the developer of the API doesn't bother with documentation, the GraphQL schema is fully usable as documentation. Plus the way to input and output data into the API is standardized and the same across all GraphQL API's.
> terrible performance
How? Is the API slow to respond? Or are you making too many calls (which you shouldn't do) increasing the round trip time?
> terrible uptimes
I fail to see how that uptime is related to GraphQL. REST API's can be just as unreliable, if the server is down, the server is down and no technology can fix that.
I would rather you not assigning hate to the entirety of autistic developers. There are plenty of autistic developers who are fully capable of designing great APIs with awesome usability. Being autistic has nothing to do with how APIs are developed.
Dear PAM69, this is great advice if you want to end up with a slow-to-load, low-performing web application that your customers complain about and hate using. But at least it'll adhere to a specific notion of architectural "purity", right? /s
Anytime clients need to make 15 async calls before the UI can be displayed, you're headed up the creek. Generally speaking, this is an anti-pattern. There are exceptions, but they're not the rule.
It's better to weigh the tradeoffs in any given situation and make a decision about bundling sub-resource references based on what you're optimizing for.
A few quick examples:
* Dev speed: Unbundled
* Quick-loading UI: Bundled
* Sub-resources are computationally-expensive to query? Unbundled
This has been my experience consistently throughout 20 years of web-tier development.
I think this is an unnecessarily harsh and sarcastic tone to take here.
The comment you're replying to set out specific reasons why they disagree with expanding/bundling sub-resources. It obviously depends on your use case - and in fact they say use GraphQL, which I heartily agree with - but the point is that you don't always know how the API is going to evolve over time, and keeping things unbundled tends to be a "no regrets" path, while bundling resources by default, in my experience, can lead to trouble later.
When the API evolves - as it probably will - using bundled resources ends up running the risk of either an inconsistent API (where some stuff is bundled and some isn't, and the consumer has to work out which), a slow API (because over time the bundled data becomes a performance burden), or a complicated API (where you need to embed multiple, backward-compatible options for bundling and/or pagination in a single operation). In addition, the bundling of resources commits you to I/O and backend resource consumption based only on an assumption that the data is required. None of this makes sense to me.
In practice, if you can keep your API latency reasonably low and take a little bit of care on the client side, there's no reason a user should notice the few milliseconds of additional latency caused by a couple more API calls during the page draw of an app.
It's not about architectural purity, it's about decomposing your application in a way that balances multiple conflicting needs. I agree with pan69, after many years of doing this in multiple contexts, my default has become to not bundle resources when responding to an API request.
One thing to add, is that there's nothing preventing you to invent a new noun for a particular rest resource that returns bundles of content. eg. /user/:id/dashboard - it makes it so this endpoint is not tied to eg. user/:id .. making that endpoint harder to change in the future, but also solves the issue mentioned by the rude comment above re: needing to perform a lot of separate rest calls.
Like I suggested, use GraphQL to solve this problem. I know front-end teams that run their own GraphQL server to abstract away clumsy APIs and to optimize client/server requests.
But those 15 requests are then all occurring over a local network (in the data center), not over the Internet.
The true power with GraphQL is that it might not even make all 15 calls because it will entirely depend on what you are querying for. E.g. if you query for a User but not the Orders for that User, then the request to retrieve the orders is simply skipped by GraphQL.
Also, of course, those 15 calls are occurring in parallel. I love how GraphQL makes all the complexity of marshalling data go away. Even when a GraphQL server is directly fronting an SQL database, I found the latency to be better than what I'd probably get if I was to code the calls manually.
I have only used Apollo for GraphQL, and I found a few things about it offputting (e.g. I need a 3rd-party library to figure out what fields the client actually requested). What GraphQL server do you use? Or is Apollo + Express generally a good "default" option for basic setups?
I used Apollo server as my first GraphQL implementation but found it really cumbersome. Plus Javascript fatigue hasn't caught up with the Apollo team yet, they like to change things around every so often. Now I use Postgraphile which basically creates the API for you based on a PostgreSQL database and extend it with plugins for custom operations. Hasura is also a good option but harder to extend.
True. But it sort of depends on the kind of relationship "orders" has. E.g. "orders" is a good example to use your suggested GET /orders?user_id=:id and that is probably because an order has a many-to-many relationship, e.g. with users and products (i.e. an order doesn't belong to neither user nor product). However, take something like an "address" which might belong to a user (one-to-many), i.e. the user has ownership of the relationship with an address, in that scenario you probably want to use GET /users/:id/addresses
But then again, when it comes to API's, domain modelling is the hard part and it is therefore the reason why you don't want to return nested results/objects.
> to obtain the "orders", the /users/:id/orders endpoint should be called
Ok, but an order has an array of order lines, and each order line has sub-arrays as well, like details of the packages it was shipped in etc.
So that might be 5-10 calls to get an order line, and for a few thousand lines we're looking at several tens of thousand calls to get a full order.
Secondly, you then make some changes and want to replace the order with the data you got. You then have to produce a delta and upload that. And those thousands of calls better be done in a transaction of sorts, otherwise you'll have big issues.
Seems easier to me to just be able to GET or PUT an entire order with all the details in one go.
> Ok, but an order has an array of order lines, and each order line has sub-arrays as well, like details of the packages it was shipped in etc.
> So that might be 5-10 calls to get an order line, and for a few thousand lines we're looking at several tens of thousand calls to get a full order.
You definitely need to pick and choose the granularity you offer. There's a level of normalization in data structuring that approaches absurdity and this would be a good example of an absurd case.
It is however an excellent demonstration of why making overbroad "you must never do X" rules is dangerous.
I think a decent test is to ask yourself the question: "How much data is here that I don't need?"
In the example referenced, if I'm looking at the full order history, I probably want to see only summary/header data with the option to view the full order details, so it wouldn't make sense to return to the user potentially a hundred thousand order lines' worth of data just because they're trying to view their history.
If I then want to view the /orders/:id for one specific order, at that point it does make sense to return all of the related lines, shipment details, etc.
Once your api query parameters reach a certain level of complexity, like such nested lookups, one should just consider giving direct proxied read access to the DB for clients that need it. Why reinvent your own query language. I Don’t know enough about graphql to compare it. Databases have access controls also.
> you will find that the interface will become increasingly muddled.
Will I? For example The Stripe API uses expand parameters and I prefer that approach to "atomic" REST or being forced to use GraphQL. There is a missed standardization opportunity for incremental graph APIs built on REST.
Can someone share how they handle versioning in their API when it comes to data model changes? For example `POST /users` now takes a required field `avatar_url` but it was not part of `v1`.
Since this field is validated in the DB, merely having `v1` `v2` distinction at the API layer is not sufficient. So I was thinking we will have to either 1) disable DB validations and rely on app validations or 2) run two separate systems (e.g., one DB per version) and let people 'upgrade' to the new version (once you upgrade you cannot go back).
Even though people refer to Stripe's API versioning blog, I don't recall any mention of actual data model changes and how it is actually managed
Those are possible, but ugly solutions. Two cleaner ones are either depracate and remove the v1 api altogether, or when inserting a record to the database from the v1 api, use a default dummy value for avatar_url.
Although I agree with the comments above, adding a field is also a breaking change, even with a default value. Especially prone to this is any openApi client (speaking from experience ...). Maybe filtering it out would be an actual solution without breaking anything. An API update shouldn't need to update my implementation because it's not working anymore, in that case it's a breaking change and a major version bump.
That's...really clever, but at the same time, I feel like there's a lot of assumptions baked into how Content-Types are used, and making your own content-type for each data model when it's all just application/json seems...wrong to me on an intuitive level, but I can't quite annunciate why.
I only half agree with the sentiment that /api/v1 violates REST patterns. I don't think there's any guarantee that /api/v1/bars/123 can't be the same object as /api/v2/bars/123.
I upvoted because I'm curious to hear what others are doing. We typically make sure to only make such breaking changes where either the now-required value or a sane filler value could be used. If it's the same API for the same purpose, it's usually not a stretch to assume the values for a new field are derived from some combination of an old field or else are primitive components of an old field such that they can be deduced and stubbed in your transition layer (or calculated/looked-up/whatever one-by-one as part of a bulk migration script during the transition). If your v2 is so drastic of a breaking upgrade that it bears no relationship to v1, I imagine your SOL and probably should have thought out your v1 or your v1-to-v2 story better, if only for the sake of the poor devs using your API (and you probably need separate tables at that point).
For other fields like your example of `avatar_url` I would use a placeholder avatar for all legacy users (the grey anonymous snowman profile comes to mind).
Thanks. This is a fair point. I made up the example only to illustrate the idea. Since Stripe is considered some sort of benchmark here I was curious to see how they tackle all the learnings they will have over time...I feel it is very hard to think through all the future cases especially when you are just about starting out with your product.
For example, in financial services and insurance, regs change and what data we need to collect change and sometimes their dependency will change. I am curious what's companies that have grown substantially had to do to their APIs.
No worries, I understood it was a throwaway example that shouldn't be looked at too closely. You just have to remember that your DB isn't a model of what you want to require from your customers but rather a model of what you actually necessarily have and don't have. A field like the ones you're talking about shouldn't be marked non-nullable in the database if there's a chance you actually don't have that data (and when you are suddenly required to collect something you didn't have before, you're not going to have it).
Coming at this from a strongly-typed background, you acknowledge the fact that despite new regulations requiring a scan of the user's birth certificate in order to get an API token, that field can't be marked as non-null if you don't in fact have all those birth certificates. You are then forced to handle both the null and not-null cases when retrieving the value from the database.
So your API v2 can absolutely (in its MVC or whatever model) have that field marked as non-null but since your API v1 will still be proxying code to the same database, your db model would have that field marked as nullable (until the day when you have collected that field for all your customers).
If a downstream operation is contingent on the field being non-null, you are forced to grapple with the reality that you don't have said field for all your users (because of APIv1 users) and so you need to throw some sort of 400 Bad Request or similar error because (due to regulations) this operation is no longer allowed past some sunset date for users that haven't complied with regulation XYZ. In this case, it's a benefit that your db model has the field marked as null because it forces you to handle the cases where you don't have that field.
I guess what I'm saying is the db model isn't what you wish your data were like but rather what your data actually is, whether you like it or not.
I think Stripe was originally built on Rails (can’t find anything to confirm that at the moment). But my guess is they enforce things at the app layer, since Rails didn’t really provide a good way to enforce things at the DB layer originally. They support very old API versions by transforming requests backwards and forward through a list of API version transforms, which also suggests to me that this sort of thing is enforced at the app layer rather than the DB.
I'm not saying you should do it this way, this is just how our startup (still very much in the "move fast and discover product fit" stage) does it. We have separate API models (pydantic and fastapi) and DB models (sqlalchemy). Basically everything not in the original db schema ends up nullable when we first add a field. The API model handles validation.
Then if we absolutely do need a field non-null in the db, we run a backfill with either derived or dummy data. Then we can make the column non-null.
We use alembic to manage migration versions.
But we aren't even out of v0 endpoint and our stack is small enough that we have a lot of wiggle room. No idea how scalable this approach is.
The downside is maintaining separate api and db models, but the upside is decoupling things that really aren't the same. We tried an ORM which has a single model for both (Ormar) and it just wasn't mature, but also explicit conversions from wire format to db format are nice.
Type
reference
Properties
Create, Filter, Group, Nillable, Sort
Description
The ID of the parent object record that relates to this action plan.
For API version 48 and later, supported parent objects are Account, AssetsAndLiabilities, BusinessMilestone, Campaign, Card, Case, Claim, Contact, Contract, Financial Account, Financial Goal, Financial Holding, InsurancePolicy, InsurancePolicyCoverage, Lead, Opportunity, PersonLifeEvent, ResidentialLoanApplication, and Visit as well as custom objects with activities enabled.
For API version 47 and later, supported parent objects are Account, BusinessMilestone, Campaign, Case, Claim, Contact, Contract, InsurancePolicy, InsurancePolicyCoverage, Lead, Opportunity, PersonLifeEvent, and Visit as well as custom objects with activities enabled.
For API version 46 and later, supported parent objects are Account, Campaign, Case, Contact, Contract, Lead, and Opportunity as well as custom objects with activities enabled.
For API version 45 and earlier: the only supported parent object is Account.
a) Use standardized error codes, not standardized error messages. Clients are responsible for internationalization, which includes presenting error messages in the user's language. If you document a set of error codes as an enum, the client can present a user-friendly error message in the user's language based on the error code. If there are dynamic parts of the error message, i.e. "404: There is no user with ID 123456 in the system", then the user ID should be extracted into the error response body, so that it can be provided correctly to the user in the user's language.
b) Pagination is the devil. The state of the server can change while the user is paginating, leading to fragile clients. Don't paginate your API. If you think you have a need to paginate, have one API call return a list of IDs, and have a separate API call return a list of resources for a given list of IDs, where the second API call accepts some maximum number of IDs as a query parameter. This ensures consistency from one API call to the next.
> Pagination is the devil. The state of the server can change while the user is paginating, leading to fragile clients. Don't paginate your API.
The problem is not pagination, and the solution is not to avoid pagination. The problem is offset-based pagination, and the solution is to use cursor-based pagination.
> If you think you have a need to paginate, have one API call return a list of IDs, and have a separate API call return a list of resources for a given list of IDs, where the second API call accepts some maximum number of IDs as a query parameter.
This has the same problem as you described above. The state of the server can change between fetching the list of IDs and operating on them.
Cursor-based pagination doesn't solve your state issue, it forces the server to create a copy of state for the cursor request. This is complex to implement correctly - for example, if the user does not actually paginate through all the entries, when do you dump the unused cursor? If the user issues the same request over and over, do you return a cached cursor or re-copy the state into a new cursor? If you re-copy the state, are you defended from a malicious actor who sends 1,000 new requests? Of course, all these concerns can be mitigated, but it's easier to just design without pagination in the first place if you can.
> The state of the server can change between fetching the list of IDs and operating on them.
Right, for example, a returned ID may have been deleted before the user can issue a query for that resource. But this is usually far more comprehensible to clients, particularly if the resource requires authorization such that only the client is permitted to delete that resource.
It is quite simple to implement pagination in the application layer using a unique record identifier (primary key or ULID or ...) as an anchor for the navigation. From that unique ID, we can then fetch the previous `n` or the next `n` records, depending on the direction of the navigation.
This way, the server remains stateless, since the anchor (possibly sent as an encoded / obfuscated token, which can include some other parameters such as page size) is supplied by the client with each pagination request.
Pagination only makes sense in the context of an ordered collection; if there is no stable sort order then you can’t paginate. So you identify the last record seen with whatever fields you are ordering by, and if the last record has been deleted, then it doesn’t matter because you are only fetching the items greater than those values according to the sort order.
Anyway, there is plenty of documentation out there for cursor-based pagination; Hacker News comments isn’t the right place to explain the implementation details.
You should use stable page boundaries instead of cursors. If you are returning results by timestamps, the page boundary should be the timestamp and secondary ordering keys of the last row returned.
Cursors take too many server resources and require client affinity.
I think pagination is only predictable under these conditions:
1) The offset used for the next fetch must be based on a pointer to a unique key. We can't rely on the number of rows previously seen.
With this rule, deletes which occur to rows in previous pages will not cause unpredictable contractions.
2) The paged result set must be sorted based on a monotonically increasing field, like created_at, plus enough other fields to create a unique key. You could lean on the PK for this, i.e.: ORDER BY (created_at, id) ASC.
With this rule, new inserts which occur during page enumeration will only affect unseen pages (and we'll see them eventually)
SELECT *
FROM orders
WHERE (created_at, id) > (:offset_created_at, :offset_id)
OR (
:offset_created_at IS NULL
AND :offset_id IS NULL
)
ORDER BY (created_at, id) ASC
LIMIT :page_size
Any pagination is brittle. Regardless of whether its by page, cursor or something else. You can't have equal sized pages if there is a risk that there are deletions on a previous page or at the exact index you use as a cursor etc.
The solution is usually simple: assume it doesn't matter. Write a spec for your feature and explicitly state that "In the solution we assume it doesn't matter if the same record is ocassionally reported twice or a record is missing from the pagination in some cases". Done.
Pagination by offset can be bad. Pagination by cursor which is specifically designed for your list of resources is perfectly reasonable. Use stable pagination queries, not ?*
There's typically business-logic ways to require a range to be specified, and then you specify a maximum range. For example, if the resource in question is audit events, you require the user to specify a time range, up to a maximum of a day, or seven days, or whatever performance / budget constraints allow for.
So if I specify a range and something changes between in that range while I'm interacting with those, sounds like same problem is back? Just same issue; different package :)
If I do a search, a paginated search result can have an ID so I can paginate between the data without a new data messing up my pagination.
But for normal entities, simple pagination is (mostly) more than enough.
The solution you are describing is overkill and almost no benefit at all.
A few things I see rarely discussed that I often do:
- Always use a response envelope
- HTTP status is not the same as your application status. I always include a "status" field in the response envelope (OP recommends standardizing errors but I think standardize all of it)
- Always have an unique error code for every error your API can return (they should also be URL safe ("some-thing-went-wrong"), basically an enum of these should exist somewhere, the more specific the better.
- Offer OpenAPI whenever you can.
- There is no such thing as a publicly exposed "private" API. If it can be hit (and is not protected), it eventually will be.
- Do blackbox testing of your API, E2E tests are the most important kind of test you could have.
- (controversial) build in special test/debug endpoints -- these help with blackbox testing.
Here's an example straight from code I've rewritten at least 5 times because I'm allergic to saving myself time:
export enum ErrorCode {
InvalidEntity = 1,
NotSupported = 2,
UnexpectedServerError = 3,
InvalidRequest = 4,
Validation = 5,
// ...
}
export enum ResponseStatus {
Success = "success",
Error = "error",
}
export class ResponseEnvelope<T> {
public readonly status: ResponseStatus = ResponseStatus.Success;
public readonly data: T | Pruned<T> | null = null;
public readonly error?: {
code: ErrorCode,
message: string,
details: object | string,
status: ResponseStatus,
};
constructor(opts?: Partial<ResponseEnvelope<T>>) {
if (opts) {
if (opts.status) { this.status = opts.status; }
if (opts.data) { this.data = opts.data; }
if (opts.error) { this.error = opts.error; }
}
if (!this.data) { return; }
// Prune if the thing is prunable
this.data = prune(this.data);
}
}
Hopefully this isn't too hard to grok, it's Typescript.
BTW, If you're cringing at the number scheme for the ErrorCode enum, don't worry I am too. This is why I prefer to use strings rather than numbers, and it basically just ends up being stuff like "invalid-entity", etc.
I find envelopes to be unnecessary cruft that just junk up otherwise clear code. And I think packing metadata into an envelope along with the actual data keeps people from thinking clearly about their own APIs, mostly with respect to ambiguity surrounding the word “error”. Validation errors are errors and network failures are errors, but they’re very different animals and should never be conflated.
I don’t want to check for a status code in metadata ever, when an HTTP status is provided. I don’t want to read a time stamp if that time stamp is simply a property of the request. However if the time stamp is a property of the data, then I care - but then it shouldn’t be part of the metadata.
Well I imagine this is why sages like uncle bob and sam newman will always be needed.
> Validation errors are errors and network failures are errors, but they’re very different animals and should never be conflated.
> I don’t want to check for a status code in metadata ever, when an HTTP status is provided.
These seem like conflicting views. If the HTTP status is all you check you must be conflating application/operation status and network status.
> I don’t want to read a time stamp if that time stamp is simply a property of the request. However if the time stamp is a property of the data, then I care - but then it shouldn’t be part of the metadata.
Sure -- I'd like to say that I think the disagreement here boils down to this question: should metadata be represented in data responses.
Correct me if I'm reading you incorrectly, but it seems like you're arguing that metadata should not be represented in the response, only data. I'd argue that the in-practice and academic standards dictate the opposite, with standards like hypermedia, json schema, HAL, JSON:API, OpenAPI, etc.
If it's just a question about the degree of metadata that should be present then that's one thing, but it sounds like you're against it in general. Once you have any metadata you must have an envelope of some sort, by definition (whether it's a good or bad one), as far as I can see.
I think I'm with Jack on this one, at least when it comes to REST interfaces. When I query GET /bar/123, I want an object with a Bar protocol/content-type. I don't want a Envelope[Bar]. What if it's any data type other than json-isomorphic, e.g. image or video? Is /videos/123.mp4 going to return a json object like
You already have an envelope, it's the HTTP protocol. The real trick is generically mapping data structures to HTTP responses with headers. In fact HTTP-response-shaped-objects make halfway decent general purpose envelopes in the business logic itself. They are basically Result[T, E] but with even more metadata available.
We wrap the response body because it's important to give clients of an API endpoint, something that they will use to query/modify data, a clear indicator of the application response status. We do this in the response body because http status codes aren't good enough, and people tend to miss headers. It's hard to miss it when it's part of the body.
And no, we don't do that for static content for a simple reason: static content isn't served from our API servers.
I would mostly agree except in 1 case, streaming data is easier without an envelope. Making some array inside an envelope stream is usually more code and messier than just getting some metadata out of header. So if you have something like data integration endpoints and you expect someone could pull many megs of records, consider no envelope.
“Always” was probably a bit too absolute in my original phrasing.
I agree, and would go so far as to say streaming data would probably have a different delivery mechanism all together - SSE/websockets/etc. if you were doing long polling I may still want to put the metadata in the body but I agree it could be kept out of sight quite nicely in the headers.
> 5xx for internal errors (these should be avoided at all costs)
An anti pattern I’ve often seen has devs avoiding 5xx errors in bizarre ways. I would change the above to make to have monitoring in place to address 5xx errors. By all means, let your code throw a 500 if things go off the rails.
Author here. I generally agree - IF something goes terribly wrong within the application, a 500 is definitely the way to go. The thing is - these just shouldn't happen that often. Because in that case we're having a bigger problem. :D For example I have never seen Stripe return a HTTP 500 from their API. Thing just works.
Generally agree with everything else but strongly disagree with pagination envelopes and offset pagination. Allow keyset pagination using ordering and limiting query parameters.
It’s a good posting. I do many of these things, but not all.
I’ve been designing SDKs for a long time; APIs, for a somewhat shorter time.
I don’t like “pure” RESTful APIs, because I feel as if they are “human-hostile.” I tend to follow the patterns set by companies like Google, where the stimulus is a fairly straightforward URI (with parameters, as opposed to a block of XML or JSON —if possible, as I know that we often still need to send big data as transaction data, which can be a pain, depending on the method), and the response is a block of structured data. That also makes it a lot easier for API clients to build requests, and allows the server to “genericize” the processing of requests.
> 10. Use standardized error responses
Is something I do, along with a text header payload, describing internal information about the error. It is my experience that this text payload is never transferred, and I am forced to send a text response via the body, if I want to know internal information. That can be a pain, as it often breaks the transaction, so I have to play with the server and client, frequently, using a cloned server instance.
I should note that my forte is not backend development, so the chances are good that I am not familiar with all the tools at my disposal.
I don’t feel like doing that in a comment, but feel free to check out some of my work. The BAOBAB server[0] is one of my more recent examples. I’m using a modified version as the backend for the app I’m developing, now. It works great.
I should note that the BASALT layer uses “plugins,” that can be extended to include “pure” REST APIs. I just haven’t found these practical, for my purposes.
Many rest apis are lazy and developer friendly, not consumer friendly.
If you have related resources, let’s say, product and product options as two distinct endpoints:
- /api/product
- /api/options
Then, and I want to be clear here, it is impossible for a client to perform an atomic operation on multiple distinct objects types.
Let’s say the client needs to add a product with a single option or fail.
You can create a product.
You can create an option.
You can add an option to a product.
…but, at some point in time, a product will exist with no options on it.
This kind of “pure” rest api is simply convenient to the developer because they push the problem of object consistency to the client.
…but that’s not a client concern.
If a product needs to be associated with an option on creation, then your api should offer that as an endpoint.
It doesn’t meet the “standard” for a rest api?
Too bad.
Write APIs that do what the customer / consumer needs not lazy APIs that just make your life easier.
I’ve worked with a lot of APIs and you know what I do not give the tiniest moment of care about?
Consistency.
I do. Not. Care. If POST /api/foo creates an object, or if the endpoint is /api/foo/create.
Messy? Inconsistent?
I don’t care. Just put it in the documentation and I’ll find it and use it.
…but if your api forces object relational consistency onto me as an api consumer, you have no idea how much pain and hassle you’ve caused me having to implement my own set of fake transactions over your stupid api.
Please, write APIs for consumers, not according to “the rules” of rest APIs.
Generally internal APIs are developed alongside one or two apps. They don't need a pure, resource-oriented, API that perfectly represents the domain models but ignores how the API is used in practice.
A good example is the tip on PUT vs PATCH to update objects. That seems to be missing the point. Why are you forcing the clients to calculate the correct resource fields to PATCH to the server? This is supposed to be an API, not a database. Just expose methods that correspond to the actions that the users will perform.
Sure, HTTP only has 5 verbs, but that doesn't mean your API should solely consist of five possible interactions on a resource.
You can also just like...make up your own verbs, the HTTP police won't arrest you, though this rustles the jimmies of HTTP purists. No guarantees any middleware will support it, and this is probably a bad idea if you actually do this in public apis, it's more so "hey, these verbs aren't magic incantations, it's just a string which gets switch-cased".
Same with HTTP status codes. There's obviously some well known ones but you can just like...make up your own.
If options are children of products and not many-to-many then the only logical way to have two endpoints in the API is if the parent (Aggregate root in DDD speak) items are readable/writable as a consistent tree, while the query for child items in /api/options is a read only api.
There is nothing nonstandard about such a REST api. If a product must be associated with an option then the /product/create endpoint should only accept a product with an option already attached.
> This kind of “pure” rest api is simply convenient to the developer because they push the problem of object consistency to the client.
It's not like REST means that to be "pure" or "standard" you end up exposing all your database tables with read/write and just let callers CRUD anything they want, including creating inconsistent data?
> “the rules” of rest APIs.
The rules are very simple: the REST api should expose and enforce the consistency rules of the business logic!
Agree that the tendency exists but don't see the tradeoff between transactional and messy in this example--you should be able to create a new product with options [...] in the same request, perhaps using /options to ascertain what's available before posting to /products, depending on the situation.
maybe a more illustrative example of where this can be messy is where a customer has placed an order for a bundle of products A B and C, and there's no "order bundle" API that can do that operation atomically, instead the client code trying to place the order has to call a bunch of random stateful create-product APIs to attempt to construct each product in the bundle, and deal with the mess if suddenly some bundle items error while others succeed. bonus points if some of the create-product APIs are not idempotent.
"congratulations, we've provisioned the bundle A, C, C, C that you ordered! sorry about B!"
i was in the process of writing a snarky reply describing how the next level enterprise approach is to host api/product in microservice A and api/options in microservice B, each with their own internal data store using completely different tech stacks.
but, if you've already suffered through the pain of attempting to implement some kind of best-effort ad-hoc distributed commit logic in your client code, then needing to call two completely different services for api/product and api/options doesn't really make anything _worse_.
Everyone has different experiences but my view of the problem is that it's not so much down to developer laziness as it is technological ignorance and reactive direction from higher up - the people who actually fund and approve the work.
Outside of the bigger providers I don't think most APIs are designed with general consumption in mind. The impetus usually seems to be reactive - "we've got this idea for a mobile app using data from our legacy WMS/CRM/whatever, and it needs to be finished yesterday!"
So generally (not always but usually) what I've seen is whatever the underlying system supports and requires gets reflected into the API, and there's minimal proper engineering or sensible abstraction, partly because of time limits and also because the requirements are discovered mid-flight.
"Oh... Products have options in a separate entity? Oops. Uh, we'll add an options endpoint. Easy."
Several years later, the app is mildly successful, the business decides to whitelabel the API for some of their B2B clients, there's no budget for a redesign, "It works doesn't it? If it ain't broke don't fix it..."
Where possible I try to design and build APIs with some forethought and push for as much information on requirements and capabilities of existing systems beforehand but quite often the stakeholders don't really know and don't want to pay for a few weeks of in-depth analysis.
The endless facepalming of hearing the cycle of "Don't overengineer it" when I ask too many questions people can't answer, through to "Why didn't we pick up on this requirement" and having to listen to "we can all learn from this" when something fails because of some if/else statement buried in the legacy business logic, right back to "Don't overengineer it" when you propose the clean way of encapsulating that in the API, has left a permanent indent on my skull. So much leakage of black-box logic into the clients which bloats frontend logic and provides plenty of double handling and scope for unknown business logic to be missed, and no one in charge really gives a shit other than the developers and engineers, and that's fine because we can deal with it and it's their fault for not thinking of it anyway... they're the smart ones right?
You'd think this is just a problem with SMEs but some of the biggest national and multinationals I've worked with have a ton of horrid shit in their API designs. One industry-wide government-mandated reporting spec you'd post a SOAP envelope that wraps an undocumented XML document (they had an example document and the envelope was documented, but no doctype or anything to go off for the contents), a major real estate interchange format has you polling a REST API for an update flag, then loading and saving CSVs over FTP. Some godawful affiliate marketing API essentially just exposed their nth normal form relational database as an API. One government department just used HTSQL for their API using a python script to reflect some of their silos into PGSQL and called it a day (even though whether or not your interactions would work or made sense depended entirely on their internal application logic which wasn't represented or documented).
And too many projects where it turns out the promised APIs didn't even exist until the app was well into development. "Don't worry about it, that's their problem and gives us an excuse when the inevitable delays occur..."
No wonder we're hearing of massive breaches daily, from top to bottom the entire industry is barely held together by glue and sticky tape. It seems like an engineering problem but I think it goes much higher than that, it's a cross-industry failure in planning and management. Engineers are not given the time and budget to do the job that the data deserves. The market self corrects though, massive breaches have closed more than one business down for good.
It feels futile but I do support and encourage people to work towards just reasonable and well-documented endpoints, I doubt there's one true standard, but just some level above slapping it together at the last minute would be nice. I don't care if it's JSON or XML, or whether paths follow some semantics or another. As long as the relationships, types, and mandatory vs optional fields are explained clearly and succinctly, and the auth mechanism isn't a nightmare, and I can work with it.
Sorry for the rant this is just one area that triggers me. I've seen too much bad practice, and fighting against it for decades and still hearing the same lack of concern in the architecture phase has left me more than a little jaded.
> it is impossible for a client to perform an atomic operation on multiple distinct objects types.
> It doesn’t meet the “standard” for a rest api? Too bad.
It is perfectly fine to model the creation of a product with options via a new resource. Another possibility is to model this as a transaction, a series of requests. Both meet the goals of atomicity and also following restful constraints.
But since you disallowed those possibilities as the premise, there is no constructive way forward from there. Nice construction of a straw-man, must feel satisfying to see it topple over, but who do you hope to impress with that?
A lot of implementation are actually based on an ISO draft which changed in final release. But most dev do not have access to the spec. For example, timezone can only be specified as offset in the standard, while implementations accept name.
Globally avoid ISO for software, it is non free crap.
Also, do not use those standard for dates with timezone in the future. Use wall time/date and location. As timezone can change way more often than you think.
My own experience is that (unix) timestamps are much less error-prone than textual representations like ISO 8601 and such. A field like `update_time_seconds` is clear and easy to convert into any representation. This is what the Google API Improvement Proposals recommends in most cases, though civil timestamps are also described.
https://google.aip.dev/142
Of course, when preparing queries against the API, you may need a helper to build valid timestamps. But this is mostly relevant to the discovery phase which isn't automated. And textual dates also have drawbacks, for instance the need to encode it when used a an URL parameter.
Why would you ever want anything to do with timezones in an API? Even for appointments, it's still a timestamp. Timezone should be applied the very last moment, as part of date formatting for output on the client.
If you allow your users to create appointments this much in advance and then they miss them, it's a UX problem, not an API design problem.
Why do you say appointments are a time stamp? They are made with humans in mind, people dealing with a local time zone.
In the example in the post you’re replying to, the time zone change means the time stamp would be wrong by an hour. This doesn’t mean that people should show up an hour earlier or later than initially intended!
Because they are a timestamp. In an overwhelming majority of cases when you're going to deal with time in software, it's a timestamp. Timezones are a client's concern. You ask the user to input the time in their local timezone, then convert that to unixtime and send it to the server.
Timezone isn't a simple constant offset either. For places that use DST, any decent date library would take that into account. If you entered a date in summer while it's winter, it would still result in a correct unixtime.
Well, how do operating systems handle this sort of stuff? With updates. When it's at all possible to update. There are many devices that don't receive any updates at all any more. There are some devices that weren't designed to be updatable to begin with.
It's a necessarily manual process and it creates a lot of confusion either way. And it's sufficiently rare to ignore. And these changes tend to be announced in advance.
This argument is complete non-sense (just because a device doesn't receive software updates it doesn't mean it can't receive date/time updates).
But who cares. We told you, that there are valid use-cases where timestamps are not the best solution and brought examples. If you don't want to see them, it is your loss.
So you'd rather push out an update every time for your app to rewrite the times it has stored to match the timezone change, instead of "just" updating the timezone database and the app automatically doing the right thing?
And we've had examples in the past years of this happening with only weeks of notice, so no, it's not something that never happens.
While I agree with the most aspects of this very good blog post, there is a minor detail that I would like to note:
While pagination is important, there is another possibility of pure size - it is using cursors, like mentioned in the JSONAPI specification[1] (containing many of the hints in the topics' post) and in this blog post[1]
Author here. Thanks for the nice words - I appreciate it. I've read that a couple of times in the comments already. Definitely something worth considering, Stripe does this as well. Thanks for your comment.
I really hate when APIs use different api-key headers depending on the role of the consumer.
It is very annoying when you get dates that are not in the ISO format. There are reasons to not use UTC everywhere. One should make that decision.
The reason why many APIs use POST instead of DELETE is that POST is said to be more secure.
Many APIs that I use do not have neither of PATCH, PUT or DELETE. An order for instance will have an order status resource that one just keeps adding status entities to. In general, well-designed systems minimize the need for changing data.
Authentication and Authorization are two subtly different things. In this case, you may want an API key (Authentication) to be required to ensure things like rate limiting is enforced, but then want proof that the call is operating on a user, or is a machine-to-machine interaction which OAuth2 Bearer tokens work nicely for (Authorization)
We use the form that represents a collection of objects. For example books and clothing both refer to a collection of things so I take them as valid forms
Helped me get started with API design in my early career. Learning from other existing APIs helps too. such as stripe, github, shopify. Any others?
Something that We do at my current job:
* set a standard and stick with it. tweak it if needed. we even have naming standard on some of the json key. for example, use XXX_count for counting. when it doesn't make sense, use total_XXX.
* Document your API, we use postman and code review API changes too.
We don't use PATCH but use a PUT for partial objects. We have validator code at every endpoint and we validate both creates and updates. When a PUT comes in, the validator knows what can and can't be changed. Depending on your role, the validator lets you change certain things can be updated as well. A PATCH would need these too and now you have more code to deal with. Also, it requires the developer to now worry that they have all the fields for a complete object or not.
> I don't bother with these methods and stick to GET and POST.
Most people don't bother with them. If you need caching or want to be able to manipulate params in the browser/link to resources, use GET.
This idea that you need additional verbs for web services is a classic case of in-theory vs in-practice. Introducing non-trivial complication for very tiny benefit is a strange tradeoff.
Something I wonder is how to design search/list endpoints where the query can be long (a list of IDs, ex: /users?ids=123e4567-e89b-12d3-a456-426614174000,123e4567-e89b-12d3-a456-426614174001,123e4567-e89b-12d3-a456-426614174002,...), so long that it can exceed the url max length (2048), after 50 UUIDs, you can quickly exceed that length, so GET is not ideal, so which method, SEARCH with a body? POST with a body?
There was a new RFC published a few months ago to address this use case. It defines a new HTTP method QUERY, which is defined to be safe & idempotent like GET and explicitly allows a request body like POST. See https://www.ietf.org/id/draft-ietf-httpbis-safe-method-w-bod...
I would go with POST with a body in that case, where I interpret is as a "new search item" and use GET to scan through results, if there are many results available. I don't think I've needed something like this more than once or twice though. From users perspective, asking information on specific 50 items at once is not something commonly done.
it's used for an export feature, where we join data from the client-side, not ideal, but for now it's a good compromise in terms of complexity for the API and client
If QUERY is too new, I was always a fan of base62 for URLs, but base64 and straight binary encoding could do well for compacting UUID lists, they are essentially just giant verbosely written integers.
Computers are the main consumers of APIs, and ISO 8601 is far from machine-readable.
For example, in Elixir, DateTime.from_iso8601/1 won't recognize "2022-03-12T07:36:08" even though it's valid. I had to rewrite a chunk of Python's radidjson wrapper to 1-9 digit fractional seconds (1).
I'm willing to bet 99% of ISO8601 will fail to handle all aspects of the spec. So when you say "ISO8601" what you're really saying is "our [probably undocumented, and possibly different depending on what system you're hitting] version of the ISO-86001 spec."
2. The ISO date format is "big endian" by default, which makes it trivial to chunk and compare just specific date parts. Want all events that occurred on the same (UTC) date? Just do dateField.substring(0, 10). Everything in the same year? dateField.substring(0, 4).
They can include a local timezone. Sometimes there's a big difference between 12am in UTC+0 and 3am in UTC+3 despite representing the same instant in time.
True enough, but I would still recommend that API responses normalize to UTC (Z suffix) in the general case and document as much, and if actually returning a timestamp with a specific timezone, document the intended meaning.
+1, if your application cares about time then you should just tell your users all dates will be normalized to UTC and they are responsible for displaying them in a preferred time zone.
Time zone rules change relatively frequently - future dates may be better stored in the appropriate time zone, optionally along with the UTC offset at the time of recording, so you don't report incorrect information when the rules do change on you.
Past dates should always be in UTC, for the same reason - timezone rule changes are sometimes even retroactive.
My recommendation: Use PUT everywhere instead of POST. PUT has to be idempotent, so if the request fails, the client can simply post it again. This solves issue like worrying about creating a second copy of an item if a POST timed out.
Using PUT to create elements means that the client has to supply the ID, but this is easily solved by using GUIDs as IDs. Most languages have a package for create a unique GUIDs.
I don't think we should strive to remove non-idempotent cases. If something is not idempotent does not mean it is bad. It just means that request should be handled differently.
In your example (and I ask this as I remained confused after also reading SO):
Let's say that you need the client to provide the ID in the request body.
In this case, how is using PUT when creating a new resource idempotent if the ID should be unique and you have a constraint on the DB level for example?
What happens when the second call will be made with the same ID?
If I execute the following sequence:
Call 1: PUT resource + request.body {id: 1} => a new record is created so the state of the system changes
Call 2: PUT resource + request.body {id: 1} => no record is created, maybe the existing one is updated
IMO this is not idempotent nor should it be. Creating a new resource is not an idempotent operation.
I also don't like that depending on the state of the DB two requests with the same attributes will have different impacts on the system.
In my mind as a consumer of an API it is simpler: POST it creates a new record so I know what to expect, PUT updates an existing one.
Upserts are more powerful because the client can always generate a new uuid to get the POST behavior you desire, but the reverse is not the case: there is no straightforward way to safely retry timeouts, partial success, side effects, etc.
You say the same request has a different impact on the system, but what that means is the system converges to the requested state.
Maybe this is overkill if your retry strategy is to have a user in the loop, but I don't see how it's simpler for the client even without any retry behavior (if/else new/reuse uuid vs if/else put/post).
Author here. I read that a couple of times already. RFC 3339 is what you should use - I should probably change that (it's what I actually meant). :) Thanks for pointing that out.
The PUT vs PATCH is debatable in different levels. One simple issue is how to resolve complex merges in a PATCH. For example if we patch a key that contains a list, what will be the expected result?
Even beyond PUT vs PATCH I found this statement to be rather naive:
"From my experience, there barely exist any use cases in practice where a full update on a resource would make sense"
Just in finance I can think of *dozens* of use cases around invoicing and purchasing *alone*. A lot of times thousands of resources may need just one field "corrected", but hundreds others "reset" to ensure the proper retriggering of routing, workflows, and allocations. That the resources need to exist as originals is incredibly important for these kinds of things.
Side note: I find it quite annoying that web devs have hijacked the term “API“ as a synonyme for “web API“, to the point where many devs are not even aware of the original broader meaning.
Nevertheless, as a (largely) non-web dev I found the article interesting and useful.
Author here. Haha! You are right. I admire Stripe's API and there's absolutely a lot of things in it that are my "benchmark" when it comes to good APIs. However, they return UNIX timestamps, for example, which I don't like that much (as stated in the post). They also use POST for updates, because, if I recall correctly, PUT/PATCH wasn't a thing yet when Stripe was built initially. Thanks for the comment.
The author has a poor understanding of HTTP and adjacent standards. I find it is so insufficient that he should not dispense advice yet, he must learn much more, especially about the essential yet completely unmentioned parts of a Web system: media types, hyperlinks, Link relations, cacheability. Critique:
2. ISO 8601 is a shit show of a standard. Last time I surveyed, there was not a single compliant implementation in the world. Recommend "Date and Time on the Internet: Timestamps" <http://rfc-editor.org/rfc/rfc3339> instead. This standard is public/not encumbered by stupid copy fees and is restricted to a much more reasonable profile that can actually be implemented fully and in an interoperable fashion.
5. This goes against the design of the Web. Do not version URIs. If the representation changes and is not backward compatible, change the media type instead, e.g. by adding a profile <http://rfc-editor.org/rfc/rfc6906>. You can serve multiple representations on the same URI and vary on the request headers (e.g. Accept-).
6. HTTP already contains a customisable mechanism. Use the standard header <http://rfc-editor.org/rfc/rfc7235#section-4.2>, not the custom header `Api-Key` which is not interoperable.
7. "Don't use too many [HTTP status codes]": why? This opinion is not backed up with an explanation. The correct advice is: use as many status codes as arise from the requirements. `422 Unprocessable Entity` is useful in nearly every Web system.
8. What does "reasonable" mean? This lacks an explanation.
11. "It's a good idea to return the created resource after creating it with a POST request": why? This opinion is not backed up with an explanation. If you follow this advice, the user agent cannot programmatically distinguish between a representation reporting on the requested action's status, and the representation of the newly created resource itself. Use the Content-Location header <http://rfc-editor.org/rfc/rfc7231#section-3.1.4.2> to make that possible, use the Prefer header <http://rfc-editor.org/rfc/rfc7240> to give the user agent some control over which representation to respond with.
12. "Prefer PATCH over PUT": disagree, ideally you offer both since there is a trade-off involved here. The downsides of PATCH are not mentioned: the method is not (required to be) idempotent meaning it becomes moderately tricky to keep track of state, and the client is required to implement some diff operation according to the semantics of the accepted media type in the Accept-Patch header which can be difficult to get right.
15. Instead of complicating both server and client, the better idea is to simply link to related resources. We are not in the 1990s any more. A well written Web server offers HTTP persistent connections, HTTP pipelining, all of which make round-trips cheap or in the case of HTTP/2 server push, even unnecessary. Benchmark this.
1. + 9. betrays a weird obsession with naming. The advice is not wrong, but it shifts attention away from the genuinely useful areas that benefit much more from a careful design: the media types, the hyperlinks between and other hypermedia mechanisms for traversing resources (forms, URI templates). If an inexperienced programmer follows the advice from the article, he will the idea to spend lots of time mapping out resources in the identifier space and methods and possible responses for documentation purposes. This is both a waste of time because the single entry point is enough and the rest of the resources can be reached by querying the server via OPTIONS and traversing hyperlinks etc., and dangerously brittle because of strong coupling between the server and the client.
Mostly common-sense things, but I can't wait for the community to stop trying to use PUT, PATCH, DELETE and the like. There's a reason that in 2022 web forms only support GET and POST (and implicitly, HEAD).
There is no mention of HTML in these recommendations.
The recommendation is perfectly good for any API using HTTP as the substrate. The wisdom of using HTTP as a protocol substrate is questionable, but having made that decision the verbs it supplies work perfectly well.
Incidentally, the HTML living standard supports three form methods, not two: get, post, and dialog. Which rather reinforces the point that HTML != HTTP.
I don’t know what OP thinks is the reason, but the actual reason is CORS.
Web Devs (such as me) have asked for e.g. DELETE as an acceptable formmethod (e.g. https://github.com/whatwg/html/issues/3577) however WHATWG always pushes back citing security concerns such as CORS.
I suspect this is not what OP had in mind since it is trivial to send a DELETE request with a simple JavaScript:
If CORS can be weakened in any simple way with that HTRP-DELETE method, then your database could simply disappeared via HTTP-DELETE method.
Besides, webmasters’ HTTP DELETE method is a different domain scoping issue than the web developers’ HTML/JavaScript FORM deleteThis row-entry approach.
I marvel at designers trying to flatten the scoping/nesting of abstractions without factoring apart the disparate error and protocol handling.
The example uses a different code per issue, for instance: "user/email_required". Most integrators will build their UI to highlight the input fields that contain an error. Making them parse the `code` field (or special-case each possible code) is pretty toilsome.
Make it parseable: In addition, I:* rewrote `message` to be an acceptable error message displayed to (non-technical) end-users
* moved message to be the first field: some developer tools will truncate the JSON response body when presenting it in the stack trace.
---
As an added bonus, structured data allows you to analyze `error` frequencies and improve frontend validation or write better error messages: https://twitter.com/VicVijayakumar/status/149509216182142976...