I don’t understand why everyone wants to replicate AWS APIs for things that are not AWS.
S3 is a horrible interface with a terrible lack of features. It’s just file storage without any of the benefits of a file syste - no metadata, no directory structures, no ability to search, sort, or filter.
Combine that with high latency network file access and an overly verbose API. You literally have a bucket for storing files, when you used to have a toolbox with drawers, folders, and labels.
Replicating a real file system is not that hard, and when you lose the original reason for using a bucket —- because your were stuck in the swamp with nothing else to carry your files in — why keep using it when you’re out of the mud?
Does your file system have search? Mine doesn’t. Instead I have software that implements search on top of it. Does it support filtering? Mine uses software on top again. Which an S3 api totally supports.
Does your remote file server magically avoid network latency? Mine doesn’t.
In case you didn’t know, inside the bucket you can use a full path for S3 files. So you can have directories or folders or whatever.
Some benefits of this system (KV style access) is to support concurrent usage better. Not every system needs it, but if you’re using an object store you might.
What personal experience do you have in this area? In particular, how have you handled greater than single-server scale, storage-level corruption, network partitions, and atomicity under concurrent access?
You have server-client state. The concept of opened files, directories, and their states. Locks. The ability for multiple writers to write to the same file while still providing POSIX guarantees.
All of those need to correctly handle failure of both the client and the server.
CephFS implements that with a Metadata server that has lots of logica and needs plenty of RAM.
A distributed file system like CephFS is more convenient than S3 in multiple ways, and I agree it's preferable for most use cases. But it's undoubtedly more complex to build.
It's a legitimate question and I'm glad you asked! (I'm not the author of Garage and have no affiliation).
Filesystems impose a lot of constraints on data-consistency that make things go slow. In particular, when it comes to mutating directory structure. There's also another set of consistency constraints when it comes to dealing with file's contents. Object stores relax or remove these constraints, which allows them to "go faster". You should, however, carefully consider if the constraints are really unnecessary for your case. The typical use-case for object stores is something like storing volume snapshots, VM images, layers of layered filesystems etc. They would perform poorly if you wanted to use them to store the files of your programming project, for example.
> I don’t understand why everyone wants to replicate AWS APIs for things that are not AWS.
It's mostly just S3, really. You don't see anywhere near as many "clones" of other AWS services like EC2, for instance.
And there's a ton of value on being able to develop against a S3 clone like Garage or Minio and deploy against S3 - or being able to retarget an existing application which expected S3 to one of those clones.
S3 exposes effectively all the metadata that POSIX APIs do, in addition to all the custom metadata headers you can add.
Implementing a filesystem versus an object store involves severe tradeoffs in scalability and complexity that are rarely worth it for users that just want a giant bucket to dump things in.
The API doesn't matter that much, but everything already supports S3, so why not save time on client libraries and implement it? It's not like some alternative PUT/GET/DELETE API will be much simpler-- though naturally LIST could be implemented myriad ways.
You wouldn't want your "interactive" user filesystem on S3, no, but as the storage backend for a server application it makes sense. In those cases you very often are just storing everything in a single flat folder with all the associated metadata in your application's DB instead
By reducing the API surface (to essentially just GET, PUT, DELETE), it increases the flexibility of the backend. It's almost trivial to do a union mount with object storage, where half the files go to one server and half go to another (based on a hash of the name). This can and is done with POSIX filesystems too, but it requires more work to fully satisfy the semantics. One of the biggest complications is having to support file modification and mmap. With S3 you can instead only modify a file by fully replacing it with PUT. Which again might be unacceptable for a desktop OS filesystem, but many server applications already satisfy this constraint by default
Because at this point it's a well known API. I bet people want to recreate AWS without the Amazon part, and so this is for them.
Which, to your point, makes no sense because as you rightly point out, people use S3 because of the Amazon services and ecosystem it is integrated with - not at all because it is "good tech"
Storage is generally sticky but I wouldn’t be so quick to dismiss that reason because it might explain why anything would fail to displace it; a bunch of software is written against S3 and the entire ecosystem around it is quite rich. It doesn’t explain the initial popularity but does explain stickiness. Initial popularity was because it was the first good REST API to do cloud storage AND the price was super reasonable.
Oh, I’m definitely not saying integration or compatibility have nothing to do with it - only that “horrible interface with a terrible lack of features” seems impossible to reconcile with its immense popularity.
First mover + ecosystem wins over interface. And also, I really don’t have so many issues with the interface as others seem to and we had to implement it from scratch for R2. There’s features we extended on top of S3 so it’s not really the interface so much but rather that Amazon doesn’t really invest in adding features so much. I suspect it’s really hard for them to do so. We also exposed a native JS API that was more ergonomic but the S3 endpoint saw more traffic which again points to the ecosystem advantage - people have an S3 codebase and want to do as a drop in replacement as possible.
They were first mover for cloud storage and combining object storage with HTTP. Previous attempts were WebDAV (not object storage and very complicated to the point that kk one implemented) and Hadoop which didn’t have HTTP and couldn’t scale like S3’s design.
S3 is a horrible interface with a terrible lack of features. It’s just file storage without any of the benefits of a file syste - no metadata, no directory structures, no ability to search, sort, or filter.
Combine that with high latency network file access and an overly verbose API. You literally have a bucket for storing files, when you used to have a toolbox with drawers, folders, and labels.
Replicating a real file system is not that hard, and when you lose the original reason for using a bucket —- because your were stuck in the swamp with nothing else to carry your files in — why keep using it when you’re out of the mud?