I'd be super interested in hearing from someone who has set it up whether it's something an experienced but generalized sys admin could build, or if you need an expert in the matter.
Yup, Ceph is going to be the solution to power it. There are quite a few options in the space but I was basically down to Swift (powers OVH) and Ceph (powers Digital Ocean Spaces) and OpenIO (power's OVH's new experimental offering).
Ceph has much more public resources available (SwiftStack recently got acquired, seemed like they had most of the knowledge in the space), and I've actually set it up a few times now thanks to the excellent Rook[0].
It's definitely a lot easier to set up with Kubernetes (the tradeoff being you need to understand Kubernetes), but it's definitely manageable for a generalized sys admin (albeit one with a bit more experience). I've written about the process:
I am in the process of migrating off of Rook Ceph after using it in production for two years. Setting it up is easy thanks to Rook, but wait until Ceph gets under load, then the real fun begins. If you only need object storage, I suggest looking into SeaweedFS[0]. It's a far more lightweight and performant solution.
Thanks for the suggestion -- I'm definitely aware of SeaweedFS and it was actually a really strong contender but I didn't choose it (and didn't mention it) for a couple reasons:
- Some sharp corner cases are definitely out there (issues/bug reports)
- Supported APIs aren't quite as extensive as the other options yet
- The requirements/expectations for
There's also some previous discussion from 2020[0] that was interesting. I actually planned to use SeaweedFS and dip my toes with what I'm calling the "CloseCache" feature -- on-demand nearby proxies for the data that's really in your object storage. The idea was to take advantage of seaweed's excellent proxying features and kick the tires at the same time.
Somewhat off topic but I'd love to pick your brain, would you mind if I sent you an email?
I'd be super interested in hearing from someone who has set it up whether it's something an experienced but generalized sys admin could build, or if you need an expert in the matter.