They sell servers, but as a finished product. Not as a cobbled together mess of third party stuff where the vendor keeps shrugging if there is an integration problem. They integrated it. It comes with all the features they expect you to want if you wanted to build your own cloud.
Also, they wrote the software. And it's all open source. So no "sorry but the third party vendor dropped support for the bios". You get the source code. Even if Oxide goes bust, you can still salvage things in a pinch.
Ironically this looks like the realization of Richard Stallman's dream where users can help each other if something doesn't work.
Its a huge deal. I'm biased though because my own takes on how things should evolve were very similar. I was however completely unsuccessful in getting those ideas into production! And that, that is a huge deal. Through out my career it has been interesting to meet people with great ideas and then they are unable to get them into production, and when the idea does come into production everyone feels like "Wow, this is so obvious why didn't we do it sooner?" and some folks are banging their head against the wall :-).
One of the more interesting discussions I had during my tenure at Google was about the "size" of the unit of clusters. If you toured Google you got the whole "millions of cheap replaceable computers" mantra. Sitting in Building 42 was a "rack" which had cheap PC motherboards on "pizza dishes" without all that superfluous sheet metal. Bunches of these in a rack and a festoon of network cables. What are the "first class" elements of these machines? Compute? Networking? Storage? Did you replace components? Or a whole "pizza slice" (which Google called an 'index' at the time). Really a great systems analysis problem.
FWIW I'm more of a "chunk" guy (which is the direction 0xide went) and less of a "cluster" guy (which is the way Google organized their infrastructure). A lot of people associated with 0xide are folks I worked with at Sun in the early days and during that period the first hints of "beowulf" clusters vs "super computers", was memory one thing (UMA) or did it vary from place to place (NUMA). I have a paper I wrote from that time about "compute viscosity" where the effective compute rate (which at the time largely focused on transactional databases) scaled up with resource (more memory more transactions/sec for example) and scaled down with viscosity (higher latencies to get to state meant fewer transactions/sec) Sun was invested heavily in the TPC-C benchmarks at the time but they were just one load pattern one could optimize for.
These guys have capitalized on all that history and it is fucking amazing! I just hope they don't get killed by acquisition[1].
[1] KbA is a technique where people who are invested in the status quo and have resources available use those resources to force the investors in a disruptive technology to sell to them and then they quietly bury the disruptive technology.
Can you clarify a bit on what you mean by "chunk" guy? Are you alluding to the ability to distribute work by an isolation mechanism such as cgroups vs machine a-la borg/google?
More on infrastructure composition software is an abstraction above that.
Is the unit of composition a rack (chunk), a server (smaller chunk), or a blade (smallest chunk)? In what I think of as classic systems architecture you've got a 'store' (storage), 'state' (memory), 'threads' (computation), and 'interconnect' (fabrics). In the 90's a lot of folks focused on fabrics (Cray, Connection Machine, Sun, etc) somewhat on threads (compute blades), and state came along for the ride. How these systems were composed was always a big thing, then along came the first Beowulf clusters that used off the shelf motherboards (a "chunk" of threads/state/store) with a generic fabric (Ethernet). Originally NASA showed that you could do some highly parallel processing on these sorts of systems and Larry and Sergei at Stanford applied it to the process of internet search.
Collectively you have a 'system resource' and with software you can make it look like anything you want. When you do compute with it, its performance becomes a function of its systems balance and the demands of the workload. Its all computer sciencey and yes there is a calculus to it. This isn't something that most people dive into (or are even interested in[1]) but it was one of the things that captured my imagination early on as an engineer. I was consumed with questions like what was the difference between a microprocessor, a mini-computer, a workstation, and a mainframe? Why do they each exist? What does one do that they other can't? Things like that.
[1] At Google I worked in what they called 'Platforms' early on and clearly most of the company didn't really care about the ins and outs of the systems bigtable/gfs/spanner/etc ran on, they just wanted APIs to call. But they also didn't care about utilization or costs. By the time I left some folks had just figured out (and one guy was building his career on) the fact that utilization directly affected operational costs. They still hadn't started thinking about non-uniform rack configurations for different workloads.
> I just hope they don't get killed by acquisition.
Private equity in the US has collectively determined that no company shall exist outside of investment ownership. I don't know what the ownership structure looks like, but generally speaking, it seems that nearly everyone has a "fuck you" number. Now that Oxide is venturing into Dell and HP's turf, I worry someone will get a fix on what Brian's number is.
Coupling vs Decoupling is not some one-sided thing. It's a major trade-off.
One of the most obvious examples of the problem with this approach is that they're shipping previous generation servers on Day 1. One can easily buy current generation AMD servers from a number of vendors.
They will also likely charge a significant premium over decoupled vendors that are forced to compete head-to-head for a specific role (server vendor, switch vendor, etc).
Their coupling approach will most likely leave them perpetually behind and more expensive.
But there are advantages too. Their stuff should be simpler to use and require less in-house expertise to operate well.
This is probably a reasonable trade-off for government agencies and the like, but will probably never be ideal for more savvy customers.
And I don't know how truly open source their work is but if it's truly open source, they'll most likely find themselves turned into a software company with an open core model. Other vendors that are already at scale can almost certainly assemble hardware better than they can.
> will probably never be ideal for more savvy customers
IDK about every use case, but slightly older generations of CPUs would affect me roughly zero. I'm sure there are things so compute-intensive that one would care very much, but a lot of people probably wouldn't bat an eye about that, and not because they're unsavvy.
To the extent that these things are supported as a whole by the vendor rather than a bunch of finger pointing though, that could be massive, specifically in terms of how many staff members you could "not hire" compared to if you had to employ someone to both build and continually maintain it.
I'm posting this not to invalidate what you're saying, just to say that a little predictable upfront amount of money (the premium) will be spent very happily by lots of people who value predictability and TCO over initial price.
If you're not rapidly scaling it probably doesn't matter. But if you're still buying (and maybe even using) Haswell CPUs in 2023, you may be missing out in a big way.
A moderately large Haswell cluster is equivalent in power to a moderately powerful modern server.
No not buying new, just using what was bought years ago. It still works, it does the job. Is it the best performance per watt, clearly no but the budget for electricity and the budget for new capital expenses are two different things.
If you go on Google cloud and select an E2 instance type (atleast in `us-central1` where my company runs most of it's infra) you'll usually get Broadwell chips.
> They will also likely charge a significant premium over decoupled vendors
It seems like they're trying to hit a middle ground between cloud vendors and fully decoupled server equipment companies.
Using Oxide is likely cheaper over the life of the hardware than using a cloud vendor. A company who already has in-house expertise on running racks of systems may be less the target market here than people who want to do cloud computing but under their own control.
> A company who already has in-house expertise on running racks of systems may be less the target market here than people who want to do cloud computing but under their own control.
True, but Oxide may find themselves competing against Dell or HP if they adopt Oxides software for their respective servers. Additionally, Oxide may find itself competing against consultants and vendors in specialized verticals (e.g. core Banking software + Oracle DB + COTS servers + Oxide software). Oxide, and their competitors are going for people who used to buy racks of Sun hardware.
HP and Dell would have to fundamentally change the way they design hardware and software to be that kind of threat, and if that ever happens I think I would be pretty okay with that outcome.
> One of the most obvious examples of the problem with this approach is that they're shipping previous generation servers on Day 1. One can easily buy current generation AMD servers from a number of vendors.
> Their coupling approach will most likely leave them perpetually behind
This is a startup that took years to get their initial hardware developed. The time between this version and the version using the next version of AMD chips will be shorter than the time it took to develop this product. This is not an inherent issue with coupling vs decoupling.
Also, most servers are rarely running on the most recent cpus anyway. At least in companies I've worked at with on-site hardware they're usually years (sometimes even a decade) out of date getting the last life sucked out of them before too many internal users start complaining and they get replaced.
Coupling requires more integration work, including writing and testing custom firmware. Oxide will be a tiny market player for a long time, even if things go very well. Are AMD and Broadcom really going to spend as much time helping Oxide as they do helping Dell? Of course not, Oxide's order volume will be a rounding error.
I'm sure they'll improve their processes over time but the lag will probably always be a non-zero value. Hopefully they'll be able to keep it low enough that it's not an important factor but as a customer it's certainly something one should consider.
It would be surprising if they don't run into some nasty issue that leaves their customers 6+ months behind on servers or switches at some point.
From listening to their talks they've actually gotten pretty good direct responses from AMD and AMD likes them quite a bit. They've done what no other system integrator has done and brought up the CPU without using AMD's AGESA firmware bootloader. By simplifying the system they've reduced the workload on what they need to handle.
As to your second point, unless AMD somehow becomes supply constrained and only wants to ship to their most important customers first I don't see a future where there would be any lag. Again, the delay this time is from how long it took from company start until product release. Future delays will be based on the time it takes from them getting early development parts to released products, which they could even possibly beat Dell to market on given the smaller company size and IMO more skilled employees.
> It would be surprising if they don't run into some nasty issue that leaves their customers 6+ months behind on servers or switches at some point.
I mean they've already hit tons of nasty issues, for example finding two zero-day vulnerabilities in their chosen security processor. They've shown they can work around issues pretty well.
> it would be surprising if they don't run into some nasty issue that leaves their customers 6+ months behind on servers or switches at some point.
I just think your premise is wrong - most customers don't care about not having the absolute latest and greatest. Indeed they will often avoid them because
1. They are new so more likely to have as yet undiscovered issues ( hardware or drivers ).
2. If you buy top end, they sell at a premium well above their performance premium.
ie the customers who are perennially chasing the latest hardware are in the minority.
Most customers care about having the best of the available options. Rarely would any company deliberately choose to be behind where their competitors can be.
1. The way to run into undiscovered issues is to choose a completely custom firmware/hardware/software stack that almost no one else in the world is running.
2. Not sure where you're getting this from. There is almost always a price:performance calculation that results in current generation smashing the previous generation with server and switch hardware. Often this means not buying the flagship chips but still the current generation.
And a major reason to get off old generations of hardware is that they become unavailable relatively quickly. It's always easier to buy current generation hardware than previous generation hardware, especially a couple years into the current generation. This has nothing to do with chasing the latest hardware.
> And a major reason to get off old generations of hardware is that they become unavailable relatively quickly.
That's not in the customers interests per se- in fact it's a pain. Having control of their own stuff could mean they could offer a much longer effective operational life.
> The way to run into undiscovered issues is to choose a completely custom firmware/hardware/software stack that almost no one else in the world is running.
What breaks stuff is change - sure when they are starting up it's higher risk - but again if they can manage the lifecycle better, not have change for changes sake, then they could be much more reliable.
> Not sure where you're getting this from.
I was talking about not taking the flagship stuff - which is typically a few months ahead of the best price/performance stuff.
If they standardize and open the server shape and plug interface then it gets really cool. Then I could go design a GPU server myself and add it to their rack. The rack is no longer a hyperconverged single-user proprietary setup and becomes something that can be extended and repurposed.
I don't see it as a big deal - rather, I see it as a huge amount of venture cap spent on some very bright people to build something no one really wants, or, at best, is niche.
Also, it has little to do with the cloud; it is yet another hyperconverged infra.
Weirdly, it is attached to something very few people want: Solaris. This relates to the people behind it who still can't figure out why Linux won and Solaris didn't.
When you're deploying VMs, which is the use case here, the substrate OS becomes significantly less important. Those VMs will mostly just be linux.
Yes they are using illumos/Solaris to host this but they don't sell on that, they sell on the functionality of this layer — allowing people to deploy to owned infra in a way that is similar to how they'd deploy to AWS or Azure. How much do you ever think about the system hosting your VM on those clouds? You think about your VMs, the API or web interface to deploy and configure, but not the host OS. With Oxide racks the customers are not maintaining the illumos substrate (as long as Oxide is around).
You could be right about demand, there is risk in a venture like this. But presumably the team thought about this - I think folks who worked at Sun, Oracle, Joyent, and Samsung and made SmartOS probably developed a decent sense of market demand, enough to make a convincing case to their funders.
I have a feeling they knew exactly from the start who their customers would be: People who have the budget to care about things like trust and observability in a complex system. But these would also be the kind of customers who require absolute secrecy and so this why you don't hear about them even though they might have bankrolled a sizable portion of the operation. Just like the first Cray to officially be shipped was actually serial number 2...
> When you're deploying VMs, which is the use case here, the substrate OS becomes significantly less important. Those VMs will mostly just be linux.
Now you need to know both the OS they chose and the OS you chose...
(No, I don't believe it'll be 100% hands-off for the host. This is an early stage product, with a lot of custom parts, their own distributed block storage, hypervisor, and so on.)
This true for other hypervisors too. Enterprises are still paying hundreds of millions to VMware, who knows what's going on in there?
I wouldn't have picked Opensolaris, but it's a lot better than other vendors that are either fully closed source, or thin proprietary wrappers over Linux with spotty coverage and you're not allowed to touch the underlying OS for risk of disrupting the managed product.
What's more important is that the team actually knows Illumos/Solaris inside out. You can work wonders with a less than ideal system. That said, Illumos is of high quality in my opinion.
Seems risky considering how small of a developer pool actively works on illumos/Solaris. The code is most definitely well engineered and correct, but there are huge teams all around the world deploying on huge pools of Linux compute that have contributed back to Linux.
They had a bug in the database they are using that was due to a Go system library not behaving correctly specifically on illumos. They've got enough engineering power to deal with such a thing but damn..
Linux grew up in the bedrooms of teenagers. It was risky in the era of 486 and Pentiums. The environment and business criticality of a $1-2M rack-size computer is quite different.
I had similar thoughts about VMware (large installations) back in the day. Weird proprietary OS to run other operating systems? Yet they turned out fine.
This appears to be a much better system than VMware, is free as in software, and it builds upon a free software operating system with lineage that predates Linux.
I say this in the most critical way possible, as someone who has built multiple Linux-based "cloud systems", and as a GNU/Linux distribution developer: I love it!
It was totally a risky choice for companies in the 1990s and early 2000s to put all their web stuff onto Linux on commodity hardware instead of proprietary Unix or Windows servers. Many did it when their website being up was totally mission critical. Lots did it on huge server farms. It paid off very quickly but it's erasing history to suggest that it didn't require huge amounts of guts, savvy and agility to even attempt it.
Indeed, for me GNU/Linux was always a cheap way to have UNIX at home, given that Windows NT POSIX support never was that great.
The first time I actually saw GNU/Linux powering something in production was in 2003, when I joined CERN and they were replacing their use of Solaris, and eventually alongside Fermilabs came up with Scientific Linux in 2004.
Later at Nokia, it took them until 2006 to consider Red-Hat Linux a serious alternative to their HP-UX infrastructure.
Completely tangential, but this reminds me of an interview I had for my first job out of college in 1995. I mentioned to the interviewer that I had some Linux experience. "Ah, Linux" he said. "A cool little toy that's gonna take over the world".
In hindsight of course it was remarkably prescient. This from a guy at a company that was built entirely around SGI at the time.
This is a skewed view - the critical piece that made Linux "enterprise-ish" was the memory management system that was contributed by IBM, part of the SCO lawsuit
Back in the day... Sun Micro was a GOAT and pushed the envelope on Unix computing 20-30 years ago. Solaris was stable and high performing.
I don't run on-prem clusters or clouds but know a couple people who do and, at large enough scale, it is a constant "fuck-shit-stack on top of itself" (to quote Reggie Watts). There is almost always something wrong and some people upset about it.
The promise of a fully integrated system (compute HW, network HW, all firmware/drivers written by experts using Rust wherever possible) that pays attention to optimizing all your OpEx metrics is a big deal.
It may take Oxide a couple more years to really break into the market in a big way, but if they can stick it out, they will do very well.
It won't. In the same way that AWS customers aren't debugging hypervisor, or Dell customers aren't debugging the BIOS, or Samsung SSD customers aren't debugging the firmware. Products choose where to draw the line between customer-serviceable parts and those that require a support call. In this case, expect Oxide to fix it when something doesn't work right.
When Apple supports OSX for consumers, they don't exactly surface the fact that there's BSD semi-hidden in there somewhere.
That's because they own the whole stack, from CPU to GUI and support it as a unit. That's the benefit of having a product where a single owner builds and supports it as a whole.
My impression of Oxide is that that's the level of single source of truth they are bringing to enterprise in-house cloud. So, I strongly doubt the innards would ever become customer-facing (unless the customer specifically wants that, being open source after all).
Apple is a horrible example, with Apple when you have a problem, you often end up with an unfixable issue that Apple won't even acknowledge. You definitely don't want to taint Oxide's reputation with that association.
As for why I think Helios will become customer facing: Oxide is a small startup. They have limited resources. Their computers expensive enough to be very much business critical. You'll get some support by Oxide logging in remotely to customer systems and digging around, but pretty soon the customer will want to do that themselves to monitor/troubleshoot the problems as they happen.
Imagine you're observing a recurring but rare I/O slowdown that seems to trigger under some certain conditions, and tell me a competent sysadmin wouldn't want to log in on all the related boxes (client Helios, >=3 server Helioses for the block store) and look at the logs & stats.
> As for why I think Helios will become customer facing: Oxide is a small startup. They have limited resources.
Have you looked at the pedigree of many of the people behind the project? I don't say this because "these guys smart", but because these guys bent over backwards for their customers when they were Sun engineers. Bryan didn't write dtrace for nothing.
> Imagine you're observing a recurring but rare I/O slowdown that seems to trigger under some certain conditions, and tell me a competent sysadmin wouldn't want to log in on all the related boxes (client Helios, >=3 server Helioses for the block store) and look at the logs & stats.
I think you're simultaneously over-estimating and under-estimating the people who will deploy this. There's a lot of companies who would want a "cloud in a box" that would happily plug hardware in and submit a support ticket if they ever find an issue, because their system engineers either don't have the time, desire, or competence (unfortunately common) to do anything more. The ones who are happy to start debugging stuff on their own would have absolutely wonderful tooling at their fingertips (dtrace) and wouldn't have any issue figuring out how to adapt to something other than Linux (hell, I've been running TrueNAS for the better part of a decade and being on a *BSD has never bothered me).
Apple is a great example of the benefits of an integrated system where the hardware and software are designed together. There are tons of benefits to that.
What makes Apple evil (IMO, many people disagree) is how everything is secret and proprietary and welded shut. But that doesn't take away from the benefits of an integrated hardware/software ecosystem.
Oxide is open source so it doesn't suffer from the evil aspect but benefits from the goodness of engineered integration. Or so I hope.
In practice I don't think it's as good as in theory. I had Apple Macbook Pro with Apple Monitor, and 50% of the time when unplugging the monitor the laptop screen would stay off. Plugging back in to the monitor wouldn't work at that point so all I could do was hold the power button to force it off and reboot. That's with Apple controlling the entire stack - software, hardware, etc.
I think the real benefit is being able to move/deprecate/expand at will. For example, want an app that would require special hardware? You can just add it. Want to drop support for old drivers? Just stop selling them and then drop (deprecate) the software support in the next release.
I fully agree about the evilness, and it baffles me how few people do!
Android is potentially a better example. Compare Android to trying to get Linux working on <some random laptop>. You might get lucky and it works out of the box or you might find yourself in a 15 page "how to fix <finger print reader, ambient light sensor, etc>" wiki where you end up compiling a bunch of stuff with random patches.
Afaik Android phones tend to have a lot more hardware than your average laptop, too (cell modem, gps, multiple cameras, gyro, accelerometer, light sensors, finger print readers)
Apple is the survivor of 16 bit home micros integration, PC clones only happened as IBM failed to prevent Compaq's reverse engineering to take over their creation, they even tried to get hold of it afterwards via PS/2 and MCA.
As we see nowadays on tablets and laptops, most OEMs are quite keen in returning back to those days, as otherwise there is hardly any money left on PC components.
Funny how you mentioning BSD got me to thinking of Sony Playstation and Nintendo Switch. Which are proprietary and not user serviceable. A Steam Deck, Fairphone, or Framework laptop is each less proprietary and more FOSS stack, and user serviceable. Which a user may or may not want to do themselves; at the very least they can pay someone and have them manage it.
Also, Apple is just the one who survived. Previously I'd have thought of SGI, DEC, Sun, HP, IBM, Dell some of whom survived some not.
Those three consumer products I mentioned each provide a platform for a user and business space to floroush and thrive. I expect a company doing something similar for cloud computing to want the same. But it will require some magick: momentum, money, trust. That kind of stuff, and loads of it. (With some big names behind it and a lot of FOSS they got me excited, but I don't matter.)
If you have a bug in how a lambda function is run on AWS, do you find yourself looking for the bug in firecracker? It is open source, so you technically could, but I just don't see many customers doing that. Same can be said about KNative on GCP.
Their choice in foundation OS (for lack of a better term) really should not matter to any customer.
Ok but then that is purely additive then, right? Like, "have to find someone with Illumos expertise to fix something that was never intended to be customer-facing" may not be easy, but is still easier than the impossibility of doing the same thing on AWS / Azure / Google Cloud.
Right, who wants or benefits from open source firmware anyway.
Also there are many situations where renting, for example a flat makes a lot of sense. And there are many situations where the financials and or enabled options of owning something make a lot of sense. Right now, the kind of experience you get with AWS and co. can only be rented, not bought. Some people want to buy houses instead of renting them.
Well, you can buy your own hardware and set it up with OpenStack and use it as a private cloud. Companies like Canonical or Redhat make a lot of money by providing software (mostly open source) to support exactly that use case.
> Well, you can buy your own hardware and set it up with OpenStack and use it as a private cloud. Companies like Canonical or Redhat make a lot of money by providing software (mostly open source) to support exactly that use case.
Sure you can, but then who will diagnose and fix your hardware/OS interaction problems when you have parts from five vendors in the mix?
If you haven't lived through this, the answer is: nobody. Everyone points fingers at the other 4 and ignore your calls.
Back in the day you could buy a fully integrated system (from CPU to hardware to OS) from Sun or SGI or HP and you had a single company to answer all the calls, so it was much better. Today you can't really get this level of integration and support anymore.
(Actually, you probably can from IBM, which is why they're still around. But I have no experience in the IBM universe.)
This is why Oxide is so exciting to me. I hope I can be in a company that becomes a customer at some point.
>Sure you can, but then who will diagnose and fix your hardware/OS interaction problems when you have parts from five vendors in the mix?
Dell is a single vendor that will diagnose and fix all of your hardware issues.
With Oxide you're locked into what looks like a Solaris derivative OS running on the metal and you're only allowed to provision VMs which is a huge disadvantage.
I run a fleet of over 30,000 nodes in three continents and the majority is Flatcar Linux running on bare metal. Also have a decent amount of RHEL running for specific apps. We can pick and choose our bare metal OS which is something you cannot do with Oxide. That's a tough pill to swallow.
> Dell is a single vendor that will diagnose and fix all of your hardware issues.
I've been a Dell customer at a previous company. I know for a fact that's not true.
I had a support ticket for a weird firmware bug open for two years, they could never figure it out. I left that job but for all I know the case is still open many years later.
Dell doesn't know how to fix things like that because they don't design and engineer the systems they sell. Dell is a reseller who puts components together from a bunch of vendors and it mostly works but when it doesn't, there's nobody on staff who can fix it.
I've been a Dell customer for decades at this rate and I know for a fact it's true.
I've had support tickets open for all kinda of weird firmware, hardware, etc. bugs and they've been well resolved, even if it meant Dell just replaced the part with something comparable (NIC swap).
>Dell doesn't know how to fix things like that because they don't design and engineer the systems they sell.
Of course they do. That's like saying Oxide doesn't know how to fix stuff because they don't design the CPU, NVMe, DIMMs, etc. Oxide is still going to vendors for these things.
Ironically, it was Dell's total inability to resolve a pathological rash of uncorrectable memory errors very much is part of the origin story of Oxide: this issue was very important to my employer (who was a galactic Dell customer) and as the issue endured and Dell escalated internally, it became increasingly clear that there was in fact no one at Dell who could help us -- Dell did not understand how their own systems work.
At Oxide, we have been deliberate at every step, designing from first principles whenever possible. (We -- unlike essentially everyone else -- did not simply iterate from a reference design.)
To make this concrete with respect to the CPU in particular, we have done our own lowest-level platform enablement software[0] -- we have no BIOS. No one -- not the hyperscalers, not the ODMs and certainly not Dell -- has done this, and even AMD didn't think we could pull it off. Why did we do it this way? Because all along our lodestar was that problem that Dell was useless to us on -- that we wanted to understand these systems from first principles, because we have felt that that is essential to deliver the product that we ourselves wanted to by.
There are plenty of valid criticisms of Oxide -- but that we don't understand our system simply isn't one of them.
As a side question, what's the name of your custom firmware that is the replacement of the AGESA bootloader? I tried searching on the oxide github page but couldn't find anything that seemed to fit that description.
(The AGESA bootloader -- or ABL -- is in the AMD PSP.) In terms of our replacement for AGESA: the PSP boots to our first instruction, which is the pico host bootloader, phbl[0]. phbl then loads the actual operating system[1], which performs platform enablement as part of booting. (This is pretty involved, but to give you a flavor, see, e.g. initialization of the DXIO engine.[2])
Thanks, are the important oxide branches of illumos-gate repo (and any other cloned repos) defined anywhere? I definitely wouldn't have found that branch without you mentioning it here.
Interesting enough I also ran into something somewhat related with Dell that they were not able to resolve so they ended up working in a replacement from another vendor.
Nonetheless, it is quite interesting what you've built, but as the end user I'm not quote convinced that it matters. Sure you can claim it reduces attack vectors and such but we'll still see Dells and IBMs in the most restricted and highest security postured sites in the world. Think DoD and such. Core/libreboot with RoT will get me through compliance the same.
The software management plane y'all built is the headlining feature IMHO, not so much what happens behind the scenes that the vast majority of the time will not have a fatal catastrophic upstream effect.
>There are plenty of valid criticisms of Oxide -- but that we don't understand our system simply isn't one of them.
That's not what I said. There's a line in the sand that you must cross when it comes to understanding the true nature of the componentry that you're using. At the end of the day, your AMD CPUs may be lying to you, to all of us, but we just don't know it yet.
> Off by a few orders of magnitude. Dell on-site SLA with pre-purchased spares was about 6 hours.
You're talking about replacement parts. Yes Dell is good about that.
The discussion above is asking them to diagnose and fix a problem with the interaction of various hardware components (all of which come from third parties).
But they _are_ writing the firmware that runs most of them and need to understand those devices at a deep level in order to do that, unlike Dell. Dell slaps together hardware and firmware from other vendors with some high level software of their own on top. They don't do the low level firmware and thus don't understand the low level intricacies of their own systems.
No they're not unless I'm mistaken. They're not writing the firmware that runs on the NVMe drives, nor the NICs (they're not even writing the drivers for some of the NICs), etc.
There's a line in the sand that you must cross when it comes to understanding the true nature of the componentry that you're using. At the end of the day, your AMD CPUs may be lying to you, to all of us, but we just don't know it yet.
I'm not speaking hypothetically. If you hit a "zero-day" bug that Dell has never seen it's going to take time. And somehow every large customer finds bugs that Dell certification didn't.
> And somehow every large customer finds bugs that Dell certification didn't.
It's a law of computer engineering.
In the Apollo 11 decent sequence the Rendezvous Radar experienced a hardware bug[0] not uncovered during simulation. They found it later, but until then, the solution was adding a "turn off Rendezvous Radar" checklist item.
[0] The Rendezvous Radar would stop the CPU, shuttle some data into areas it could be read, and woke the CPU back up to process it. The bug caused it to supuriously do this dance just to tell it "no new data", which then caused other systems to overload.
It's ironic coming from a company who's CTO has harped about containers on bare metal for years. Maybe a large swath only need to deploy VMs, but the future will most definitely involve bare metal for many use cases, and oddly Oxide doesn't support that currently.
See the pattern? Dell only care about the big guys.
Set aside the childish tone ...
> Dell is a single vendor that will diagnose and fix all of your hardware issues.
There are two anecdotes here disagreeing with you, and frankly that's enough to say what you said above isn't true, not universally so. I doubt Odixe is targeting big deployment like yours, but more like theirs. Whether they will succeed is another matter, but they do have a valid sales pitch and the expertise to pull it off.
So OpenBMC is fine (happy for them!), but having open firmware is much deeper and broader than that: yes, it's the service processor (in contrast to the BMC which is a closed part on Dell machines) -- but it's also the root-of-trust and (especially) the host CPU itself. We at Oxide have open source software from first instruction out of the AMD PSP; I elaborated more on our approach in my OSFC 2022 talk.[0]
Dell uses trusted platform modules (TPM). It's a separate chipset than the BMC chipset.
For a mostly open source solution, not only would you need open source BMC firmware, you must have an open source UEFI/BIOS/boot firmware like CoreBoot, LinuxBoot, Oreboot, Uboot, etc.
The fact that it's not on linux is one of the great things about it. There is too much linux on critical infrastructure already and the monoculture just keeps on growing.
At least with Oxide there is a glimmer of hope for a better future in this regard.
They sell rack-as-compute.[0] Their minimum order is one rack: You plug in power and network, connect to the built-in management software (API), and start spinning up VMs.
It would be interesting to sell a data center in a container. Cooling, power supply, compute, storage, and network, all in a box. You supply power, a big network pipe, and the piping to external heat exchangers.
IIJ has a project like this, data center in a container, just add power. They build it all up in Japan, ship it to rural areas across the world to basically jumpstart a local data center (I imagine mostly for industrial sites). They had a fun project where they had a half rack, powered by solar and connected to the net via Starlink.
There are whitebox Windows laptops, OpenWRT routers, Arduino boards, ArduPilot drones, etc. It almost sounds strange that there are no prepopulated 12U racks intended for OpenStack(is that still a thing?)
No idea how they do things today, but v0.1 of Azure (before it was called Azure) was a bunch of containers in a field. I remember seeing an aerial photo at the time.
Yeah including networking and storage together with virtualization is what makes hyperconverged infrastructure hyperconverged. Otherwise it's usually just called converged infrastructure.
It's not a new concept, it's a new product. Ideas do not become uninteresting the moment the first person instantiates the idea. Further iterations on a possibly-good idea are also interesting.
Obviously different marketing copy speaks to different people. But that is referring to how, when you buy a rack from us, you don't need to put everything together and cable it all up: you pull it out of the box, plug in networking and power, boot the thing up, and you're good to go. Installation time is hours, not days or weeks, which is the norm.
"no cables no assembly just cloud" is completely misleading to any kind of people - tech or marketing or not.
When people hear cloud, it means that aspects such as electricity costs, electricity stability, Internet, bandwidth, fire protection, safety, etc etc are abstracted away.
Oxide IS on-premise, right? The website is very vague and wishy-washy.
It is on premises. You interact with the rack the same way you interact with the public cloud: as a pool of resources. The specifics are abstracted away. “Private cloud” is pretty well established terminology in this space, and that’s what we’re doing.
At this stage of the company, everyone gets a white-glove installation process. I suspect that will change over time but I don't work on that part of things, so I don't personally know the details.
Sorry to be slightly obtuse, which details are you referring to here? Help upon installation? At the moment, we are helping customers individually, yeah. But we do have a documented process we are following https://docs.oxide.computer/guides/system/rack-installation-... (and more on other pages there)
Ah yeah, so the "facilities" section of https://oxide.computer/product/specifications has some of these things, probably the closest we have to publicly publishing that in a general sense.
Yes I understand, but will your included service actually verify that everything is set up correctly, meets advertised parameters, and sign off on it? (Such that the customer can start using it immediately afterwards.)
Or does the customer need to take on some risk and hazard associated with installation, configuration, initial boot up, etc.?
e.g. If someone buys with the intention of using it up to X FLOPS, and the machine only delivers Y FLOPS once it's all said and done, what happens?
It’s not the area of the company I personally work on, so I don’t know those details, to be honest. We certainly make sure that everything is working properly.
I mean, we absolutely sell support. I just don't know anything about the details personally. You shouldn't take my lack of knowledge as a "no," just a "steve doesn't personally know."
To be super clear about it, this is referring to not needing to cable up all of the individual sleds to the rack upon installation. It doesn't mean that we recommend connecting a rack of compute to your data center via wifi.
This is pretty big, as someone who has deployed servers to datacenters before. Remote hands are very good at plugging in the network uplink and the PDUs. Doing a complete leaf-spine 25GbE network with full redundancy is something they are pretty much guaranteed to screw up at some point.
I wouldn't be dismissive of people telling you that the product description can be improved. My opinion is that the description of the product in this thread will outperform your site 10 to 1.
I'll try to explain, not in the spirit of being argumentative, but with the hope of being useful.
The comment you replied to was not questioning the value of integrated cabling. It was pointing out that the product description on the site does not make sense.
"Cloud computer" sounds like a server you rent from AWS. It's kind of like calling Rust "cloud compiler."
If you choose to use words that your audience doesn't understand, or even worse understands to mean the opposite of what you want them to mean, it's a good idea to explain these words immediately using conventional words with conventional meaning. The comments by throw0101a did that.
The product seems really cool, but there is no way I would've understood what it was from the website.
I understand that's what you're saying, and I understand what the parent is saying. I chose to explain what that alluded to, in case anyone in this conversation is also finding it hard to understand what is meant by that specific copy. That doesn't mean I don't understand the broader point, or that I think the website copy is perfect.
Perhaps if you don't understand what the copy means, then that is a sign that you are not the target audience, rather than that the copy is bad? From what I've gathered from reading other comments in this thread, that copy will make perfect sense to Oxide's target audience, as it uses words in a way that will be very familiar and make perfect sense to the kind of person who might make a purchasing decision for a system like this.
And for what it's worth, I don't think you need to explain what's happening to Steve, it seems to me that he understands perfectly well. To me you come across as being rather condescending and in my opinion Steve is being commendably polite in response.
"Real" mainframes have RAS (Reliability, Availability, Servicing) features such as hotswapping for all hardware components and automated HA/workload migration across physical racks. They can also do SSI (single system image), i.e. run a single workload across physical nodes/racks as if it was just multiple 'cores' in a single shared-memory computer. Oxide computers will probably end up doing at least some of this (namely workload migration across racks for HA) but saying that it can comprehensively replace mainframe hardware as-is is a bit of a stretch. In terms of existing hardware it's closer to a midrange computer.
The Oxide and Friends podcast had an episode on virtualizing time, specifically for the purpose of live-migrating a container from rack to rack without the VM being aware, and allowing operators to take the rack offline on their schedule. Otherwise, apparently, you end up having
to leave racks running because you cannot evacuate all of the containers currently running on it. (e.g. perhaps your contracts or SLAs are such that you cannot afford even the few seconds of downtime a shut-down-here-and-spin-up-elsewhere would cause)
I believe the episode name was "Virtualizing Time"
The first iMac famously made it easy to connect to the Internet; The 'i' in iMac was for "Internet". Its setup manual was a couple of pages long, mostly pictures and IIRC, just 37 words.
Existing vendors will provide rack integration services and deliver a turn key solution like this. Also vendors of virtualization management software have partnerships with hardware suppliers and be happy to deliver fully integrated solutions if you're buying by the rack. The difference is in those cases you have flexibility in the design which seems to be missing here.
Proxmox and a full rack of Supermicro gear would not be as sophisticated, but end result is pretty much the same, with I imagine far far better bang for buck.
I like it, but it doesn't seem like a big deal or revolutionary in any way.
Those of us who've bought large "turn-key" solutions from Dell etc. have often discovered that it's actually just a cobbled-together bunch of things which may or may not work well together on a good day, depending on what you're trying to do. Just because it's all got the word "Dell" written on it, doesn't mean that the components were all engineered by people who were working together to build a single working system.
Total agreement. Another point: Having the "Dell" name on the front doesn't give you a "throat to choke" as so many people seem to think is important. Unless you're very large scale then, at best, you can threaten them that they don't get your next business. You're certainly not going to get help.
You're no worse-off with Oxide from that perspective. Their open source firmware means that thr opportunity to pay somebody else to support you at least exists.
Even small shops can use bad experience as leverage for credits and discounts, especially if the vendor has account managers. This is one of the (few) benefits of having a human involved in invoicing vs. self-serve.
Same is true of Oxide, it'll be up to actual experience to see how well it works. Oxide seems to have written their own distributed block storage system (https://github.com/oxidecomputer/crucible), have their own firmware, kernel and hypervisor forks, etc -- when any of that breaks, good luck!
The premise is that you don't need luck, you can call Oxide. As you said, they wrote all of it, so they own all the interaction so they can diagnose all of it.
When I call Dell with a problem between my OS filesystem and the bus and the hardware RAID, there's at least three vendors involved there so Dell doesn't actually employ anyone that knows all of it so they can't fix it.
Sure, Oxide now needs to deliver on that support promise but at least they are uniquely positioned to be able to do it.
> That's the same premise as with all "turn-key" solutions. If it didn't come with software support, it wasn't really turn-key.
Just about any company will sell your company a support contract.
The more interesting question is, can they back it up with action when push comes to shove? I suspect most people have plenty of stories of opening support tickets with big name vendors that never get resolved. And through the grapevine you find out that they won't fix it because they can't fix it. They might not even have access to the source code or anyone on staff who has a clue about it because it came from who knows where. Sales is happy to sell you the support contract but it doesn't mean your problems can be fixed. BTDT.
From listening to the Oxide podcasts, my impression is that Oxide actually can technically fix anything in the stack they sell, which would make them vastly different from Dell et.al.
Skill-wise, yes for sure (except perhaps for storage -- I haven't heard them talk about that much). Bandwidth wise, though?
I used to work for a company targeting Fortune 500s. At that level of spend, when a client had a problem, somebody got on a plane. Only a fraction of those problems escalated all the way to R&D, which is where Oxide skills are. That's where VMWare etc are hard to beat.
The premise is that the bandwidth needed will be orders of magnitude less, because the engineering will be orders of magnitude better. The opportunity makes sense as we've long been climbing up the local maximum peak of enterprise sales driven tech behemoths built on a cobbled together mix of open source and proprietary pieces held together with bubblegum.
Can an engineering first approach break into the cloud market? Hard to say as enterprise sales is very powerful, and the numerous "worse is better" forces always loom large in these endeavours. That said, enterprise sales driven companies are fat, slow and complacent. Oxide is lean and driven, and a handful of killer use cases and success stories is probably enough to sustain them and could be the thin end of the wedge on long-term success. We can hope anyway.
> Proxmox and a full rack of Supermicro gear would not be as sophisticated, but end result is pretty much the same, with I imagine far far better bang for buck.
I think the question is how well they can do the management plane. Dealing with the "quirks" of a bunch of grey box supermicro stuff is always painful in one way or another. The drop shipped, pre-cabled cab setups are definitely nice but that's only a part of what Oxide is doing here. No cables and their own integrated switching sounds nice too (stuff from the big vendors like UCS is closer to this ballpark but also probably closer to the cost too).
I suspect cooling and rack density could be better in the Oxide solution too, not having to conform to the standards might afford them some possibilities (although that's just a guess, and even if they do improve there these may not be the bottlenecks for many).
> Existing vendors will provide rack integration services and deliver a turn key solution like this.
My experience with the likes of Dell is that they'll deliver it but they won't support it.
Sure, there's a support contract. And they try. But while they sell a box that says Dell, the innards are a hodgepodge of stuff from other places. So when certain firmware doesn't work with something else, they actually can't help because they don't own it, they're just a reseller.
AWS outposts have been there in the market for a long time .. though I am sure there are differences but to say extisting cloud vendors were blind to on prem requirements is a stretch.
Also future datacenter builds are going to be focusing on specific applications which means specific builds. I think Nvidia has a much better chance here with their superpod than Oxide. The target use case is pretty unclear.
On-prem buyers are doing cost reduction and cost reduction targets things like, as one example, the crazy cost of GPU servers on the CSPs. Your run of the mill stuff is very hard to cost reduce.
You can see their sort of lack of getting it by using Tofino2 as their switch. That’s just a very bad choice that was almost certainly chosen for bad reasons.
You don't build a new greenfield compute pod because you want to, you do it because it makes sense. Making sense is about cost and non-cost needs like data gravity and regulatory issues.
The cost case only works for GPU heavy workloads which this isn’t - wrong chassis, wrong network, etc.
Tofino2 is the wrong choice because even when they made that choice it would have been clear that it’s doa. Intel networking has not been a success center in, well, ever. That’s a selection that could only have been made for nerd reasons and not sensible business goals alignment or risk mitigation.
When you make an integrated solution you’d better be the best or close to the best at everything. This does not seem to be the best at anything. I will grant that it is elegant and largely nicer than the hyper converged story from other vendors but in practical terms this is the 2000s era rack scale VxBlock from Cisco or whatever Dell or HPE package today. Marginally better blade server is not a business.
They also make a big deal and have focused on things no one who actually builds data center pods cares about.
I actually hope they get bought by Dell or HPE or SuperMicro. Those companies could fix what’s wrong here and benefit a lot from the attention to detail and elegance on display.
>They sell servers, but as a finished product. Not as a cobbled together mess of third party stuff where the vendor keeps shrugging if there is an integration >problem. They integrated it.
I’m actually extremely impressed. I want one. I haven’t worked in a data center in years, but I’d be tempted to do it again just to get my hands on one.
I wish they’d sell a tabletop version for hobbyists, but realize this is probably a distraction. But… the problem with a lot of these systems (including the old Sun boxes and things like ibm mainframes and the AS/400) is that they sound cool but there’s no real way for the typical new developer to “get into them” for fun and, as a result, you lose the chance for some developer selling it to their company based on his experience with the things.
Apparently (I don't remember it, although I probably did read the Byte magazine at the time) there was a rumor in the early 1980s that IBM's PC was going to be a shrunken 370, called the 380. [1][2]
I wish IBM would shrink their LinuxONE Rockhopper 4 Rack Mount down to at least an "under the desktop" model. To my knowledge, IBM still makes quality products and has excellent customer service. They have fun names too (Rockhopper and Emperor are types of penguins!) and they even have 3D models of their rack mount cloud computers with shadows. [3] In fact, when I first read about Oxide a year or two ago, I searched for "IBM cloud server", and left it at that. So IBM, could you please send someone from the LinuxONE down to Boca Raton to create our new PC? :) Thanks!
I own a Turing Pi 2 but the hardware it is running on is proprietary. The switch isn't managed. The manament software is very archaic. Yes, it is modular and stackable and probably thousands of times more hobbyist friendly than Oxide but so is edge computing in general.
For example, this form factor looks really nice for a “hobbyist edition” or “evaluation edition”: https://zimacube.zimaboard.com/. I would probably buy an Oxide rack like this as soon as pre-orders were announced.
They won't even tell you how much a rack will cost. Infuriating typically B2B "talk to Sales so we can decide exactly how much we can get out of you and segment the market on the fly" approach persists even here, it seems.
I wouldn’t expect anything else for a full rack in this segment: it’s going to be tens or hundreds of thousands of dollars, and big enough that there will be some inevitable negotiation about prices.
That doesn't mean you can't have a thin spec builder and a pricing page, even if what that mostly gets used for is devs putting together a comparison of that to a cloud deployment or similar and taking that to the procurement department to argue it's worth opening the conversation.
Same here. I really want to work on one of these. I got in the industry at the tail end of the time when people used Sun and DEC gear. I got to use just a little bit of it and it seemed so much more "put together" then PC stuff is even now.
Oxide feels like it'll be that "integrated" experience, but with the added benefit of software freedom.
I would assume so. They've said before you can make modifications to the firmware and deploy it yourself if you so wish. That's one of the major reasons that making the firmware open source is so useful.
While working in telecom data centers circa 2016 I've seen many single rack computers from Dell, IBM, HP, Huawei... Not sure that's a new ideia, ex. the open source bits.
You own this. AWS Outpost is leased and you still also pay for the resource usage on top of the outpost unit itself. And this would not be integrated with your AWS account.
It's mostly not true that you still pay for resource usage on top of the Outpost unit. That's only true, AFAIK, for EBS local snapshots and Route 53 Resolver endpoints. The big boys--EC2 instances, S3 storage, and EBS volumes--are all "free" on Outposts. That is, included in the cost of the unit and not double-charged.
Charging for EBS local snapshots on your own Outpost S3 storage and Route 53 Resolvers on your own compute is a weird one. I don't know how they defend that. To me, it seems indefensible.
> You can purchase Outposts servers capacity for a three-year term and choose between three payment options: All Upfront, Partial Upfront, and No Upfront. … At the end of your Outposts servers term, you can either renew your subscription and keep your Outposts server(s), or return your Outposts server(s). If you do not notify AWS of your selection before the end of your term, your Outposts server(s) will be renewed on a monthly basis, at the rate of the No Upfront payment option corresponding to your Outposts server configuration.
> You can purchase Outposts rack capacity for a 3-year term … either renew your subscription and keep your existing Outposts rack(s), or return your Outposts rack(s)
Route 53 on outposts doesn’t charge for your own compute. The outpost resolver is free. The oupost resolver endpoint must be backed by an in-region resolver endpoint, which doesn’t become free just because you own an outpost. What you’re paying for is the ability to sustain high query volume from instances in your VPC to an on-prem DNS server.
This is a turn-key solution, ready to use without eventually dealing with multiple devices with its own firmware and caveats revealing after where put to work together.
The closest to that is that AWS managed rack that works with the web APIS you know already
> Ironically this looks like the realization of Richard Stallman's dream where users can help each other if something doesn't work.
that's only true if you think that "users" means "people who operate cloud computers", which is about as far from understanding what Stallman is talking about as is possible. Someone who makes SaaS and runs it on an Oxide computer is no less of a rentier capitalist than someone who makes SaaS and runs it on AWS.
I know you are half joking, but it would really be helpful to have ball-park pricing available. Are we talking Sun-level markups here, or how should we think about it? Given the enterprise sales contact form, I'm thinking yes, but I'd love to know for sure.
I am not in sales and so I hesitate to speak on it in case I am incorrect, but the way that I personally think about it is that it is true that it is not an inexpensive product: there's a LOT of computer here. But the goal is to be competitively priced.
The last time (one of them) Oxide hit HN there were some ballpark estimates based on the CPUs in use, switches etc. Someone else said 500K and up.
I wish there was a 4U version (10-25K but I don't think that they could come close to that price point - regardless even that is out of reach for me to ever get to play on one :-/ )
It's meant for orgs running at a certain scale, but you'd be surprised how early that starts making sense. AWS isn't exactly paying the economy of scale savings on to you.
True but at for companies operating at scale they not only operate on AWS but on other cloud providers as well as legacy data centers … but business wise it’s a hard sell , it may sell for couple of cycles to build a new dc or use an existing ones but then it will be back to the cloud again for many more cycles.
I've found that the scale these start making sense is only a handful of racks. We're not talking full DCs, but a room in a building or some colo space.
And beyond that, there's all sorts of weird environments that need a lot of local compute. You'd be shocked how many servers are on a cruise ship for one example among many.
I am CTO of a large global data center provider posting with throwaway account.
As a technologist, I really appreciate what they have done. Impressive work, high quality, however I don't understand who this is for.
The meaningful market for Data Center hardware is pretty well defined in two clusters. People that build/make custom gear (such as Hyperscalers) and people that buys HP/Cisco/IBM/Dell... (blades or hyper-converged). To scale, you obviously want your DCs as standardized as possible.
Until this company has a certain/size and scale, no one serious will trust their black boxes at any type of scale.
Beyond the tech, how would support services really work? We can have a technician from any of the large vendors on-site in less than 2 hours. In some of our DC clusters we actually have vendor support personnel 24x7 on-site with vendor paid spare parts inventory. How would they provide that level of service?
Maybe I am not the target audience for this offering.
I think they're mostly targeting customers who want an AWS- or GCP-like experience from a developer perspective (compute is abstracted and you can provision it with an API, etc.), but want to own their own compute infrastructure and have it on-prem. That market has mostly had to cobble together consumer-inspired HP/Cisco/whatever stuff historically (like, one of the early talks about the Oxide value proposition was complaining about why every server in the rack needs a CD drive, which was the norm from Dell), because the kinds of stripped-down, super-efficient hardware designs the hyperscalers were building weren't available to the general public, so this is that: hyperscaler-like technology for people who want to own it themselves.
I think the motivations for why people would want to own their own are probably a mix of financial (at a certain scale there's a tipping point and it gets cheaper), and regulatory/compliance/whatever, like if it's healthcare data, or defense, etc.
Thank you for the response. The problem you described has been solved by the large vendors with Hyper-converged offerings for many years so it sounds like Oxide might be a bit late to the party.
I do understand well the rational of running your own servers vs hyperscalers, as well as the repatriation trend but I see Oxide at best as a niche player.
> The problem you described has been solved by the large vendors with Hyper-converged offerings for many years
All Oxide has to do to win that market is ship software and firmware that doesn't suck, because there are incumbents but the incumbents are clearly incapable of doing so.
It’s a bit like Apple entering the Bluetooth headphone market. Tons of players, but they all suck for a variety of reasons. Apple announced the AirPods, which pair extremely well with your phone, and they really don’t suck.
Oxide has customers who have been waiting for real integration and innovation from HP, Dell, and Cisco and are ready to take a risk on something new.
I set up some Cisco server hardware a few years ago, and only by the time I'd managed to order it I was already wishing I had a better choice. When it arrived and the remote serial was unusable to fix the BIOS ("American Megatrends copyright 1984" at 9600 baud? No thanks.) I was ready to give up and go back to AWS.
This is a market ready for a kick in the ass, which Oxide plans to do.
Genuine question, you mention Hyper-converged, can you point to anything that comes even close to the experience you presumably get from the Oxide offering.
Part of the problem here is that the people who make the purchasing decisions (like this CTO guy) don't care about "the experience" because they're not the ones unpacking boxes and plugging in cables.
They pay other people to do that and they don't really care if it's a miserable time. And if it takes days instead of hours, who cares? Rarely is someone setting up a data center under the gun (unless you're Elmo and we all saw how that went).
Factors like scalability and ongoing support are much more top of mind.
Not saying that Oxide can't address this, and I love Oxide's focus on the experience, but I think this bottom-up approach to convincing customers is going to be a steep climb..
But they seem to be up for steep climbs, so I wish them all the best!
Yeah I think this is the interesting question. My (naive, uninformed) interpretation is that Oxide is betting that there is some area in the overlap of the venn diagram between "people who have reasons not to run their businesses on the public clouds" and "people who are not suits but technologists and care about the user experience of setting up and maintaining their infrastructure because they want to do it themselves / without hiring lots of people and service contracts".
I'm not sure how big that space is, or whether it is likely to grow or shrink over time, but I'm intrigued by the proposition they're testing here!
I wonder if we'll see this space you describe grow as AI startups and scaleups - or even internal departments in large enterprises - start to increase in number...
I know there's already plenty of concerns about giving cloud-based AI services access to corporate data, so maybe there's a growing market there..
I mean... it went quite well all considering? There was some site instability for a brief period of time and now it's back to working normally. The initial hypothesis was that the data center was vital and so couldn't be shut down quickly. Turns out the hypothesis was incorrect. So I'm not quire sure that makes the point you're trying to make.
Of course Twitter is definitely not all use cases, so trying to generalize from one data point isn't a great idea in general.
I was making a bit of a joke there, sorry. It was more about the rather rare scenario of setting up a data centre in a hurry than about the long-term quality of the result.
Elmo's experiment came to mind so I tossed it in there.
These seem to solve the ‘host your own cloud’ problem, but are still standard server blades requiring a ton of surrounding hardware and maintenance. Oxide is entirely integrated.
"A ton of surrounding hardware" is vague, generally speaking it's bog standard network switches that are needed.
Oxide is entirely integrated with their own custom (though based on OSS?) networking via Tofino based switches and software. Could be good, but a lot of subtle bugs can occur at this layer. It's a risk.
VxRail's the market leader? Last I touched it, it was a mountain of integration faults and vendors pointing fingers at each other. Maybe it's easier in larger or more traditional environments, but I'd be apprehensive of hyperconvered if VxRail's the best available
Although it might be true that the existing vendors don't provide a good experience, I'm still a little miffed that the OP does not even mention the existence of the existing vendors.
Lack of local support does make them a niche player, but everything starts from a niche and those who believe they don't start from a niche disperse their efforts. So, with the hypothesis that Oxyde is smart, the question therefore is: what niche is Oxyde focusing on ?
I mean, I don't know anything about this space like you do, I'm just an intrigued observer, but my impression is just: They are going to try to compete with those large vendors. Maybe they won't be able to! Such is life, and indeed is the most common outcome for new ventures.
But precious few new ventures have ever entered a market where nobody said some form of "I don't get it, there are already established products in this space, how will this new thing ever get traction?". And yet, lots of new ventures turn out to be competitive with the established players in their market, for one reason and another.
>I think the motivations for why people would want to own their own are probably a mix of financial (at a certain scale there's a tipping point and it gets cheaper), and regulatory/compliance/whatever, like if it's healthcare data, or defense, etc.
Yea, there's definitely a market in defense here. Because even though Azure/AWS offer Govcloud, its inadequate for non-civilian connected infrastructure. This offers benefits of writing "modern software" and deploying it in similar modern fashion while keeping it completely running isolated. Imagine being able to make your command and control operations actually decentralized and not vulnerable to a missile strike on a single datacenter.
Essentially this same sentiment, applies to any number of things:
- Why would anyone buy the Framework laptop, they don't have nearly the support/pedigree that Dell, HP, etc. has?
- Why would anyone use iPhones in the enterprise/IT world, they don't have nearly the support/pedigree that Blackberry, Microsoft, etc. has?
- Why would anyone use Google Fiber, they don't have nearly the network or support that AT&T, Spectrum, etc. has?
- Why would anyone ever use Linux (in enterprise, let's say), compared to the support and adoption that Microsoft/Windows offers?
- ...
I'm purposely picking different examples with varying degrees of success or adoption. I am not claiming that Oxide will be an instant category-dominating success. I don't think Oxide expects to replace HP/Cisco/Dell/etc. overnight, and I don't think a business has to launch with that ambition from the start, to prove that it's worth launching.
But this take is so repetitive as to be bordering on cliche -- I don't know if you're self-aware enough to realize, you are literally just a living embodiment of the "Innovator's Dilemma" right now...
Apple was an incredibly well established company and the initial iPhone was not used in enterprise… it didn’t really start to take off there until iPhone 4 (2010-2011).
Not to mention the comparison is inane to begin with. Using an iPhone for your enterprise and moving your tech infra to a relatively unknown company are not equivalent at all.
This just seems like a twist on hyper-converged infrastructure + open source.
I mean, I love their design. I just don’t think it’s special enough to warrant mass adoption. And I certainly wouldn’t be deploying at scale in an enterprise environment with zero widespread adoption.
Small entities will just use cloud same as always. Large companies have a multitude of unique needs that won’t all be catered for by a single box. The big vendors will clean up on that front as usual.
"Niche" is not a dirty word. Framework is niche, but if they can make their cost structure work (which I have no idea whether or not they can), that's ok. New products don't need to dominate their market in order to be succesful.
(But I do agree that Google Fiber is a bad example / a good example of a new product in an established market not working.)
Bryan Cantrill is famously against vendor lock-in. He wrote a[n in]famous blog about the "FYO point" while at Sun. Oxide may be going for customers that also have the same aversion to vendor lock-in.
One thing that Bryan understands is that you can "lock" the customer in with great products and services, as well as continuing development, while also making the customer feel secure in having a way out should you turn into a company that treats locked-in customers as cash cows. The open source strategy (it is a strategy for Bryan and Oxide) is there precisely to do this: make the customer feel they can leave you, but then not.
For your deeply technical staff, having source code access is a big deal too, since it enables them to better understand the products they use.
How big is the market of sufficiently-vendor-lock-in-averse customers? I don't know -- that's not my remit. But there's the size of that market right now, and whether Oxide (and any other companies with similar visions) can grow that market by sheer willpower. I make no predictions.
What if Oxide can get the next Netflix to use their stuff instead of a public cloud?
Oxide is the definition of vendor lock in. All of their hardware is unique... even down to the choice of fans. Fan burns out? Now you've got to buy another one... from them.
One of the amazing shifts in the last 20 years was realizing that commodity hardware, when deployed correctly, could do the job.
Not sure if we're talking metaphorical fans or literal ones but assuming the latter: replacement is covered under warranty. And for me personally, the "amazing shift" that you describe was in fact a decade-long experiment that left me with an inescapable conclusion: commodity hardware cannot, in fact, do the job -- and not for lack of trying!
Metaphorical fans, but also fans, and everything else. It was an example from the blog post. I looked at that super innovative and cool backplane. Sure, warranty covers it for the first N years, but then what? What happens when things turn into a Tesla situation and you get horror stories of delays and poor results?
I think 'commodity' is generalized at this point. Given the choice of Supermicro and Oxide. I'm going to pick Supermicro. They can deliver me a clean rack of machines too. Why would I go with Supermicro? Big company, lots of products, lots of choices, I can work with them to get what I want.
Oxide is too singular. It is one rack, one design, one set of specifications. Don't get me wrong, there is some value in that... and I'm sure you'll sell plenty of hardware, but I'm not finding the value in it for myself. That's the part that I mean by 'commodity'.
I'm not ignoring the software stack, I just don't see any value in the additional vendor lock in on it. I'd rather use open source stuff developed by a large community of people and not a single small vendor tied to a very specific and limited hardware stack.
I don't think they're planning on forcing you to buy their products :)
As far as I can tell, there exists no "open source stuff developed by a large community of people" to run integrated server rack hardware. But if you're using such a thing, then I'd like to know what it is!
If their SW and FW source code is MPL 2.0, that's good enough to limit the extent of vendor lock-in. Sure, it would take time to take over maintenance of that code and then add support for different HW and so on, but there can be a cottage industry of consultancies that can help if ever Oxide vendor lock-in or bankruptcy becomes a problem.
No it isn't good enough. This is a hardware play because you could theoretically take that software and run it on whatever hardware you want. You're not going with this business because of their open source software though, you're going with it because they are making innovative hardware.
If you're buying millions of this stuff, what says that you're going to get support for it in 5 years. Who knows... maybe Cisco wakes up and gives them an offer they can't refuse and then shuts down the company.
By the way, people endlessly gripe about Google deprecating things and that's just software...
Their hardware isn't really innovative though... Even the hardware integration isn't really innovative, as others here have pointed out.
My two cents is that this isn't really an innovation business model, it's an execution business model. Their proposition seems to be that they can execute a a server rack with integrated hardware and open-source software so well that customers will love their product.
Maybe they'll just end up with tons of nerds our there thinking, "I really wish I could justify investing in Oxide racks, but it just doesn't make sense for my business / I just can't sell it to my CIO", but hey, maybe not!
It's better to just think to yourself: "this doesn't seem like a useful product to me, I don't understand why people find it interesting, oh well", and move on.
No, I'm taking all of your words and distilling it down to a common saying.
I actually do think what they are doing is innovative across the board. They are taking all of the common feedback anyone who has run large scale data centers (which I have) and applying it to a brand new product. Unfortunately, they are doing it in a way that is extremely vendor specific.
`import { Api } from "@oxide/api"`
No, thanks.
`Understand and debug issues faster`
How is this any different than throwing Netdata onto a server?
> How big is the market of sufficiently-vendor-lock-in-averse customers?
Very. Just look at the USA defense spending budget. If you’ve ever worked on AWS-govcloud or secret, you know there’s a market here.
This has huge use for military too. Imagine having a black site or off-grid location but still needing a rack of things. What if you could spin up an entire enterprise infrastructure by just loading up this rack?
If this team manages to get this thing government certified, there’s a lot of profit to be had.
You are strongly misinterpreting both how real-world customers perceive vendor lock in and how DoD procurement works. Everything here is so far off from reality I don't even know where to begin.
You may not be aware of the pain that many large, non-software companies currently have on AWS. Gigantic monthly bills (hundreds of thousands per month) coming from subdivisions that aren't capable or motivated to reduce their AWS budget or usage. To the office of the CTO, Oxide's value proposition (buy instead of rent) could be very motivating.
"Hey subdivision A, could we buy a few Oxide racks and move your workload there from AWS? It looks like they would have all the storage and compute you need. Yes? Ok, in 36 months we'll pay your current IT department employees a bonus of 50% of whatever it has saved us vs your current AWS budget."
Yeah, my example communication was a bit contrived.
But the point still stands. There's a lot of AWS spend happening (even after being optimized) that is frustrating when you look at the raw numbers and consider how much server capability you could outright buy for the same amount. And Oxide would make it so much easier to run a bunch of VMs (and infrastructure-as-code) than standard racked x86 servers.
Oxide appears to be a complete shoo-in for companies that used to run a bunch of VMs on racked Dell/HP servers, migrated their VMs/storage to AWS, hate their monthly AWS bills, and still have the old server rooms available.
Having worked somewhere with an AWS spend of $30k/mo on _virtually nothing_, I can attest to this. I think most of it was sales demos that never got cleaned up.
Most people who can do this (aren't as entrenched in AWS) end up moving to a cheap VPS provider so that they don't end up having to pay for all the internetworking, facilities, throw redundancy out the window, and then still have to pay the IT burden to heavy-lift all their workloads to this whole new "Oxide" system.
I guess there must be a largish market for this since AWS introduced Outpost to provide the "cloud" to onprem industries. I feel like this is competing with that market.
Since many of those use cases probably already run extensive on-prem infrastructure this could appeal to them. AWS outpost talks about industries like healthcare, telecom, media and entertainment, manufacturing, or highly regulated spaces like financial services. I've heard of media companies that process through things like IMAX cameras that have just tons of TB's of data sometimes just for 5 minutes worth of footage. That would simply be too cost prohibitive - in bandwidth alone - to try and move around in the cloud and you don't want to have to wait for things like AWS snowball or whatever.
While I think the space is "niche" those niche spaces are not small. Big companies with big budgets.
I think they are a competitor to the 'HP/Cisco/IBM/Dell... (blades or hyper-converged)' part of this. They basically saying 'we will do it better'.
Their marketing and story is supposed to convince you that you could save money running their things rather then Dell. And instead of paying for VMWare you get Open Source Software for most of it.
> Until this company has a certain/size and scale, no one serious will trust their black boxes at any type of scale.
I guess that a risk they are willing to take. Some costumers might wait for a few years until they see Oxide being big enough.
Other costumers might be sick of HP/Dell and might take a Risk on a smaller company.
Since they seem to have some costumers, some organizations are willing to take the risk to get away from Dell and friends.
So I think you are the target audience but you are not willing to risk it until they are larger and less likely to fail and they have a good story in regards to support. I assume they have a support story of some kind, no idea what it is 'Contact Sales' ....
In terms of 'trusting they will continue exists' all they can do is survive for a few years until they are pretty established, then more people will be willing buy their product. And hopefully in that time their existing costumers rave about how amazing the product is.
Lets hope they don't go bust because all potential costumers are just waiting. Then again, you can't anybody for not buying from a startup.
> Their marketing and story is supposed to convince you that you could save money running their things rather then Dell. And instead of paying for VMWare you get Open Source Software for most of it.
As someone who has dealt with mostly Debian and Ubuntu in recent years, every time I had to deal with even small numbers of RHEL licenses I often asked myself "Why do put up with this?" (I know why, but still… such overhead.)
I think that ship has sailed in the 90s. From the 80s to the 90s IBM dropped like 50% of the people that worked there and lots of companies were moving away from mainframes in droves. I'm sure some people got fired for sticking to mainframes to long and wasting money. And likely some companies went bust in the late 90s for having a IT infrastructure based on IBM Mainframes.
That's mostly speculation but it seems like it has to be true.
The current state of the art is a fucking train wreck. Choose any layer of abstraction and start picking at it and you’ll find mostly gaffer tape. Scaling up sucks. Rewriting old monoliths to micro services wasn’t a panacea except maybe for cloud vendor profits. You said the hardware market is well defined, which is another way of saying ossified. Particularly when you start comparing it to the expected pace of software. How’s Open19 going? How much liquid cooling do you have in customer racks?
In my opinion this is ultimately a software offering that happens to come on vertically integrated hardware. It offers a complete, highly polished API to a minuscule-scale
DC. If they can find market fit and make a little bit of money, the next step is to start making deep improvements to things behind the abstraction. From where you sit, you are well aware that you could make huge improvements inside the DC demarc if you could do it without disrupting customers. But you’re probably limited by the terrible, terrible APIs you would be forced to use, that don’t offer the capabilities you need, from vendors who would be happy to chat but would provide timeframes in the 6-18
month range for even a modest improvement.
So this, IMO, is about defining that interface to a DC on a single hardware platform with vertical integration, and then scaling up from there. The current hyperscalers will adopt the suite of APIs and capabilities directly, make lower quality copies, or die.
Of course all assuming they’re successful enough to get the revenue machine going in the first place which seems likely to me, given the absolute dogshit state of the cloud world today, the trend towards multi cloud, and the business case for moving certain loads back on prem or to a bare metal colo++ offering.
> you obviously want your DCs as standardized as possible. Until this company has a certain/size and scale, no one serious will trust their black boxes at any type of scale.
This gets to the crux of my first thoughts when I read the marketing copy: can they deliver (reliability).
They do admit very clearly that what they're doing is hard and that at many points during development they were reluctant to be too ambitious (for obvious reasons), but at each stage they did just that: proceeded with the most ambitious option. That takes a huge amount of self belief that might be warranted or might be hubris. As you point out, untimely the success of Oxide won't only hinge on ability to deliver on that self belief, but more on their ability to convince prospective customers that they have competency to do what noone else can.
I fully support every part of their approach in theory but my wizened traveled self thinks it smells a little too good to be true.
Maybe you are not. This offering definitely sounds like something for on prem and not a large data center. Basically, if your core competence is hosting stuff you don't need the extra value they provide. But if your core competence is basically anything else and just need more than a single server under the IT guys' desk then this begins to look very exciting.
You might be right but if a customer won't have the size/scale, it won't value the unique proposition from Oxide. I hope I am wrong because it would be great to see a new player with a fresh perspective in the hardware market.
Bryan Cantril claims the cost of running these could be worth it for many small companies, and if you compare it to EC2 costs for running let's say a CI I find that easy to believe. Maybe it won't be cheaper than Hetzner and co. but that's not what they are competing against on a product level.
If you could drop ship a rack of gear to the Colo before, with the puny compute and bandwidth potential in that number of Rack Units, didn’t it just become massively more appealing?
To be completely honest, this is for the idealists out there. Those of us who are itching to replace our vSphere with oVirt because 1) we have the time and skill to do it, 2) we believe in open source and 3) we believe we can make huge savings by using open source.
I expect the oxide supporters to have a hard few years ahead of them of finding bugs in high throughput environments. But at the end of the day it will be worth it just to have another competitor in a pretty boring playing field.
They literally had CTOs of F100 companies that want to buy this gear as part of VCs pitch. Because as you can imagine your question was the first question VC's asked.
Purchasing a ton of hardware from a startup seems extremely risky for a F100. It's one thing to be left holding the bag when a SaaS startup goes under, but when you just spent millions on gear that is now completely unsupported... eek.
I'd be curious to see what companies are interested, Oxide doesn't have any logos on their website which is a little odd given the space.
The fact that they're willing to absorb that risk is a strong signal they're solving a real problem. It's been a while since I've worked somewhere with on-prem hardware, but I remember long build-outs, unhelpful vendors: A RAID card firmware bug bricked our SAN. Our extremely expensive support contract gave us front-row seats to finger-pointing between the card manufacturer and Dell but ultimately no solution was provided to us. Our IT director, who was absolutely furious, basically had to twisted Dells arm to get them to send us replacement hardware. Whole thing was a giant fiasco.
This is the secret none of those existing vendors (Dell, Lenovo, HP, etc) are willing to tell you: They have very limited technical expertise on what they sell you and outside of some specialized troubleshooting they can do, they'll defer to their vendors. The understanding is that you've got the intellectual horsepower on staff to cope with their various shortcomings.
> Purchasing a ton of hardware from a startup seems extremely risky for a F100.
If you're working for a megacorp nothing tends to happen quickly, so there will be a slow roll-out over a multi-year period as old hardware gets phased out and new hardware is brought in for a refresh.
If there's a hiccup at any point they'll simply keep the previously purchased stuff running and start a new roll-out with another vendor next fiscal.
> I'd be curious to see what companies are interested, Oxide doesn't have any logos on their website which is a little odd given the space.
1. They're just starting out. 2. Some of their customers want to be (or start-out initially) discreet:
> Oxide customers include the Idaho National Laboratory as well as a global financial services organization. Additional installments at Fortune 1000 enterprises will be completed in the coming months.
In terms of physical hardware repairs or replacement, sure, that could be a risk when the supplier is an early-stage startup. Though I wonder if the open nature would also make it easier to create eg. third-party sleds
I doubt any F100 would go all-in on new vendor like Oxide for anything at first. I bet they could spend millions on a a few racks as a trial and have some groups work with it. A couple years down the line maybe they start expanding usage if they like it.
Of course that framing itself is bad - F100 companies aren't usually quite that monolithic. By the time they get that big there's a heterogeneous set of processes, equipment, systems, etc. Some parts of the company may use Oxide right away because they see it as a solution, and others may keep using the IBM mainframes, and other still will keep using racks/blade servers from Dell for eternity.
F100 waste so many millions already on projects that never see the light of day, why couldn't they throw money at Oxide and see if it works better than their AWS contract?
I bet the trick is to keep the perfect balance of high profile wins and and losses. You want to be defensible as an expert to the non-technical folks while obviously a fall guy to the technical ones I guess.
I think these guys aren’t that, though, they seem to be selling a real, cool product.
Yes, an industry where FUD and being SOTA is coveted, like cybersecurity can really get enterprise interest with the right connections. Few contracts of sorts to trade for logos on a slide deck and high profile references can get you further. Does your product actually do anything y is useful? Maybe, not yet, maybe never, but you were convincing enough and hey 1/10 startups fail.
Maybe something like Dell VXBlock didn't exist when they pitched their idea?
Any hardware contracts are very long term and you'll have a hard time getting me to switch to a different vendor, especially when they also want to come in with an unknown operating system which I have to run.
When my company looked into it VXBlock looked less like fully integrated hardware and software and more like a smattering of pre-selected components wired into a box ready to go with VMWare with a support contract. If you're already in deep with VMWare they're probably great. But the software side made my head spin. It looks like Oxide is a better fit for orgs that are more IaC.
I guess time will tell VXBlock just looks like amalgamation of SKUs Dell has. Oxide was built as "clean slate" from firmware to every minute detail to offer a compelling product for companies that want to have a hyperscaler style systems for their on-prem workloads.
Is it now bad to have a few offerings that you can tailor to your needs?
With compute one size doesn't fit all, maybe you need more disk space or maybe you need GPUs... I'm sure Oxide will come out with different spec modules over time.
The idea is similar: It's a rack which runs virtualized workloads and you don't have to think much about individual machines.
> Beyond the tech, how would support services really work? We can have a technician from any of the large vendors on-site in less than 2 hours. In some of our DC clusters we actually have vendor support personnel 24x7 on-site with vendor paid spare parts inventory. How would they provide that level of service?
Forgive me for being naive, but that sounds like a great way to differentiate their offering.
E.g. the famous Maytag Repairman, who sits at his desk doing nothing because Maytag washers are so reliable that he has nothing to do.
> In a time in which the laundry appliances of major manufacturers had reached maturity, differing mostly in minor details, the campaign was designed to remind consumers of the perceived added value in Maytag products derived from the brand's reputation for dependability.
I can't think of a lot of examples right now but I can already imagine one type of customers for such a product - universities wanting high performance computing. At my alma mater, the HPC cluster/server was in a different slightly distant location. Using something like AWS wouldn't be liked by almost any uni admin, and running a server on premises isn't a great idea in a place that gets the occasional (but rare) power or internet cut. Outsourcing some of these responsibilities may have been nice for our admin.
Every single one of the companies you listed started off with unverified hardware, some that failed in the field and continues to today. Why wouldn't you try something else, considering the status quo? This is a nothing-to-lose situation as long as you don't put all your chips into the bet. Every single datacenter has capacity and a need to diversify. It's not even a heroic feat of risk taking.
Having worked in a variety of related fields, my take on this is that hypervisors like VMware have largely eliminated the need for proprietary hardware with “support” contracts. Just buy 3x the capacity in white boxes and turn off anything that breaks.
The problem is that the industry hasn’t noticed this yet. The hyperscalers have, and are printing money as a consequence.
I might not be some brilliant CTO, but this product obviously seems to fit _very well_ between two very large product offerings: fully cloud being one and fully on-prem being the other. I’ve often thought it would be cool if you could _buy_ a physical space in a server farm, like literally an area of square feet where the racks are all serviced and provided, but meets all of the needs a company who doesn’t want fully cloud has. This is actually way more clever than that.
A lot of people didn’t understand the need for the iPad mini or Mac mini or myriad other products without an obviously existing market. I think this is extremely keen of Oxide to fine the borders between these offerings and fit very snugly between them.
My guess, someone who buys hyper-converged infrastructure today (e.g. Nutanix, VCE, etc) ... but that market is getting smaller and smaller by the day.
It’s an acquihire play. It might be possible to beat the hypers with $100 billion and some really good engineers. But building custom racks ain’t the angle to do it.
I really wonder what their mental image of a product market fit is. They're undeniably doing some cool stuff but myself and it seems like everybody else has to do some serious mental gymnastics to figure out who they sell this to and what needs it fulfills or niches it can fit into.
I have been following oxide for a bit, and really don't have to add to the tech conversation, but do want to say:
Congrats to the team on reaching this big milestone and (in my eyes at least) just as much congratulations on doing it in a way that has been unique and sticking to values that seems to drive a strong positive culture (at least from the outside looking in).
Shipping products is hard, and only getting harder. IMHO, one of the big drivers of that is just how complex every market has became. Building and selling software alone is so much more multi-disciplinary than it was 10 years ago and adding hardware to the mix is upping that by a huge factor. As I look around, I see so many companies struggle to build teams that can handle the huge range of required tasks. To see a company like Oxide that (once again, from the outside a least) seems to have things together on so many fronts, especially while doing it while sticking strongly to some core values, is pretty inspiring.
Not to get overly cynical, but I don't think it is an extreme opinion to say that current start-up culture feels like you have to make big compromises in what you believe in order to be successful. Whether that be open-source, how you value and pay employees, or even just rushing things to deliver that aren't ready.
While I acknowledge Oxide has some well-connected, experienced founders that I am certain enabled them to get the resources and trust to do things their way, I really hope they kill it so that other founders and builders can learn that you still can build not just financially successful products, but great organizations that truly care about their values.
Thank you very much for the kind words! It has been important for us to do things the right way and to be a model for others -- so it's really meaningful for us to hear that that's appreciated; thank you!
Yeah, I don't honestly know much about the hardware side of things, nor the business/economic rationale for why one would prefer an Oxide rack over a more conventional setup, but reading through their posts, what they've done is just singularly so impressive that I really, really, really want them to succeed.
I've worked in startups most of my life, but it's tough feeling like you've got to take a bunch of shortcuts ("MVP"-it before you run out of runway, etc. etc.), so to see a startup just put a ton of real engineering prowess into a great product is a sight to behold.
Also, while there are obviously amazing products out there in the world, it's so often a "race to the bottom" that it feels like decently made products are just prohibitively expensive. I'm really into antique espresso machines, and there is a reason that seeing rebuilds of beautiful old lever espresso machines get so many oohs and ahhs online - they truly don't make them like they used to, so it's so cool to see these beasts brought back to life. For some reason I have a similar feeling looking at these Oxide racks - they just didn't cut corners to build an awesome machine. Again, not qualified to say whether that makes economic sense long term, but it's still a thing of beauty regardless.
I've been following Oxide since they formed and I really hope they crush it.
Semi-sort-of related... there was an automation company, Bedrock automation, that went defunct about a year ago. Their PLC hardware ideas were dope but I always felt like they were missing the boat a little bit by supporting stale PLC programming languages. I used to wonder if supporting Rust and Ada on these PLC's would be been a good idea to diversify/specialize into complex control system domains. Also, iirc, Bedrock didn't support EtherCAT which I felt was a mistake.
Anyhow... one of these days it would be cool for a forward thinking company like Oxide to tackle what a modern complex, distributed control system hardware/software stack should look like using Rust/Hubris.
The industrial automation industry is highly risk adverse. In some systems, like safety, this mentality is justified without proper migration plans. In other areas, it's alarming, like choosing to run a system on hardware that has been out of support for 30+ years without adequate spare parts.
Operation/Shop floor technologies (OT) are treated like mechanical equipment, "When it fails, we'll swap it out for a new one." Well, this isn't a motor, it has programming, and it interfaces with I/O sensors and devices.
The main challenge is the lack of knowledge and skills in modern technologies among technical staff and decision makers in industrial organizations.
A an aside, in my early days I hopped on a 5-hour flight and drove 6 hours to replace a failed hard drive in a Windows NT machine used as an HMI. Then a year later, replaced it and all the other clients with a vSphere stack. The local resources, both internal and external, were too intimidated to touch it.
I'd be in favor of a "reset" in automation, it feels like fighting city hall.
What is the real solution for highly complex technology like IT, as end times are coming?
It's a critical issue. Resources are wasted, and soon we won't be able to waste as much as we did, as the abundance era is coming to an end. The desire and need for higher efficiency through highly complex technology is only to increase, as the population gets older and older.
I can easily see mandatory IT training (like the current mandatory army training in some countries) and states standardizing certain technologies to avoid fragmentation and thus freezing progress, for good reasons. Like forbidding any database to be used in production except 3 most complete ones or banning all web frameworks except React and Svelte (arbitrary). It's not that they won't want progress, but they won't be able to afford progress and the resulting skill fragmentation in the IT workers while people are hungry and money is ever tighter.
I agree with everything you said... but there are a lot of areas in control system engineering that would benefit from having really robust hardware designs coupled with guaranteed correctness that Rust/Ada provide, and then all of the upside of all of the advanced control programming that is relatively easy in a modern programming language.
Beckhoff & B&R Automation are part of the way there. They both have some really slick portfolios. IMO, on paper, these are the two most advanced control system companies (with large sales volume).
I feel like a lot of people miss the point on PLC stuff.
“Stale PLC programming languages” might be stale and in need of a rework, but “rewrite in rust” entirely missed the value proposition. Memory safety is important, sure, but these systems aren’t usually ever allocating memory or taking actions that aren’t deterministic. PLCs usually operate in strict realtime environments. Like each operation contains a known number of clock cycles, and is timed to match physical “stuff” moving in the real world. To rewrite in rust, you’d need a flavor of rust where you can deterministically know the timing of ops and calibrate the timing of each branch of any control paths.
That said, there is a market for softer realtime PLCs and companies like Rockwell sell Windows IoT hardware- but they still contain a strict-realtime companion controller. Arduino is now selling a mountable PLC that can be programmed using their tools, but even they support traditional PLC programming languages.
Yep... I programmed PLC's for periods of my career. I could write about this until my fingers fall off, but in short:
Rust and C code can also operate in strict real-time embedded environments provided some basic rules are followed like "no dynamic memory allocation". This done all the time.
If one follows hard real-time coding standards like Misra C or "JPL real-time C" (and these standards can also be applied to Rust, Zig, Ada, etc.) the code will run deterministically on a given target... just like PLC code running is not necessarily any more deterministic than C code (no dynamic allocation) running on an embedded processor. In fact, many PLC's today run a "PLC code execution" engine on top of a real-time operating system like QNX (which I think is mostly C code). Even in some older PLC's, it is still C firmware that's interpreting and running the PLC code.
PLC programming "languages" (ladder logic, FBD, CFC) were designed for their programming audience, who are not software engineers. It is difficult to represent complex logic and numerical code in these languages and that limits the sophistication of algorithms that can be implemented on these systems. For instance: try writing a model predictive control routine in ladder logic; I think it could be done but I'd lose my remaining hair doing it.
PLC Structured Text is very similar to Pascal and pretty capable, but folks generally don't write model predictive control algorithms in Structured Text either.
It is hard to orchestrate multiple PLC to run as a cohesive, deterministic unit. The interfaces between PLC often need to be kept simple and sometime this communication is less real-time. Things are a lot more advanced in cluster computing.
The limitations of current PLC architectures is already a pain point for complex control system, like large robotic manufacturing lines or optimal control of HVAC in large buildings)... and it's going to get worse as performance demands increase. Again, as mentioned above, I think Beckhoff and B&R are further along the evolutionary path than others in the industry.
I really wish oxide had a Homelab/consumer centric offering!
Spec wise, some low power systems like an Intel NUC, LattePanda Sigma, or Zimaboard. You could fit 3/4 of them in a single 1u with a shared power supply. They could even offer a full 1u with desktop grade chips on the same sleds.
I have thought about building one myself, but it's a large investment of time that I can't seem to find lately.
It would be great if Oxide had something like Canonical's "Orange Box"/cloud-in-a-box for homelabs, evaluation, training (in the management bits) - and hobby work loads!
I'd imagine they'll get to that eventually, these types of companies generally start at enterprise level because that's the most profitable and requires closing smaller numbers of deals. Once the product is proven and their support infrastructure is in place they can go for other market segments to try and maximize revenue
It's not just about maximizing revenue, it's also about getting it into developer hands early (homelabs, side projects, college students, etc) so they can become familiar with it, and become an advocate for it within their company. Cloudflare is a good example of this.
Even just a medium business offering would be great. I'd love to not have to use Dell or HP gear-- anything to get away from the cobbled-together stack of legacy IBM PC compatibility and third-party ODM/OEM stuff glue-and-taped together by the vendor.
On prem. Reliable and inexpensive network connectivity they has any resemblance to a 10G LAN doesn't exist where I am.
I work with some businesses who need very, very reliable, high-bandwidth, and low latency connectivity to their data. The amortized cost of on-prem beats the cost of any off-prem offering as soon as the cost of the necessary connectivity is factoted-in.
AWS Outposts is the solution. I like Oxide but people seem to be blind to the actual competition when they focus on Dell as the competitor. AWS has been shipping Outposts racks for years. All prices are public on their website and you can order it today. Nearly every configuration is sub-$500k. Fully managed and AWS supports the entire stack; no buck-passing among vendors, same as Oxide.
I’m not sure where his customers are, but Outpost up/downlinks are supposed to to be at least 1gbit, and they don’t behave well in situations where the latency to the paired region is high. EBS lazy loading blocks is great in region but awful when your ping is 300ms.
I'm talking shops who spend $200-$500K on servers and storage, not north of $1M (which is where this Oxide gear lives). Something like a 1/4 scale Oxide rack, perhaps.
I work at SoftIron, another startup in this space. Our HyperCloud product might be interesting for you. I'm not in sales, so I can't comment on the prices, but I'd guess we're much more competitive since you don't actually need to buy an entire rack of our gear at a time.
That said, where this product-space gets tough is actually scaling it down. It's pretty challenging to create something that is remotely stable/functional in a homelab (space/power/money) budget. Three servers and a switch would probably be the bare minimum. We (and I'm sure Oxide :) scale up like a dream.
This all has me wondering, if I just want to play with stuff in this space as an individual homelabber who earns a tech salary and wants a nicely designed rack-mounted alternative to a mess of unorganized NUCs and cables and whatnot, what are my best options?
If you're willing to spend money on rack-mounted gear you definitely have options, and what you get sort of depends on what you're interested in playing with.
A lot of homelabbers (and even some small businesses) go for Proxmox as a virtualization distribution. I don't use it myself, but IIUC it's effectively a Debian distro packaged to run KVM/LXC, with support for things like ZFS, Ceph, etc. It has some form of HA, an API used by standard open source devops tools, handles live migration, etc.
So buy some used rack-servers on Ebay (or new, if you're ballin'). A lot of businesses sell their old stuff, so you can pick up a generation or two out of date for a good price. If you want to do fancy stuff like K8s, Ceph, etc you'll probably want at least three nodes, ideally more, and a bunch of disks in them. Networking gear is a sort of pick your poison thing. A lot of people love Ubiquiti gear; a lot of people hate it. TP-Link is another that's good and budget friendly. StarTech sells smallish racks (including on Amazon), if you want to start there.
It won't look exactly like SoftIron's HyperCloud or Oxide's Cloud Computer, but you can certainly get pretty sophisticated.
Not sure if this answers your question, but other great spaces to explore are the 2.5 Admins and Self-Hosted podcasts.
I'm really thinking mostly about the hardware part here, and maybe just enough layers of the stack to feel like an integrated hardware setup. Let the nerds play with whatever software they want above that.
To go ahead and dream a bit:
I'd hope for an online configurator like the one SoftIron's HyperCloud has [1] but instead of "talk to a sales rep", show a price for what you just configured, like you're configuring a macbook.
Relatedly, there should be a standard rack form factor in the size category of NUCs and Mac Minis, rather than having to go all the way to the 19 inch monster racks that medium to large businesses use. If it were nailed down to the point of being able to blind mate (just learned that term from Oxide's article here!) gear into it, that would be kind of perfect.
Unfortunately they are not planning home lab things anytime soon, per a recent podcast episode [0].
If you want to play around with their Hubris OS: "You wanna buy an STM32H753 eval board. You can download Hubris, and then you’ve got – you’ve got an Oxide computer. You have it for 20 bucks.”
Not 1U but perhaps a box design that isn’t noisy like a pizza box server.
Don’t know if oxide would want or be able to compete in the low cost market but a bigger a more expensive desktop/workstation as a mini homelab cloud could be a great option to get people trained on the oxide platform.
"Everyone at Oxide makes $201,227 USD, regardless of location."[1]
Do they actually assemble and build their own racks of hardware, or is that outsourced? Somewhere, there must be an assembly plant. If this stuff actually exists. It's hard to even find pictures of their products. Do they have production installations?
Certainly odd but not out of the ordinary for a small Bay Area start-up where the Founders have a ton of cash and the focus is mostly on what they personally want to do. Posts like these are b/c the Founders want to hire 1-3 people who fit exactly into alignment with them.
If you don’t listen to their “On the Metal” podcast, you’re missing out. So many great stories from legends in the industry. Just start with the first episode.
And they currently run a podcast weekly called "Oxide and Friends" where they talk about miscellaneous things in the software space.
The most recent few episodes have been about corporate open source and they've had excellent guests, like Kelsey Hightower. Definitely the best computer related podcasts out there. Bryan and Adam are great hosts and their humor is always a delight.
Other podcasts I'd recommend: ADSP [1] (if you're into programming), 2.5 admins [2] (if you're into computers). But I have no recommendations about hardware design because AFAIK the podcasts you mention and what Oxide is doing are pretty unique.
Jim Salter is an insufferable has been. He's a SJW-type that threatened to delete his reddit account and closed off the ZFS subreddit but came crying back like a childish addict after the reddit admins threatened to take his fiefdom away.
There’s plenty of questions about ZFS in the show that get answered, that’s always a given.
I’d say their opinion on “ransomware being a backup problem not a security problem” was enlightening and their backup opinions are also interesting (Push vs. Pull, ZFSes snapshotting, etc.)
Allan’s experience in FreeBSD is welcome too, not every day you get to hear about FreeBSD in prod.
They both skew FOSS too but not in an idealistic way, it’s based on decades of experience.
The Amp Hour is really the only popular hardware/electronics design podcast as far as I can tell, and it's pretty decent, especially when they have interesting guests.
In a similar vein, Embedded.fm is a great podcast for embedded SW. Though I bet Oxide's take on embedded rust is a lot different than the hosts of that show!
What am I missing here? Hasn't SoftIron been building this exact thing for around 5 years? Heck, they design and manufacture all of the hardware as well.
Correct. I work at SoftIron. We have several HyperCloud systems in production right now. SI has been shipping purpose-built storage systems for years as the root-comment suggests, but HyperCloud (which is closer to Oxide's product) has been in production systems in defense, banking, internationally for well over a year now.
Your webpage is an ocean of marketing fluff and buzzwords. It's not clear that you have a product that competes with Oxide. It's not even clear how you're different than Dell or HP.
See https://news.ycombinator.com/item?id=38026306 and other related threads. Oxide's product is a rack you plug in and you're done. You don't have to rack a bunch of servers yourself and cable them.
Thanks for pointing this out! I wasn't aware of SoftIron. I think its a big deal that there are two vendors on this path. And both seem to be doing it the right way. I think it makes the whole, stronger than the sum.
I'd like to add that their open source code is also EXTREMELY high quality. If you're an embedded developer go take a look at Hubris and Humility. I ended up using those to GREAT effect for this custom one-off aerospace device and it was a fantastic experience to integrate with. Definitely a change from what I was used to that took a bit of getting used to.
Yeah, I just skimmed through the reference docs for Hubris and was very impressed with what I saw. No "rocket science", just (apparently) solid technical decisions that are extremely well justified and documented.
Some interesting rare choices there (some of which would be applicable to normal userlands or even to language VMs):
* Memory access is protected, but addresses are not virtually mapped. The lack of paging of course ties strongly to the inability to dynamically add tasks.
* They wish they could have read-only shared libraries (viable since there's only one address space) so they could truly be shared, but the current ecosystem assumes mutability is possible (even though in Rust mutable globals are rare).
Their ring logger was really enlightening to me about the value of Rust enums. Heterogeneous log events are dropped into the ring with some holding only the fact that something happened, and others holding additional data about what happened. Then Humility is able to print out the contents of the ring either online or in crash dumps. This is how you get logging in nostdlib Rust without ending up without half of a badly implemented printf. Instead, Humility, which has the full stdlib available, formats the enums for the firmware.
Tangent. In my admittedly limited experience, embedded code seems to have a tendency to be some of the worst code you can come across. The stuff is already low level and not the most easy to follow, but then embedded developers seem to despise names with more than two letters and a number. Without the datasheet at hand it is impossible to figure out what any of the code does because everything is just an abbreviation or acronym from some block or pin-out diagram.
I've been interested in Cogmind for a while, unfortunately it only works on Windows (which seems an odd choice for a text-based game) so I have no ability to play it.
I design them in Monodraw, pass it through a janky converter I wrote that converts text into a json grid of characters. I then render a number of layers that get combined, which is a mix of the static art layer, and others generated from functions that spit out a similar cell based frame.
With respect to Hubris, the build badge was, in turns out, pointing to a stale workflow. (That is, the build was succeeding, but the build badge was busted.) This comment has been immortalized in the fix.[0]
With respect to Humility, I am going to resist the temptation of pointing out why one of those directories has a different nomenclature with respect to its delimiter -- and just leave it at this: if you really want to find some filthy code in Humility, you can do much, much better than that!
I think 'throw-DO-178C' was being less literal than you are assuming here. Note that they quoted the comment about extremely high quality. What gives Hubris/Humility that level of quality?
When I see the concept of "extremely high quality" applied to software in the aerospace domain, I tend to think of methods like formal proofs and "semi-formal" methodologies like DO-178C (including MC/DC, not as a tool, but as a critical sub-process). It does appear that Humility adheres to some traditional ideas used in even "safety conscious", real-time embedded work:
> However, Hubris may be more interesting for what it doesn't have. There are no operations for creating or destroying tasks at runtime, no dynamic resource allocation, no driver code running in privileged mode, and no C code in the system. This removes, by construction, a lot of the attack surface normally present in similar systems. [1]
Oxide surely didn't invent all these concepts, we were applying some of this in the early '90s on human-safety-critical embedded projects, and they were most likely used before that. The attack surface eliminated is more than just security, e.g. even self-induced "attacks" from task priority inversion. Maybe the concept of applying them to rack mount servers is novel, but I have no experience there. Techniques like DO-178C go farther and do things like trace requirements to object code and strict coverage criteria.
When you are considering where Hubris/Humility might be applied in aerospace, also note that it would most likely be denied at the proposal stage of talking to a regulatory body like the FAA if you said, "The methodology for our brake controller is cloning a Github repo written by 100X S/W engineers in the Rust language, then modding that to fit." It's a process you have to follow in a documented fashion, from the start. Who knows--I certainly don't know everything--if you do have a identify reproducible process that improves S/W quality, the FAA might be interested.
And as far as eliminating debuggers goes, Boeing did actually do a joint academic/industry project on a zero-bug reduced (subset) Ada compiler (Zbra), which itself would be qualified as a tool instead of the more laborious process of mapping requirements to object code directly. [2]
In the spirit of humor shown on this topic, I would suggest considering a reduced subset of Rust called Iron. :) Regards, and best of luck!
1. The state or quality of being humble; freedom from pride and
arrogance; lowliness of mind; a modest estimate of one's own worth; a
sense of one's own unworthiness through imperfection and sinfulness;
self-abasement; humbleness.
-- Webster's 1918 Dictionary
Our avionics software was originally spec-ed to not NEED a debugger it was to be of such high quality, but the designer added break point op-codes in the VM anyway. I guess he was aware of the concept of humility, too. ;) Like humor, humility seems to be in short supply these days.
I know price will vary wildly based on how many you’re buying, but does anyone have the roughest ballpark for how much it would cost to buy one (1), or like two?
They mentioned it really quickly in their Oxide an Friends podcast but, IIRC, prices start at $500k. Some of the audience asked if they were going to do a smaller configuration like half or quarter rack. And they said they were looking into it but weren't sure the of the business case.
> They mentioned it really quickly in their Oxide an Friends podcast but, IIRC, prices start at $500k. Some of the audience asked if they were going to do a smaller configuration like half or quarter rack. And they said they were looking into it but weren't sure the of the business case.
That strikes me as being in the right ballpark, but it's going to be tough to swallow since that's the lowest level of granularity.
For most orgs you'd be left paying for a lot of excess capacity you couldn't immediately put to use as you migrate workloads in. I guess in ~4 years once you reach steady state and you're retiring / replacing these things it all works out, but if you're migrating from vmware or something else in a traditional blade/chassis world it's not like you can just wave a magic wand and move $500k worth of compute over to this thing at once.
If you're green fielding something, that's a lot of cash to sink in on compute you may not need for some time in the future. Never mind your DR site(s) also needing that much...
the performance may be about the same, and CapEx as well (or maybe a little higher), but OpEx could be where you make it up in large(r)-scale operations.
And space efficiency is also not to be sneezed at: for some operations DCs/compute can be place anywhere because latency is that big of a deal, but in other places you need to be close to certain things (trading), and real estate can get expensive.
It depends on how many servers you put in a rack. It's been years now I did this kind of work but I would say that an average rack with 20-25U in computing, 5-10 in storage and 5 in networking will cost you 300k$ easily. I'm pretty sure that Oxide will be more an Apple-esque experience, also on the price side, so a "normal" rack giving the same performances will be cheaper but if you want Oxide you are looking for other features beside the pure HW.
It's very expensive hardware. I think they are trying to bring TCO down on the operation side with better control plane. I work in hyperconverged systems and it's a bunch of tradeoffs. Nothing I've configured has approached $500k so their control plane and OS has to make a really good show of why this 2x expensive cabinet is better than rack and stack Dell.
Let's just say I hope I am interpreting something incorrectly, because if 500k is the minimum price and you match it to the minimum configuration found here:
That said compared to competitors it's in the right ballpark, but I have no idea how companies manage to spend so much money for this stuff. I am the founder of my own tech startup and I remember when I was looking at storage solutions and building out computing clusters there were companies charging absolutely insane prices.
I literally just spent about a week of my own time studying and learning as much as I could about it on my own and ended up building out my own custom solution for about 20-25% of the price these other companies were charging. I remember hearing people trying to scare me out of it saying if I did my own solution I'd need to hire full time operations people, and I'd always have to worry about things breaking and maintenance or headaches and nightmares etc...
It's been over 10 years now and absolutely no headaches, no nightmares, and very very minimal maintenance is needed.
Yeah sure, my process to learn was literally I went down to a local computer store, I bought a cheap desktop computer and 6 cheap hard drives, an HBA, a RAID controller, a bunch of cables, went back to the office and installed Ubuntu Server onto it and practiced several "skills", like how to setup automated backups using rsync, how to physically install these components, how to install mdadm for software RAID etc... comparing software RAID to using a RAID controller. I setup several drills for each task involving various types of failures and how to manage them to the point that it was part of muscle memory.
Some drills I practiced were setting up RAID 10 and on hardware failure having an email sent to me, so I went through the process of getting RAID 10 working using 4 of the drives, and then I would physically pull a hard drive out of the system as it was running to simulate a failure, and then I swapped in a new hard drive and ensured that the RAID rebuild process took place.
Once I was confident enough, I went to thinkmate.com and bought two JBOD expansion chassis each of which supported 78 drives and filled each of them up with 5 TB Seagate drives to get a total of 390 TB. The JBODs are managed by a 1U server running Ubuntu Server and software RAID using mdadm. I also bought 8 compute servers that were considered high density and could fit in 4U. I got a Cisco router for Internet and I networked everything within the data center together using a used Infiniband switch that I bought off of ebay. I also got a KVM so I could remotely access all of the systems. If there's one thing that I would change today, it would be to use ZFS.
I remember comparing the cost of doing this DIY setup to some other premade solutions like EMC and the cost was astronomical, like 3-4x what I ended up having to pay. I also even remember watching Linus Tech Tips and Level1Techs on Youtube and they both had good content about how they managed their storage that was fairly reasonable but still slightly on the pricey side, nevertheless I learned quite a bit from it.
At any rate, the bottom line is that it's not trivial to learn all this stuff by any means and I remember having some serious frustrations due to just how bad and demoralizing some error messages can be, but it's not thaaaat difficult either and in my situation my company is self funded, no venture capital or outside investors of any kind so every dollar my company spends is a dollar out of my own pocket. You better believe when I'm starting my business I'm not going to just blow 100s of thousands of dollars extra unless I absolutely have to.
About 5 years ago I managed to use my own storage system to ditch Dropbox in favor of Nextcloud, which is just leaps and bounds superior. I remember I got so frustrated with Dropbox because I wanted to just do something as simple as create a sub-directory and grant only read only permissions to some accounts. I also remember wanting to do some simple things like create a fake account with very limited permissions that could be used in our conference room for presentations or demos but the only way to do that with Dropbox would be to create an actual account and have to pay the full price for it.
Nextcloud works amazingly well, has all kinds of cool plugins, and gives our organization a lot of flexibility that Dropbox doesn't and I can make as many accounts as I feel like for whatever reason I feel like.
That's a lot more involved. We're a quant firm that have servers colocated globally at various exchanges, so our data is distributed across a lot of systems. The main and original purpose of this storage system is to provide one centralized repository of all of this global data so that we can run backtests and perform analytics locally as opposed to having to hit our Australian system when we need ASX data, and then hit our German system when we need Xetra data, etc etc...
So different backups happen in varying directions, with the colocated servers using the main storage system to back themselves up, and the main storage system in turn backing portions of itself onto the colocated servers.
Not everything gets backed up, however, and most of what does get backed up can be compressed quite significantly. We mostly just back things up that can't be recovered from third parties or that are proprietary to our organization.
I mean, the raw hardware surely costs around 100k, and no way that it costs more than ten million, so you're always going to be right with that "ballpark" qualification
It bugs me more than it should that they flirt with hexadecimal numbers in their branding with the whole 0x thing, but the logo is 0xide when clearly it ought to be 0x1de. Designer came up with a clever idea but didn't understand hexadecimal, and no one had the heart to tell them? :o)
I'm kidding, mostly. The engineer in me is bothered by it, but the part of my brain that cares more about design and branding understands that 0x1de would cause inconsistencies in the branding elsewhere. (E.g. 0xDocs: https://docs.oxide.computer)
You may be happy (or unhappy) to know that was definitely considered – it felt a little more trouble than it was worth, and dare I say a little too on the nose?
"we needed a name — and once we hit on Oxide, we knew it was us: oxides form much of the earth’s crust, giving a connotation of foundation; silicon, the element that is the foundation of all of computing, is found in nature in its oxide; and (yes!) iron oxide is also known as Rust, a programming language we see playing a substantial role for us. Were there any doubt, that Oxide can also be pseudo-written in hexadecimal — as 0x1de — pretty much sealed the deal!"
I know that many might have by now forgotten how an actual data center works.
"One size fits all" will never work.
Here are the main headaches that someone who actually runs a data center will run into:
- How many power circuits does the DC provide you? What voltage? AC or DC?
- How many amps do the circuits have? Is the PDU provided by the DC, or do you provide it?
- How many upstreams will you have? Dual core routers? BGP? Static routing? Failover?
- Upstream provided via Fiber? Copper?
- Given that for the last couple of years being able to re-buy a typical server model (say: Supermicro), CPU, Storage for more than a couple of weeks has been a challenge, and given that rack hardware is supposed to run for years - how can I be sure to be able to scale?
- What happens if that Oxide startup goes bust 6 months from now?
- Would any sane DC operator really buy "we just built our own switch"? I wouldn't. Those who have been in that business for 20 years still f*ck it up on a regular basis. I want a proven vendor for my mission critical stuff.
I am sorry, I don't see them addressing ANY of the real life pain points you have operating your own rack. Unless they ship a couple of data center engineers within the box.
The whole cloud stuff is simply designed for managers who believe that in a job market where it's hard to find DC/server engineers that things will be magically done in the background. With this, it's worse - it's pretended that "stuff just works" if you ship it in a single box. And that's simply not a case. A DC installation not constantly maintained by skilled engineer will go down in flames within months, one way or another.
Part of the power distribution question is answered, in a way that gives even more concerns, though. A single DC distribution without redundant PSUs? What if one of the servers trips a fuse?
By the way, the list of questions would go on. At least over here, DCs offer different rack footprints. They don't even list their footprint.
But most importantly: The TFM doesn't say "There is an experienced network engineer in the box that you will own and that will do the maintenance". Unless this is provided, this will blow up.
Again: I can imagine that they find a hand full of managers buying into this. Up until the date the first few of their racks have burned down a DC, or got hacked and all customer data stolen.
Come on, this is hacker news. Everybody here should know that nobody so far has found a way (not even via AI) to replace the competences many of us here have. Again, I understand that management would love to get rid of those expensive and annoying nerds, but the ACTUAL REALITY is: An unmaintened rack in a DC will go down in flames, and management people will be in tears.
So, let's just be realistic: Oxide is simply a terrible idea that will blow up. But I congratulate them for VCs falling for this.
Now, let's find a VC that is funding someone doing an IPMI system that is not shit, does not have access to all data of the server, and does not get hacked within 5 minutes after deploying it. That would address an ACTUAL need.
Sometimes it makes sense to ask those who you hope your future customers could be.
Yep, exactly. That's precisely the reason why this "solution" to a problem that doesn't exist is currently being praised as the soon to be heir of the throne of computing
I met a couple of their folks on site last week in Raleigh at the All Things Open conference. Saw a couple quick demos, and... it's not for me. But I can see the benefits, and many other folks stopping by the booth seemed to get the benefits as well. The folks at the booth were really nice too. Granted, that's sort of your job when being in a vendor booth at a conference, but it's surprising how often that's not the case (booths staffed by people who don't know the product, or are simply indifferent to the company, etc).
EDIT: "it's not for me" - I'm not working with organizations that have that sort of need directly. Re-reading that phrase, it came across as a bit dismissive of what they've built (which is undoubtedly impressive).
I'll second that, they should make it clear what they're selling and what they have to offer. Every company should have this in their main/landing page... If I can't understand what you're selling/offering in 30sec, then I'm not interested :-)
If you take some time and read around their site you'll see that they're offering a ready to run (turnkey) server. They have everything packed together, they've integrated everything that is needed (CPU, disk, networking...) into a nice looking cabinet, with not too many wires and they're selling it as a complete package.
If you're in need of a server (cloud computer) and you don't want to but separate components and unpack them and connect them yourself, then this looks like a possible solution.
Yes, but... what is a "Cloud Computer"? Is it a "computer in the cloud", like e.g. an AWS EC2 instance? Or is it a fancy name for a good old fashioned server rack (on which you will probably want to run your own cloud, because everybody's doing that nowadays, hence the name), like in this case? And if it's a server rack, how come you don't need any cables? And what do you do with this "cloud computer"? Do they host it in their data center or deliver it to you? So there is some potential for confusion - nothing that can't be mostly cleared up by reading a bit further, but still...
> Or is it a fancy name for a good old fashioned server rack
In a sense. Yes, it is a rack of servers. You're buying a computer. But we've designed the rack of servers as a full rack of servers, rather than an individual 1U. Comes with software to manage the rack like you would a cloud; you don't think of an oxide rack as individual compute sleds, you think of it as a pool of capacity.
> And if it's a server rack, how come you don't need any cables?
Because you are buying an entire rack. The sleds are blind mated. You plug in power, you plug in networking, you're good to go. You're not cabling up a bunch of individual servers when you're installing.
> Do they host it in their data center or deliver it to you?
Customers get them delivered to their data center.
> like e.g. an AWS EC2 instance? Or is it a fancy name for a good old fashioned server rack
I mean, the former is just the latter with some of the setup done for you no? Anyways, it’s a full server rack, with tightly vertically integrated hardware and software. Not sure if you’ve poked around the rest of their site, but it seems like their whole software stack is designed with some really nice usability and integration in mind: there’s a little half-snippet there suggesting that provisioning bare-metal VM’s out of the underlying hardware could as trivial as provisioning an EC2 with Terraform, and if that’s the case, that’s _massive_.
> And if it's a server rack, how come you don't need any cables?
Because they’ve gone to great lengths and care to design it to not need anything extraneous IIUC. I think the compute sleds all automatically mount into some automatic backplane that presumably gives
you power, cooling and networking, and then, as above, you presumably configure all that via software, as you would your AWS setup. Not an expert here though, happy to be corrected by anyone who actually knows better.
> Do they host it in their data center or deliver it to you?
Presumably the latter, given they’re a hardware company, but if their software is even a 10th as good as it seems, I fully believe there’ll be a massive market for renting bare-metal capacity from them.
> there’s a little half-snippet there suggesting that provisioning bare-metal VM’s out of the underlying hardware could as trivial as provisioning an EC2 with Terraform,
That's what some of us are saying, it's not crystal clear what they sell.
You use words like: it seems, I think, presumably, I believe...
This is what we're arguing. A company that has raised $44 million Series A for sure can afford to clearly write what they offer.
I understand, you can't have all the people happy and no matter what you do there will always be "weirdos" that don't like your page/design/wording, but hey at least recognize it :-)
It appears to be an all in one, preconfigured/partially-or-completely assembled rack server for on-prem hosting, complete with pre-configured software.
Just my perception given the front page. I agree it's a bit vague.
Every "cloud" computer is on-prem for someone, and "private cloud" has been a term for at least a decade to refer to "API-based provisioning of resources like with a cloud but in your own data centre/office" that may not be "technically" a cloud but carries the meaning clearly for anyone in the business.
What they're selling is building blocks for private/hybrid/public clusters that may or may not fit your definition of cloud depending on where they happen to be located but where what the term signifies is that it is built to be a building block of a cloud setup that includes features above and beyond a "regular" server to provide what most people tend to associate with a cloud. That is, you're getting APIs to spawn virtualized compute and storage, rather than having to install a hypervisor and management APIs etc. and combine the resources into a cohesive whole yourself.
My guess is that most of them will end up being sold to either hosting providers to provide cloud services to their customers or companies large enough to operate their own data centres where the IT department will use them to offer cloud services to other parts of the business, where the term really is not misleading anyone.
It's too big even for most "private clouds" in smaller companies, because they tend to be too small to be able to order by the rack even when there are APIs etc. offered to developers to provision compute and storage (e.g. years ago I used to operate a hybrid cloud setup with a ~1000 or so VM instances across several countries, and while we had several racks worth of gear we physically owned in aggregate, we didn't have any full racks in any individual location).
Absolutely. The point of cloud computing is that I don't have to care where something is running and that I (theoretically) have infinite elasticity and can scale in and out as fast and much as I need to.
This server is for cloud operators - i.e. for organizations that are building their own cloud infrastructure. That's why Oxide is drawing comparisons to hyperscalers.
"Private cloud" has been a term for at least a decade to refer to situations where people are building their own cloud infrastructure to provide a "cloud feature set" even though the servers are self hosted.
You may not like it, but it's been a long time since "cloud" has exclusively referred to "someone in another organisation entirely runs the computers" as opposed to virtualised allocation of resources.
"Private cloud" as a term still kinda makes me itch, but I don't recall encountering an alternative that was more obvious and it's clearly the usual term of art at this point.
I suspect that while I do appreciate how some posters upthread find the website a tad on the vague side, the target customers-in-potentia will understand it fine.
There are many “you”s in most enterprises. If your platform engineering team builds their own cloud, and offers an experience similar to other cloud providers (or even better and more targeted), this could be a clear win.
I understand that’s your definition and I’m saying that there are many companies where the cloud experience (of not worrying about physical infrastructure but having flexibility and elasticity) is offered to product teams by an internal platform team. That gives you an articulation point where you could migrate from AWS to Oxide racks, and yes, lose some functionality and some guarantees, but also gain more control and potentially make huge savings.
Actually, the definition of cloud is "a visible mass of particles of condensed vapor (such as water or ice) suspended in the atmosphere of a planet (such as the earth) or moon".
Nonsense. The operative bit of "cloud" is "provision and de-provision instantly via an API, without much concern for what's going on inderneath", not "lives in someone else's datacenter".
But that is literally not possible with hardware you purchase yourself.
Sure, you can buy X amount of hardware, and provision up to X amount of virtual hardware via an API, but then what? You can't provision any more until you go and buy more hardware. This is why "cloud, but local" is a contradiction IMO. You can only be "cloud-like" if you're under-provisioning. The moment you want to actually use all of the capacity you already paid for, you're not a cloud any more, because you've provisioned all of it.
> But that is literally not possible with hardware you purchase yourself.
Sure it is. I think TFA is talking about a company selling you exactly that capability.
> You can't provision any more until you go and buy more hardware.
But this is also true of AWS etc. When their estate gets full, they need to go buy more hardware. Regardless of who owns the tin, someone's doing a capacity plan and buying hardware to meet demand.
The point of 'cloud' is that you move that function out of the domain who are actually using resources to solve business problems, which is where it traditionally sat. Historically, if you wanted to run a service, you had to go buy some hardware and hire someone to manage it for you.
A cloud-like model means that the application engineers no longer care about servers, disks and switches. Instead, they just use some APIs to request some resources and then deploy a workload onto them. The details of what hardware, where and how is fuzzy and abstracted. Or cloud-like.
> You can only be "cloud-like" if you're under-provisioning
Everyone under-provisions. Nobody runs at 100% utilisation.
It’s called cloud because it’s not in your own data center. Usually cloud symbols were used in network diagrams to depict systems/networks outside of our concern.
Also you could don’t really get elasticity with a system like this. If anything, that would be the operative bit for me.
With apologies to everyone who has a "The cloud is someone else's computer" T-shirt, things have changed, and the language as evolved as it is wont to do.
I've spent the last decade building on-premises systems very like what Oxide is doing, but I've had to build them out of stacks of servers, switches, storage appliances and VMWare licenses. And the network cabling, and fan noise, and the number of power cables, and.. oh man, I can't wait to install one of these things myself. Having a single point of responsibility for the whole thing shouldn't be underestimated either - I've spent far too long trying to resolve problems with vendors on both sides blaming each other.
It's worth mentioning too that building something equivalent to this would be across more than one rack, and easily cost in excess of $1M.
That was the case ~15+ years ago. Private cloud has been a term for the majority of that time to evoke the elasticity and virtualisation without the "not in your own data center" bit, because to most users of a cloud the operative bit is that they don't have to worry about where the computer is or talk to someone to provision one, not whether it sits in the corporate data centre or off at Amazon.
Hybrid clouds even means devs might not know whether it sits in the corporate data centre or a public cloud, because it could be either/or depending on current capacity.
> Also you could don’t really get elasticity with a system like this. If anything, that would be the operative bit for me.
"You" as in "the organisation as a whole" don't get elasticity. "You" as in "your department" or "you as an individual" do get elasticity.
> "You" as in "the organisation as a whole" don't get elasticity. "You" as in "your department" or "you as an individual" do get elasticity.
Right, but to the degree that you get elasticity, it starts to look more and more like "someone else's computer", no? If multiple people/departments/etc are provisioning virtual instances on one shared cloud infra, with nobody who's using the provisioning API caring about the underlying capacity (and capacity is planned indirectly by forecasting, etc), then it really starts to sound like "someone else's computer" to me. That "someone else" just happens to be another org within the same company.
> It’s called cloud because it’s not in your own data center. Usually cloud symbols were used in network diagrams to depict systems/networks outside of our concern.
That might be true historically, because the only way you could get resources provisioned on-demand via an API from someone else who'd built it. You had to run in someone else's datacenter to get the capability which you actually wanted.
Times have changed. Now, businesses think about "Cloud compute" as being synonymous with "on-demand", "elastic" etc. Where the actual silicon lives is merely an after-thought.
> Also you could don’t really get elasticity with a system like this. If anything, that would be the operative bit for me.
> Where the actual silicon lives is merely an after-thought.
If you have to buy the silicon and plan capacity for it (as in the case with Oxide for example), then it cannot be an afterthought. Which is exactly why I would not consider it cloud computing.
The point is the application team doesn't have to do any of that.
Someone's always got to buy the tin and manage that. Some people are big enough that they might get a benefit from doing that themselves, rather than having Jeff Bezos do it for them.
From the application team's perspective, call API then container go whirr.
> you could don’t really get elasticity with a system like this
Of course you do, right up until the point where you’ve used all available capacity. Just like with public clouds (ask anyone using meaningful amounts of {G,T}PUs). Elasticity doesn’t imply infinitely elastic, that would be ridiculous.
> Absolutely. The point of cloud computing is that I don't have to care where something is running and that I (theoretically) have infinite elasticity and can scale in and out as fast and much as I need to.
Define "I"?
I-as-developer can call an VMware/OpenStack API with an on-prem/private cloud and get a new instance just as easily as calling an AWS API. I-as-developer does not have to worry about elasticity if the IT hardware folks have the capacity.
As an engineer, an Oxide system works like any other cloud provider. You’re just interacting with its API and tooling like you would with Google Cloud or AWS.
To someone on the IT/Operations side, obviously there are differences but theres SIGNIFICANTLY less labor required to build-out and operate an Oxide system vs a rack full of servers. The biggest difference for these people is that there’s actual hardware vs a Cloud Provider, but also costs are fixed so there’s likely no monthly or quarterly meetings with finance arguing over the cloud bill, tying people up to try and shave a few thousand off the bill every month.
In finance/accounting, Oxide is probably the most different: now compute is CapEx rather than OpEx. Depending on your company’s stage that can be a wonderful thing for the bean counters.
> To someone on the IT/Operations side, obviously there are differences but theres SIGNIFICANTLY less labor required to build-out and operate an Oxide system vs a rack full of servers.
But it’s also gonna be much more restricted. So I guess one could see as kind of “Apple for data centers”? Have a nice appliance and be happy as long as it runs as it should (but hope it never stops working as it should).
They've reinvented the IBM Mainframe. Big rack-sized box with lots of redundant hardware that serves guest VMs. This is basically a zSeries in drag.
The key difference is the price structure -- IBM leases the hardware wherever they can get away with it, and uses a license manager to control how many of the machine's resources you can access (based on how much you can bear to pay). This, however, is like a mainframe you own.
Mainframes also do lots of other things that this doesn't really do. Like allowing you to pretend its a bunch of multi-core machines, or even a single one.
The way I interpret it: an integrated stack of compute, storage and networking hardware and monitoring software that you can plug in to your data center (owned or colo) and can e.g. deploy Kubernetes to, and then use as a deployment target for your services.
The main USP: you own it, you're not renting it. You're not beholden to big cloud's pricing strategies and gotchas.
Secondary USP, why you buy this rather than DIY / rack computer vendor: it's a vertically integrated experience, it's preassembled, it's plug and play, compatibility is sorted out, there's no weird closed third party dependencies. Basically, the Apple experience rather than the PC experience.
It's a rack of servers you can buy for a lot of money and put in a datacenter. And once you plug it in and turn it on and do whatever setup is required, you're supposed to be able to spin up vms on it.
The site is really hard to navigate. I eventually looked at the footer and found a link to the technical specifications https://oxide.computer/product/specifications They give an idea of the beast, especially the dimensions and weight
Dimensions H x W x D
2354mm (92.7") x 600mm (23.7”) x 1060mm (41.8")
Weight
Up to ~2,518 lbs (~1,145 kg)
It's a rack.
BTW, they are following some no pictures policy. I found only a few pictures of boards but no picture of the product as a whole.
Indeed, which demonstrates that an explicit link to Home in a visible place is never a bad idea. I didn't click on 0xide in the top left even if that's a common shortcut to the home and I didn't notice the link in the footer, which is where you bury the least interesting stuff. I kept clicking on the text links in the top bar.
The Oxide has everything you need to stand up your own private cloud, and everything works together down to the software. It's a great alternative to buying and managing servers, switches, storage, power management, KVMs, all of the software in between, and the tens of professional services contracts required to glue it all together.
Designing a hardware-software combination to allow for the managing of compute (and networking and storage) to occur at the rack-level rather at the individual-device (server, switch, etc) level, so that large(r)-scale operators can manage cattle more easily (rather than herding cats/pets).
Also this. Looks technically great but how do you sell this to business? Whats the price range? Is this supposed to be an acceptable solution between on-prem and cloud as we start to realise the true costs of being hooked on cloud when your a large org and not a startup?
In a data center you have racks of computers performing all of the workloads. At this point these racks are fairly standardized in terms of sizing and ancillary features. These are built-out to solve the following:
* Physical space - The servers themselves require a certain amount of room and depending on the workloads assigned will need different dimensions. These are specified in rack "units" (U) as the height dimension. The width is fixed and depths can vary but are within a standard limit. A rack might have something like 44U of total vertical space and each server can take anywhere from 1-4U generally. Some equipment may even go up to 6U or 8U (or more).
* Power - All rack equipment will require power so there are generally looms or wiring schemes to run all cabling and outlets for all powered devices in the rack. For the most part this can be run on or in the post rails and remains hidden other than the outlet receptacles and mounted power strips. This might also include added battery and power conditioning systems which will eat into your total vertical U budget. Total rack power consumption is a vital figure.
* Cooling - Most rack equipment will require some minimum amount of airflow or temperature range to operate properly. Servers have fans but there will also be a need for airflow within the rack itself and you might have to solve unexpected issues such as temperature gradients from the floor to the ceiling of the rack. Net heat output from workloads is a vital figure.
* Networking - Since most rack equipment will be networked there are standard ways of cabling and patching in networks built into many racks. This will include things such as bays for switches, some of which may eat into the vertical U budget. These devices typically aggregate all rack traffic into a single higher-throughput network backplane that interconnects multiple racks into the broader network topology.
* Storage - Depending on the workloads involved storage may be a major consideration and can require significant space (vertical Us), power, and cooling. You will also need to take into account the bus interconnects between storage devices and servers. This may also be delegated out into a SAN topology similar to a network where you have dedicated switches to connect to external storage networks.
These are some of the major challenges with rack-mounted computing in a data center among many others. What's not really illustrated here is that since all of this has become so standardized we can now fully integrate these components directly rather than buying them piece-meal and installing them in a rack.
This is what Oxide has to offer. They have built essentially an entire rack that solves the physical space, power, cooling, networking, and storage issues by simply giving you a turn-key box you plant in your data center and hook power and interconnects to. In addition it is a fully integrated solution so they can capture a lot of efficiencies that would be hard or impossible in traditional design.
As someone with a lot of data center experience I am very excited to see this. It is built by people with the correct attitude toward compute, imo.
Weird there’s no mention of GPU at all when you’d think that’s what would prick up the ears of the hosting companies ….. stack a pile of GPUs in a rack, surely you can sell time on that.
Oxide's goal is that they, and by extension their customers, have as much visibility and control over the software stack in these racks as possible, and that includes firmware. They started developing these systems before the current wave of interest in machine learning led by ChatGPU and Stable Diffusion really got underway.
Nvidia GPU drivers are very proprietary, which means that admins and developers have limited visibility into them if they misbehave in any way. This goes against Oxide's philosophy of full visibility into a system that you purchase.
Nvidia's CUDA software has a significant lead ahead of AMD and Intel GPUs, and they're not going to open source it any time soon. But this is a rapidly changing landscape, and AMD and Intel and others are pouring an enormous amount of research into getting their hardware and software to match what Nvidia has going. Nvidia is in pole position, but they're not guaranteed to stay there.
There's still a large market for the CPU workloads that Oxide is offering. For now, Oxide will be concentrating on meeting this traditional compute demand. But you're right to point out that in 2023, the absence of a top tier GPU in these racks is noticeable. I suspect Oxide will want to include some form of GPU or TPU into the next version of their system, but they won't just grab whatever hardware happens to be in fashion. It needs to work with their system as a whole.
I don't think they're trying to be AWS. They're trying to sell a product that greatly simplifies doing on-site cloud for companies. So they sell the physical hardware/software bundle and not time.
> “Oxide is a strong believer in the need for open-source software at the lowest layers of the stack -- including silicon initialization and platform enablement. With the availability of AMD openSIL, AMD is showing that they share this vision. We believe that the ultimate beneficiaries of open-source silicon initialization -- as it has been for open-source revolutions elsewhere in the stack -- will be customers and end-users, and we applaud AMD for taking this important and inspiring step!”
Now to see how their development timeframes and synchronization efforts with big hardware companies go.
This is where they will enter the real “hard” part of hardware, with an exec team from software. Can they respond to the market while making hardware?
They seem to have presented as generally thoughtful about their approach. If they can release major variants about annually or even sub-annually that is what I think will enable them to win.
This thing is the definition of vendor lock in. It's a rack full of non-commodity hardware using non-standard connectors and a bunch of custom software.
The industry moved to commodity x86 for compute for a reason.
I follow a person on Github who works here! (cbiffle[0]) They wrote a fantastic series[1] on learning Rust from a C perspective and I highly recommend it.
> We’re really excited to have the first commercial cloud computer — and for it to be generally available! If you yourself are interested, we look forward to it making its first impression on you — reach out to us!
Can we please stop with the "the only way to get our pricing is by booking a sales call with us." This is a 100% surefire way for me to never pay for your product, and instead go to competitors who provide straightforward, no-nonsense pricing on the website.
This is ironic given the amount of self-praise they give themselves in this article about how much they care about shipping something you can buy once instead of renting from the traditional cloud. Great, so then tell me how much it costs...
Their pricing starts at $500K (but realistically will be $1-2M per order at a minimum). This is not intended for people who browse their website, click "buy" and fill in their credit card info. If you don't want to talk to a sales rep you were never their target customer, and I'm sure they aren't sweating the hypothetical lost sale.
Making it harder to compete with them is a good thing for Oxide, obviously, and therefore why it provides negative incentive for them to publicly advertise pricing.
It's not rocket computer science to spin up a pseudo-consulting entity [that works with larger enterprises] to get on sales call for competitive intelligence.
I wonder how much the premium is over traditional server hardware with the same capabilities. You save some integration work and need less know-how, but it’d be interesting to compare.
Their entire point is to be different then that - if you want servers just go to HPE and spec out some servers. (Which really does beyond small scale, but at the rack level it's unheard of)
If you think you pay a fair price buying racks of servers from dell or hp by clicking buttons on their website, you are missing out. Purchase teams in my first company buying all sorts of hardware for our datacenters would spend their days negotiating and the end price was nowhere near the price listed online.
I have to say, as someone who will never buy one of these, I just want to see pictures of all the cool features they are talking about. And yet, their website seems strangely devoid of product photos! Or I have been unable to find them.
I worked photojournalism in university, and it really strikes me when people talk about how cool their novel interconnect is and how clean the design is, but their long writeup doesn't feature any pictures?? Would love to see what it looks like.
I feel the same. I'm also amazed at how much traction this is getting and I'm skimming through the comments to understand why this is so important. Pics would have helped.
They're selling a rack that hosts an entire virtual cloud. Nobody has done that yet. There have been servers with hypervisors preinstalled, but nothing like this.
> Oracle, AWS & Azure all have "Cloud at Customer" offerings.
AFAIK they manage it for you, as if you're just a colo. Whereas oxide just hands the entire rack + software over, but with no self install of any software stacks required (such as with azure)
maybe its about allowing you to create your own cloud, not using a third party vendor's software?
its not a computer you hook up to a cloud, it is its own cloud?
i only know about aws outpost tho, so I might be wrong
Kick ass project, been following bcantril since before Joyent and as a (server/infra + sysadmin nerd this really hits home): Are there plans to build some public-cloud-esque software add-ons to offer cloud-native equivalents to s3, serverless-functions, message queues, etc?
I feel like the "running your own datacenter" complexity of on prem is a big part of why companies go to the cloud in the first place - but that a good amount of cloud lock-in is actually in the software/service stack and reliance on it. I know there is a competent open source option for most of these services (and in fact most are repackaged open source software...), but they do seem pretty disparate and require quite a fair bit of knowledge and expertise to deploy and run at scale.
Secondary question in the same thread: What about "accounting"? Running infrastructure at large-scale companies (such that would look at a rack-level solution) usually entails budgeting and accounting of resources - how does oxide simplify this for infrastructure ops trying to keep the various IT consumers honest?
Great questions. We are careful about new service exposure -- object storage certainly makes sense (though having built one in the past, very mindful about trade-offs between cost/performance/availability/durability) but some higher level services may not (do not expect an Elastic Beanstalk equivalent!). There's a big gray area in the middle, where we will be customer- and demand-driven.
On accounting: yes! This is something that we have designed abstractions for from the start with respect to tenancy with our silo and project abstraction[0], and we are actively adding resource allocation and monitoring; stay tuned!
I really like the company name! It's like hexadecimal notation with IDE!
It's high time the datacenter takes a more holistic approach. For years I have been thinking that it's silly to stick a bunch of tiny fans in individual servers when one large impeller pulling air from an inclosed rack would be more efficient. Imagine large fans with ducts that come down to each rack or row of racks. Current in == current out; so a large diameter fan (5 feet) coupling down to small hose (10 inch) means the air moving through the small end will be really moving fast through those components!
The irony is that Oxide caters to the HN crowd, but the HN crowd would not be the buyer of their offerings.
Old school CIO's, who make enterprise decisions without developer support, are typically the buyer of hyper-converged infrastructure (or Utility companies who struggle with CapEx vs OpEd).
I wish them luck. There's a bunch really good folks over there.
I always thought it's just about the hardware but it seems like it also includes it's own kind of virtualization / provisioning interface for VMs, firewalls and also an overview over running software.
Does this "lock" you into the Oxide platform and you just buy into the whole thing instead of buying some server from Dell, then running some Proxmox like software and a Docker host?
“Cloud computing is the future of all computing infrastructure.”
My dad, who worked for IBM through the 90s and 00s always points out how amusing this is. We started with the cloud. Then went PCs. Then are going cloud again.
Doesn’t mean this is wrong. It’s just amusing.
If we do it properly, it should be far more optimal than all the local computers not doing anything most of the day.
Huge fan of the people behind Oxide as well as the concept behind the product. Can someone tell a layman like myself where a 44M Series A funding round lands among other Series A rounds? I was at a startup during 2021 and when we raised 25M, it was a big deal for us and seemingly huge. Compared to that 44M is insane. Is this notably high or is my frame of reference just small? Either way, I really look forward to seeing Oxide's progress going forward.
It's a company run by people who care and have cared for a long time, and they have all my faith.
People raise those sorts of numbers for web apps all the time. It actually doesn’t sound that big for such a broad hardware and software company with long and high touch sales cycles ahead of them.
Do they raise those sorts of numbers in series A, though? Other comments in this thread seem to suggest they've raised a surprisingly large amount of money for such an early stage of the typical fundraising process. If it is atypically high it would make sense to me—it's hardware which requires more up-front capital, and it's a rockstar team so I understand why investors would have confidence in them—but from what little I know about fundraising it does seem remarkable to me.
Congrats, Bryan and team! I've been following your evolution for a while now. People might just look at it and say "It's just a rack of CPUs and storage." And I can only imagine just how much you might be tempted to throttle them (or at least flame them in online posts).
Over the decades the separation between "software" people and "hardware" people has grown. With "cloud" people have grown comfortable papering over poor basic performance by abstracting away even your visibility into how a system is running. "You don't need to worry about that! It's ~serverless~." But you'll worry when that bill comes due at the end of the month.
With systems like Oxide, you are allowing users to once again actually see what they get, and ensure they get what they paid for, from a high-end cloud server.
It should be putting other systems providers on-notice that the days of flaky, non-performant, and poorly-integrated components are behind us. People want beasts from their servers.
And software designers, this is also your wake-up call that you can't just put lousy-performing software with poor CPU utilization and memory hogging on big metal and hope that it's "good enough." You really need to design your software to run ~efficiently~ in such systems.
> People might just look at it and say "It's just a rack of CPUs and storage." And I can only imagine just how much you might be tempted to throttle them
I'm not even the target market for this and I completely understand the feeling there. There's a lot of people who just have no conception of computing hardware anymore. People who are tied to AWS and have spent their entire careers working with AWS and now simply believe with a passion that "that's just how it is" as AWS continues to raise prices while their own cost of compute continues to get cheaper and cheaper. This is a race to the bottom.
This is a hot take, but I think Moore's law/Wright's Law has actually been disastrous for the entire field of software engineering even while it's been an amazing boon to software businesses.
Yep. The real world is still far from being fully hosted in the cloud. I've been in a few places which would massively benefit from Oxide's approach. Also I'm a fan of the team from the joyent days so probably I'm biaised. But I've seen a few poor implementations of the "private cloud" and oh boy, I wish well from the people who will maintain that in the long run...
> The computer that runs the cloud should be able to be purchased and not merely rented.
This 100%!
Not everything needs to be a subscription. Sure there are running costs for operating that cloud computer as well as ongoing software development costs. But having photo albums in the cloud shouldn't cost me $30/month just because of storage - let me buy that hard drive (and maybe compute why not) and pay $10/y for operational costs
$10 per year, minus credit card fees and taxes. If you send a single email to customer support in 5 years the company already loses money. The company can't just sell you the HDD and put a warning in the FAQ that you are responsible for data loss. Because that will result in angry emails and bad reviews.
How many engineers and how many years does it take to build the infrastructure, write all the software, and deal with all the hardware to provide a photo service that "just works"? How many customers do they need before they break even? And obviously they can't raise VC because a business model that is predicated on making basically no money per customer can never have an exit. How would you bootstrap such a thing?
It feels strange that physical storage costs almost nothing on amazon but the storage attached to a cloud machine is expensive. But run the numbers and you'll see that running a cloud service isn't about hardware costs at all.
As it appears Intel Capital is one of their Series A investors, I wonder if this has some bearing on the chances that there could also be an Intel CPU-based Oxide rack (or maybe individual sleds?) in the future.
It seems that Oxide's product is rather tied to AMD's CPU offerings for the time being (which perhaps will serve them very well for now), and given what they've accomplished so far, I imagine it would take quite a bit of effort on the platform initialization side (and other lower level stuff) to get things working with Intel CPUs instead. Surely the ability to have vendor diversity for many of their components should be an advantage for Oxide (and their customers downstream as well), so maybe there's something to think about there. Of course, this is all interesting only once Intel actually gets competitive again on the datacenter CPU side of things, where they seem to have really dropped the ball in recent years.
On the other hand, their networking switch hardware uses Intel components (Tofino), so maybe that'll be the extent of Intel's integration in their rack for the foreseeable future?
This is interesting as I think Intel have stopped developing Tofino, after buying Barefoot networks back in 2019.
AMD recently purchased Pensando, I'm not sure if their network chips are similar to Tofino but they seem like they are P4 compatible DPUs so they might be a good choice when migrating off of the dead Tofino platform.
So if anything Oxide will be moving further away from Intel!
I would love to work on this kind of stuff, not sure how to get started to end up working somewhere like Oxide..
> This is too startup-y and niche and too small to attract enough companies on the market.
From their press release:
> Oxide customers include the Idaho National Laboratory as well as a global financial services organization. Additional installments at Fortune 1000 enterprises will be completed in the coming months.
> And what was the reaction of lucky rack customer #1? “I think it’s fair to say that the customer has appreciated the transformationally different (and exponentially faster!) process of going from new rack to provisioned VMs.”
> The same day Cantrill appeared as a guest on the Software Engineering Daily podcast.[1] (“The crate, by the way? Its own engineering marvel! Because to ship a rack with the sleds, it’s been a huge amount of work from a huge number of folks…”) Cantrill wouldn’t identify the customer but said that “Fortunately when you solve a hard problem like this and you really broadcast that you intend to solve it… Customers present themselves and say, ‘Hey, we’ve been looking for — thank God someone is finally solving this problem’.”
Niches can be profitable. Not every company has to be "web scale".
> I know a 4bn/y revenue company that could greatly benefit from this but they will never even consider buying from a company like this.
What does "like this" mean? Small? Every company starts out that way.
I think its cool that someone is trying to do startup hardware things, as in recent years it seems like its mostly software. I don't a use for the product in my area of IT, but I wish them luck.
I wonder how effective AMD 7840 would be as a cloud CPU. 8 cores, fast GPU, AV1 video encoder. Tiny SBC machines …. you could probably stack a bunch of them in a rack.
"Microserver" designs have failed several times. AMD Epyc is basically twelve 7840s in an even smaller space and you can partition it any way you want.
Is this a big deal? I clicked on this skeptically, but...it actually seems like kind of a big deal? It is at least technically impressive, since it seems like they're eliminated a ton of datacenter pain points with the mechanical design.
Oxide is a bit of a darling child on HN - I think a lot of people (myself included) find it interesting that they are trying to re-engineer something somewhat commoditized and boring (servers) into something cool
It sort of reminds me of a Dell blade chassis that I pulled out of a DC last year - it was pretty impressive tech that I hadn't dealt with. 6 power supplies that I think could take 110v or 220v, but you needed 220v for full capacity. All of them were hot-swappable and as long as you had 3 powered it would keep running. The compute sleds were modular and it also had integrated networking as mentioned in this article. All obviously crafted to a very high standard; even the plastic fan cages felt premium (and I'm told it cost accordingly back in 2010 or so).
Also a former employer who was a bit below the hyperscaler level also had what they called "roll-in racks", though I believe they were just taking standard servers and networking gear and integrating them somewhere (presumably with cheap labor) and then bringing them into the DC as needed.
So I don't think it's a particularly new concept, but I agree it looks like it has some potential as a product.
It seems like they have managed to fit 32 1P servers in a 42U rack. That's pretty impressive, I have to admit. I don't think I've seen that level of density before.
32 1P servers in a 42U rack is impressive? If you use 1U servers, they would only use up 32 units, leaving with 10 units for storage and networking.
Also, I remember for example HP BladeCenter BL20p systems. There you would have up to 8 or 16 servers per 6 rack units IIRC. So about 56 to 112 servers in 42U.
They use Tofino as the networking switch. But Intel is discontinuing Tofino so I'm wondering what does future upgrade path look like for Oxide? Would they consider something like a p4 programmable DPU as the replacement?
Oxide racks curren use AMD CPUs and and after buying Pensando AMD now offer P4 DPUs. I would hazard a complete guess and say maybe a move from Tofino to Pensando is on the cards for future networking revisions.
It kind of can if you want to add programmability to your switches. So on top of traditional switches, you can add a cluster of DPUs in order to add networking functionalities to your switches. Microsoft has done just that. https://www.usenix.org/system/files/nsdi23-bansal.pdf
But there's no pricing! How will I know if this could potentially save me money compared to AWS if I cannot make a sensible comparison until I contact sales (or probably after I contact sales).
If you are genuinely in the market for multiple racks of servers you (a) know how much a rack of hp/dell gear costs, which gets you within an order of magnitude of what this is going to cost, and (b) would not buy one of these without a sales call even if you could.
I wouldn’t buy these without a sales call, but costing it out to less than an order of magnitude would make it a lot easier to determine economic viability.
I just hate calling people to determine if we’re going to match, when they could have given me that information up front.
Why though? Quantity based pricing is not hard to express in a table.
What is so different about CapEx and a bag of peanuts that defies even providing an approximate range? Number of zeroes, no?
I don’t like it, but it is the norm sadly. It’s just a signal for “it’s expensive and if you even care about the number we don’t want to talk with you”.
In fairness, the blogpost argues that owning is more economical than renting. Everybody knows what cloud compute costs and if this more economical then maybe there should be some price indicator?
I see lots of comments debating this technology choice or that one, as if the economic buyer actually cares. I dont think they do as much as the HN commentariat might think.
All that really matters is whether Oxide's guesses about who'll buy and what they'll pay are right.
Lets say that Oxide are guessing that their lead customers are telco and oil/gas: conservative industries, regulated, very strict about uptime and data sovereignty, etc. Oxide might spend 1 to 2 years running the gamut of all the people in those types of customers that can say no: ops, swcurity, network teams etc. But if Oxide can reach the finance team and get them to believe that a one time chunk of $$ is better than opex in the current interest rate environment, them it really doesnt matter what the tech stack is. (And bear in mind that financial models are f(t) so things might turn upside down as the economic picture changes over time)
Assuming they have done this math, (surely a yes, or else VCs would not be lending them money) Oxide must need sufficent funding for a long sales cycle, and plausible enough bench strength to support customers who are going to hold them to ruthless SLAs and penalty clauses. All of that take serious coin up front, eg theres no outsourcing your 24x7 support to some kid reading from a script.
There's a fundamental misunderstanding of what The Cloud is that seems to pervade discussions like this. Too many technical people don't understand what they're looking at.
The Cloud is not "a computer". The Cloud is a public utility. You don't rent the transformers at an electric utility company - because why the hell would you? It's just one component of a much larger system that is valueless to the consumer without all the other components.
What these people are selling is more like a battery that you can purchase that plugs into the electric grid. But why would you want to purchase a battery? It degrades over time, losing value, and eventually needs to be recycled, and somebody has to deal with all that. Renting is the perfect way to push all of that time-consuming and complex maintenance out to someone else who can lower cost with volume.
The idea that renting servers is not sustainable would suggest that somehow computers are not like housing, cars, shop vacs, or any of the million other things you can rent. A computer is an expensive commodity like any other, and renting makes perfect sense a lot of the time.
If you need to purchase, for economic, load, tax, regulatory, or other reasons, there are already ready-made computers with (or without) service contracts and supported OSes that can be plopped into any colo, and colos sell dedicated machines already. This pitch doesn't include anything new other than buzzwords. Yeah I'm sure they did a bunch of engineering - that nobody will ever need. I'm calling shenanigans.
I'm a bit confused about them calling it a cloud computer. The primary benefit of cloud is you spend opex instead of capex. Isn't this moving back to capex again?
There's two independent issues: capex vs. opex, and flexibility.
When opex is a primary benefit of the cloud, that's specifically for start-ups and other businesses that have little working capital. The actual primary benefit of the cloud for most cloud customers is flexibility - prior to AWS adoption, getting a server provisioned could be a multi-week if not multi-month affair negotiating between devs or operators and on-prem infrastructure workers to get the servers through capacity planning and provisioning. AWS made provisioning so simple, that you could start to set up auto-scaling, because you had extraordinarily high confidence that the capacity would be immediately available when your autoscaler tried to scale up.
But opex is not a benefit of the cloud for heavy/established businesses. Cloud opex is a financial expenditure every month that you can't get rid of and counts directly against your profits. Indeed, the desire for capex in the cloud is so high that businesses routinely purchase Reserved Instances and other forms of committed usage, which allows accountants to treat the cost of the RI as capex and then discount the expenditure through depreciation (to zero, since there will be nothing to sell when the RI expires) over the lifetime of the RI. It is normal and frequent for businesses to make capital expenditures to reduce their operational expenditures over time, thus increasing their monthly/quarterly profits.
Oxide's unique value proposition is to give customers, particularly those with high monthly cloud bills that they have difficulty reducing, the operational flexibility of cloud computing with the profit-improving benefits of capital expenditures.
The way you put it, this capex thing though is sounding like just sacrificing cashflow for some accounting sophistry.
Surely the main benefit of reserved instances is lower TCO and if you can show you can afford AWS for 3 years a bank would surely loan you the money to pay upfront if you can save say 50% it is simply cheaper even with interest.
Again this goes back to flexibility. RIs necessarily take away from your flexibility. AWS and others try to grant you the flexibility anyway, by allowing you to shift RI credits between physical instances, but lower TCO is definitely not guaranteed. If you buy an RI for a server type you're not actually using, you're spending money on servers that's getting wasted compared to not actually buying the RI.
That was the line we were sold by cloud vendors. There is a reason that significant portions of both native cloud and migrated workloads are shifting back to self-hosted; the cost savings never materialized.
Of course - every pay as you go service that goes above a certain amount of utilization will be better replaced with up front investment. But that doesn't mean that the opex model is bad in general; you've just only picked cases where it's bad.
So, what you're saying is, that whether opex or capex models are better or not...depends? That there is no silver bullet, golden gun, one-size-fits-all platitude anyone can post in an HN comment?
If the opex model doesn't work for an organization, that they perhaps just went above that utilization threshold where it doesn't make sense anymore, and would be a potential 0xide customer?
Private Cloud allows infrastructure team to run big server parks efficiently, while product teams can "buy" resources easily. It's essentially why Amazon made aws, why Google made Borg.
For a regional hospital for example, there might be a desire for in-house network and hardware - but perhaps the system for digital patient journals run on Kubernetes and managed databases.
Ya, in the past the messaging was more like "get the ergonomics of the cloud with the economics of owning your own hardware", I think it's confusing to just call it a "cloud computer"
Yes - that makes loads more sense to me. It's cool that you can spin stuff up and it's all integrated with the hardware - having seen fairly large VMWare installations and all the faff it takes to make them work, this seems really useful. I just don't think it's the same as the cloud, and the article seems to discount the main reason for the cloud: opex, letting you get started (and stopped) quickly and cheaply.
1. Are there integrator companies that take these building blocks and build datacenters for other companies?
2. Are these racks intended to be bought in large quantities and put into new datacenters?
3. Is liquid cooling an option?
4. Do you sell datacenters as well as just the racks? Do you have any of your own datacenters built out?
5. Tough question: if these are so competitive and good for your customers, why aren't you also making lots of them for yourself and also selling cloud compute?
2. Eventually, right now we are just starting, so working closely with customers and in small quantities, but eventually that is the idea.
3. No. We are already doing very well with fans, the design means a very tiny power draw. The racks are quite quiet.
4. No.
5. Selling hardware and running a private cloud are two different businesses. The public clouds don’t also sell servers. We don’t plan on running a cloud, even if doing so would be a nice experience, we have to focus on what we’re doing.
1. Seems like you would need quite a bit of capital to be one of these integrator companies, to float the cash to buy land, build a facility and buy the racks. Does your own vs rent philosophy extend so far to also say that companies should also own the land and facility on and in which a server rack sits? Early on, Amazon talked about purpose-built datacenters being important, with huge power and internet connectivity requirements, earthquake/fire/flood protection and HVAC requirements. Sounds like the best way to go, if you have a huge amount of cash, seems like unit economics are now favoring data center costs in the billions of USD.
2. For now, would it make much sense to buy one or more of these and co-locate it with a cloud company's existing facilities in an office building? Or, if you had some space in a "good" datacenter where you leased space and sublet space, connection, power and maintenance costs to your customers, would that reduce the friction for buying one of these? Or does that essentially defeat the purpose and advantage of owning your own racks? If you remove the price of renting the rack, what's the ongoing cost to park one of these in an existing datacenter? Is that even an option?
3. From a purely mechanical perspective, the performance and efficiency advantages with liquid cooling vs. air cooling seem great. Seems like the problem is mostly around installation, maintenance, application requirements and reliability. Would be great to eventually see an integrated rack with liquid cooling plumbing hookups going directly to heat exchangers. At the cost of massively increased initial cost, maintenance cost, and with a built-in severe, catastrophic failure mode... Understood that this doesn't really make sense if these racks are being put into existing air-cooled datacenters or office buildings, which seems like it's going to be the case for awhile?
5. Too bad, seems like your prospective customers are in a similar sales funnel as people who want to rent capacity in a private cloud (and it's always nice to be your own customer!) What prevents your competitors from also selling servers? Do you think that the reason that public cloud companies don't sell servers is that they think their competitors will get ahold of the hardware and copy their technical advantages? Or is it that they have the deep pockets for building datacenters that their customers don't necessarily have, and they would rather have monthly revenue anyways?
For 1 and 2, to be honest, I don't personally know much about all of that. I'm just an engineer :) I think there's a whole lot of "it depends."
3. Software engineer, not hardware, haha. I don't know if people have tried liquid cooled racks, personally.
5. "What prevents your competitors from also selling servers?" I mean, like I said, they're just totally different businesses. Doesn’t mean it’s impossible, but it feels like there’s not a ton you’d be sharing there, other than one half knowing where they’d be getting some supply from.
I saw a demo from them and it looked nice but then they told me they are stopping at the VM and storage blob level of abstraction.
Virtually all the value we get from cloud is in their managed offerings of complex and difficult to admin systems like Kubernetes, Postgres, and managed storage layers.
That’s where all the value is but it’s also where all the cost is. If you just want compute, storage, and bandwidth all those can be had at commodity prices. Look at Hetzner, Hivelocity, FDCServers, DataPacket, OVH, or VPS providers like Vultr. You can get dozens of cores, terabytes of SSD, and gigabits of unmetered bandwidth for a few thousand dollars or less. It’s very cheap.
Without the high level managed stuff we would be off cloud five minutes from now.
I’m sure there is a market for this among people who run on premise data centers, but I think they would have a much larger market if they went further up the software stack. Right now they just look like they are competing with Dell and Supermicro, not Amazon or Google Cloud.
I don't know enough about Azure Stack HCI, though I expect it's similar. For AWS Outposts you can't actually own the hardware. It's hardware leasing on your own premises. More so, it's a black box that you can't inspect, while Oxide's firmware is all open source. You can even flash your own modified firmware without going through Oxide.
I really appreciate the vision and the technology (also the dedication to opensource).
The difficulty is that HP/Dell/NetApp/etc are well established and maybe if you're a SME you can appreciate the benefits of an end to end integrated solution but most of the people making purchasing decisions largely don't. It's easy to market on CapEx but difficult to market on OpEx - you're asking customers to (likely) pay more for a similar performance invariably less featureful solution from a small company which they're then locked into largely leaning on the promise it'll be easier to operate (where is your GPU integration, object storage, shared filesystem, integrated kubernetes, etc). They then have to convince all the incumbent software vendors to support their software (e.g SAP/oracle/etc) on this specialized hardware.
To people who agree with the stated premise: "Cloud computing is the future of all computing infrastructure"
Do you think people want to design games with unreal engine on the cloud? Is there no desktop application that's safe? I don't fully accept this premise and I wonder if I'm the crazy one sometimes.
That website is interesting. Maybe it's a Firefox bug, but when I load the page with NoScript on, the text is very weird, like this:
> TODay WE aRE aNNOUNCING tHe
It's actually much weirder than that, as some of the capitals in my quote are actually big lowercase letters, not capitals (i.e. the opposite of smallcaps).
That is an odd bug I have seen before with a browser (not sure when or when). It freaked me out back then though.
Open it in a private window and see if you can reproduce. By default, a private window uses no extensions though I prefer to have uBO enabled by default there, too.
Another peculiar detail I saw on landing page is screenshot of management interface. It must be old as it shows various VMs including FreeBSD 12.1 and Ubuntu 20.10. Ubuntu 20.10 is EOLed long ago and also wasn't a LTS release. Not something you wanted running in enterprise 3 years ago, nevermind recently.
I saw the same thing and wondered why nobody in the comments mentioned how painful this site was to read. Copy/paste the text and suddenly it's all back to normal. Saved the page to disk and loaded the local copy in the browser - also perfectly normal. Very strange. Maybe something to do with their CSS?
Can't repro in FF with JS off (not exactly NoScript). Could it be an extension doing something weird? Curious if you see normal text when you view source on the page.
Wow! Congratulations! I've been following you guys for a while and have loved what you've been doing. Really want to get my hands on some of this hardware to play with.
I'd love to see a video exploring the rack and showing the first time setup.
> While the software is an essential part of the Oxide cloud computer, what we sell is in fact the computer. As a champion of open source, this allows Oxide a particularly straightforward open source strategy: our software is all open.
But their homepage says this:
> As soon as power is applied to an Oxide rack, our purpose-built hardware root of trust – present on every Oxide server and switch – cryptographically validates that its own firmware is genuine and unmodified.
That sounds like tivoization to me. The value of their software being open source is heavily diminished by their hardware refusing to run anything but their "genuine and unmodified" software.
Not sure about the tivoization. You could never send a pull request to the Tivo repo on Github, because it does not exist.
And maybe the Oxide hardware allows you to change the certificates so you can roll your own?
From my own understanding its neither (unless by "remote" you don't mean Oxide). From my understanding there's no involvement of Oxide in running the computer. You should never need to talk to them to configure anything nor should you ever need to talk to them if you want to flash the firmware with something else entirely.
Reminds of what they worked on before at Sun ZFS storage appliance.
Can it run Linux or Windows Data Center as ex-Sun folk they seem to have loved the last open source release before Oracle but now is still community developed variant of SunOS ???.
The hypervisor OS is based on Illumos, which was forked from OpenSolaris, and it uses Bhyve from FreeBSD for virtualization.
I would imagine the system architecture is different enough that running Linux as the base OS would take some work, for example it doesn't have an AMIBIOS.
> Some say that you can end up using unsafe a lot and so it’s better to stick with C or even use Zig
Using unsafe in Rust isn't inherently bad. If you're doing embedded work, using unsafe is mandatory once you get to the level of interacting with the hardware.
That doesn't mean the code is literally unsafe, just that the interactions are happening in a way that the compiler can't guarantee. That's completely expected when you're poking at hardware registers.
You still get most of the benefits of Rust, the language. I've had good success with it.
> it's just that using unsafe reduces the benefits you get from Rust's model.
No, you still get the benefits where it matters.
The "unsafe" is basically marking a boundary where you're doing things outside of what the compiler can verify. If you're poking at hardware registers, that's expected and normal.
Putting "unsafe" in a program at the hardware boundary doesn't reduce the benefits of Rust elsewhere in the program.
They seem to be cagey about their networking. Nothing besides vague "100GbE to the switch". Is this some home-brew asic or fpga? What offloads does it support? Is the driver upstream?
When Hewlett-Packard released a similar product (rack-integrated compute), it was hard to tell if the backplane bandwidth killed it or the terrible Java management software killed it. It looks like this has a better design for each.
There doesn't seem to be a way to provision a bare metal operating system, so HPC is out, and the networking is previous-generation, so there are two opportunities for progress right there.
Now that they're VC-funded I expect an OEM to snap them up before either opportunity can be pursued.
The business they're trying to build is capital-intense, and they're basically writing an OS from scratch, in addition to hardware from scratch. 78 million is actually really low
OS is not from scratch but based on proven tech like Solaris, Bhyve, Dtrace, Rust, etc. Stuff they either wrote themselves or are otherwise very familiar with.
I meant the "cloud" components, which typically represent a quasi distributed OS (because real distributed OSes don't exist) and thus are complex to create and maintain. But that's, uh, concerning, that they're also using Illumos. On the one hand, they're guaranteed to understand the system well, because they're probably the only people on the planet who do... On the other hand, makes it harder to hire for, port software to, or later sell the company due to the risk of maintaining such a one-off stack. But Joyent was able to pull off a big exit after almost a decade, so maybe these folks can too.
Canonical did something similar back in 2014 but mostly for demo purposes with "Orange Box": the promise was that you could easily do it with any available hardware.
Anyone recall Egenera? Compute only, virtualization, and fault-tolerance with the Ferrari of servers only MAANG-equivalents could afford in small numbers.
Since bhyve can't do nested virtualization, i guess they can't "dog food" the platform during development. I wonder if for that they use Linux.
The same hardware and software using kvm would be much more appealing. Not being able to run nested virt or GPUs on such a powerful and expensive rack is pretty brutal. Perhaps that is planned for the future though.
As a HPC guy, I really like the idea of one or two of these racks serving as the cluster front end, providing the login nodes, controllers, etc. You would still need generic servers for the main bulk of the compute for density and cost reasons, but it would basically make the back-end part of the cluster into an interchangeable blob of cores, allowing us to focus on the interesting bits of running a service.
I think I'm missing something here. Do they sell servers or are they selling some interesting service where you can buy a server AND they physically host the machine in their own data centers? If the former, what value add do they provide over IBM/SuperMicro/[INSERT COMPUTER COMPANY HERE]? The custom OS? You could get 99% of the way there with bare-metal Kubernetes.
We are selling servers. The value is in building the entire thing, hardware and software, for purpose. This has many knock on effects, quality, debugability, the Buck stops with us, etc.
I haven't listened to Oxide & Friends, but I absolutely love On The Metal. The guests are eminent and extremely interesting, the hosts are passionate and fun, and there's always a wonderful chemistry among the hardware geeks on the show that just makes me so excited to get into hardware and see what the future brings for us. Highly recommended.
That's an interesting juxtaposition to Microsoft's move to "cloud computer" which is a thin client with everything (including the OS) streaming from the cloud.
Or maybe it'll be a curious unintended match instead, with whatever "streaming" devices Microsoft pushes next being connected to one's own "cloud computer" instead...
I hate websites that aren't transparent about pricing. Like, how much is this crap going to cost? What are they offering exactly? I hate having to speak to some normie in sales just to get access to something. I swear - with how hard it is to buy from some of these companies its a wonder that they sell anything at all.
I feel the same. I am interested in the product, but I want to know how much it would cost. Without knowing at least the range of the prices, I find it unproductive to contact the sales representative, since the price could end up way beyond what I can afford.
How do you think this will affect companies building single-tenant apps?
I can imagine SaaS businesses offering their product as a "box": making on-prem significantly easier. Deliver your software along with the hardware, hook it into their network.
Done.
Anyone with more experience have thoughts on if this will potentially become a common option for SaaS?
Appliances were a big thing 20 years ago; they've been replaced by virtual appliances. You do not want to try to get hardware support from a SaaS company.
How can I build something for the Oxide Cloud Computer as a developer? Will there be something to simulate the APIs and so on?
Similar hardware for home labs and for personal computers with open firmware would be great. I'd also love to see servers with hardware which can be upgraded (rather than completely recycled).
We publish an OpenAPI spec for the API. The console and CLI tools are built on top of this. We have a mock server for the console to test against, you’d have to adapt that if you wanted to go that route, but you could.
What's the interface point by which I can utilise this "cloud" - is it providing an AWS like API, is it something standard I can tap into so I'm not "locked" into this just as much as I am my cloud vendor? Or is it all in on Kubernetes and I have to use that?
There is an API, on which we have also built a console, so you can use the API directly, use various CLI clients we've provided, or the console. Or you can sling JSON via curl, if you prefer.
If you want kubernetes, you'd run it yourself on top of what we offer, think EC2, conceptually.
We ship you the whole rack you bought, either to your own on-prem datacenter or your preferred colo partner etc. Once on-site, assuming the infrastructure is already in place, it's ~3hrs to uncrate, connect power and network, and do the site-specific install configuration (like DNS, time servers, identity etc etc).
> The computer that runs the cloud should be able to be purchased and not merely rented.
How is this different than colo space that has been predominant for years? Perhaps simply that it can be purchased more easily and standardized like a typical cloud VM?
I believe-as a few other commenters have pointed out and you allude to, the advantage is that it can be configured and operated like a massive set of cloud VM’s. There’s monitoring, provisioning, network, etc etc all setup and fully integrated.
I imagine colo’s have something like this, but I _suspect_ operationally it’s a lot more powerful, easy to use and functionally closer to the API’s and behaviour users are used to on current cloud providers.
Now, if someone who actually knows for sure feels like chiming in, I’m happy to be corrected.
That it's a turnkey, vertically integrated and open platform with everything included. Seems big enough differentiator for me. It's like going to going to a restaurant with table service vs a BYOB pizza shop.
It is the combination of things. Full FOSS stack, only cables being power and networking, and it being AIO package instead of conventional server rack.
Regardless of how good the product is, are companies really going to trust a startup with 100% of their hardware, software and support dependencies with no outside ecosystem and no interoperability with any of their existing infra?
I work for a competitor that has actually been shipping a similar product for quite some time now.
Yes, companies are really going to trust a startup for a stack like this. Will they go all in for 100% of their infra? Of course not, but as a test against a similar infra stack at a lower cost with an appealing feature set? Why not?
With the rack being so quiet, I wonder if they would ever sell these in smaller versions for more local installations. Would be pretty niche, but it would be interesting. Same kinds of markets that ui.com tries to serve.
Yeah the rack is custom designed to be very quiet. And it was early mentioned in the article. I'm sure they thought it through... but why? Do they want people to store this in their office workspace? It seems like a niche to me? For smaller embedded devices like home appliance and the like, fanless or the option to have low or no fan speed (like modern GPUs) seems to make sense. Because they're stored in the household or office. But servers are not and if they are they're still stowed away or fans are replaced with less noisy ones. You want the fans as noisy as possible to generate as cold as possible hardware which helps to increase heat dissipation for improved performance before any throttling.
>That the rack is quiet wasn’t really deliberate (and we are frankly much more interested in the often hidden power draw that blaring fan noise represents).
They optimized for power. That roar of fan noise doesn't necessarily mean they're doing a better job cooling.
So for power and therefore noise. Why though? I can see how it is interesting given energy crisis plus not going to squeeze out max W for performance at the cost of energy. If that is the reason, then it is going to be more cost efficient than competitors on the longer term, at the cost of perhaps requiring more hardware to possibly scale up.
The short answer is density. Smaller fans are exponentially less efficient (that is, dissipate exponentially more power for the same airflow) and datacenters -- (and racks within them) -- have a power budget. The traditional rack-and-stack approach is held captive by a geometry that really doesn't make sense (19" wide x 1RU/2RU); if power is spent on fans (and in a traditional cram-down rack, somewhere between 20-30% of power is spent on fans), it can't be spent on compute. Density is critical, as DC footprint is often at a premium.
Wow. It makes sense if I think about it; heat is the primary byproduct and therefore a typical server rack is chock-full of chunks of spinning copper coils and plastic fins. But still... wow.
Even more galling about that 20-30%: it is really, really hard to measure. Believe it or not, you can't measure the draw even in fans in a traditional rack-and-stack server enclosure (that is, they are either not on their own voltage regulators or not regulators that have PMBus support, or the PMBus support is not plumbed through the BMC -- or all of these). And these aren't the only fans! In a traditional rack-and-stack, each server has AC power supplies -- and these supplies also have fans! These supplies are often entirely dark: they don't have PMBus support at all -- and definitely don't provide insight into how much power they are dissipating in their own fans. (Just empirically: if you don ear protection and walk up to your local 1RU/2RU server under load, those power supply fans will be cranking, and you will feel a hot, stiff breeze from just the power supplies alone!) This is where all of these effects start to reinforce one another: the per-server conversion is stupidly inefficient and the geometry is stupidly inefficient (power supplies are forced to have even smaller fans than the servers!) -- all of which results in more draw, none of which is observable.
The Oxide rack is really the opposite in all of these dimensions: we do our AC/DC conversion in a power shelf and run DC on a busbar; the geometry of the sleds allows for very efficient 80mm fans; and (importantly!) every regulator is observable through PMBus and plumbed through our service processor and management network -- so you can see where it all goes.
And if you think I'm worked up about this problem, you should talk to sufferers of at-scale rack-and-stack who have been burned by this. ;)
I worked on designing computerized machinery in the past, and having sensors for as much stuff as possible just seemed like a good idea. We had quite a few current sensors for various components. Then we could do stuff like log in to the unit remotely when a customer had an issue, and diagnose based on if something was using too much current or zero current, and that beats having to send out a technician to go poke and prod. There are a lot of uses for that kind of data. And you've got a CPU right there to send it to.
> These supplies are often entirely dark
Good grief. What happens if a fan dies? The whole rack mysteriously shuts down? Or does the power supply just turn into a Minecraft lava block?
In the traditional rack-and-stack, if a power supply fan dies, the power supply will shut down -- which is why a server has two of them. (And indeed, in commodity servers, the fan in the power supply is what is most likely to fail.) All of this becomes a bit of a self-fulfilling prophesy: because there are so damned many of them, the power supplies in rack-and-stack servers are really not great -- which means they are more likely to fail. This is all sidestepped by having conversion consolidated in the rack, running high voltage (54V) DC up and down the busbar that is converted by each sled. (That converter throws off heat -- but can then leverage the high quality fans in the sled.)
Without JS this site's text is a weird mix of upper and lower case letters, even within the same word. Even large lowercase mixed in. The underlying text is perfectly normal. Can any web devs suggest why this is? TIA
Oh. I get what this is. It's a "cloud" mainframe. They're trying to be the Apple of mainframes.
Reinvent OpenStack, slap it on some 8Us and storage arrays, shove it in a rack, and ship it to a colo with a professional installer. So basically one of the larger server vendors but with integrated "cloud" software, minus the 2-hour service turnaround and spare parts.
The fact that they're writing the software from scratch is going to add years of lead time until they reach parity with other solutions. My guess is they're hoping they can get sticker price or TCO low enough that it outweighs the lack of functionality and uncertainty of a brand new everything. If you just need some VMs in a lab in the office closet, might work.
They only have $44M to build that company? I hope the next funding round is better :/
As a long time follower from the first "On the metal" podcast and as someone who has been cheering for you silently from the sidelines - Congratulations!
You all have arrived. Wishing you decades of prosperity and good fortune.
Cloud is a deployment model. Rent be own is an ownership model. You can do all four combinations.
If you’re trying to implement a hybrid cloud solution, with part of it being on-prem, you would be in the market for hardware that to use in that implementation. We are now vendors of such.
Sort of. The big deal with Oxide is that all the legacy compatibility with the IBM PC platform is gone, and the whole stack, top-to-bottom, is built by then (including firmware).
It's not like commodity x86 gear with black-box (often buggy) firmware and layer-upon-layer of hacks and compatibility kludges to present the hardware interface of a late model IBM PC AT.
The difference is that when you're buying a x86 system, the entire CPU bringup (incl. AMEGAS/openSIL on AMD) runs proprietary and poorly documented firmware. You're entirely at the mercy of the vendor.
Oxide has put immense effort into writing open-source platform initialization code, and built their own open-source BMC/RoT solution.
So effectively I’m at the mercy of Oxide, at least as long as their system does not become some kind of standard.
Not in theory maybe, but in practice. Because as a customer, I would probably also need to put in immense effort to understand and maintain that software myself.
Their firmware is open source. You can pay whoever you want to maintain it. You can't do that with Dell, HP, Supermicro, and the unknowable rabbit hole of ODMs and sub-suppliers and contractors who actually make the hardware and firmware for these companies.
Until you've dealt with a malfunctioning Dell or HP server and have to live with being told "we don't know why it acts that way, we'll try to get the ODM to repro" I don't think you can appreciate how cool Oxide's offering seems.
If I have server under maintenance with Dell or HP, they would replace the server or component for me in such a case.
Which would probably be a lot faster than trying to find someone who could maintain some non-standard firmware (as good as it might be).
Even if I had to replace the server on my own cost it would probably still be cheaper. And it would be easy to replace because it's commodity hardware, that was kind of my point.
I have had experiences with tens of Dell servers with the same model NIC having the same fault. The servers were absolutely under maintenance. I fought with tech support for weeks before I was finally told it was a driver/firmware issue and that I had to work around it (and lose performance for the sake of reliability).
Maybe if I had hundreds of servers Dell would have helped me out. At the scale of tens they told me to take what I got. The Customer got a lower performance solution and nobody anywhere could help them for any amount of money, short of replacing the gear.
That's just a performance issue. I've heard horror stories about reliability-- All the way down to disk firmware and RAID controllers. I consider myself lucky.
But how much effort (or money) do you think it would have taken to fix this issue if the NIC firmware was open source?
And with standard hardware, depending on the model, you might have had the option to add dedicated PCIe NICs for example. Not great, but at least something. Now try that with something proprietary (as in non-standard) like this Oxide system.
Replacing hardware? Sure, they'll help. What about debugging firmware though? I'm curious how much help you would get from Dell fixing and patching complicated firmware errors. A side benefit of the openness is that firmware issues can be discussed publicly, and the patches can be upstreamed into the main repo and made available to every customer (and even competitor). This gives you the kind of network effects that you'd never see in a locked-down ecosystem.
The CPUs are x64 but the architecture is not that of a PC, there is no BIOS, etc. You couldn't boot Windows or Linux on the bare metal. The hardware, firmware and hypervisor are custom built for control, safety and observability. On top of that, the application OS all run on VMs which _do_ have a (virtual) PC architecture.
To me as a casual outside observer, the fact that they're using hardware virtualization at the top of the stack, after bcantrill gave so many talks about running containers on bare metal, is the most disappointing part. They could have had unbroken control and observability from the bottom of the stack all the way to the top. They got so close!
It's possible that the hypervisor can reserve you a full CPU or full cores for the guest OS to work with, so you still get most of that bare metal goodness.
The CPU is commodity, nothing else is. Costume Mainboard and firmware without BIOS and their own BMCish thing and their own Root of Trust. Same for their router. Standard chip, everything else is costume.
Similar in some ways different in others. But in terms of not being a PC architecture. Yes it is. But in many other ways its not at all like a Mainframe.
It's similar to hyperscale infrastructure — it doesn't matter as long as it looks like a PC architecture from the OS running inside a VM. The layers and layers of legacy abstraction firmware, BMC, drivers BIOS, hypervisors you get with a typical on premise Dell/HP/SuperMicro/... server motherboard are responsible for a cold start lasting 20 minutes, random failures, weird packet loss, SMART telemetry malfunctions, etc.
This is the type of "PC architecture" cruft many customers have been yearning to ditch for years.
I’m not in the bare metal/data center business anymore at the moment, but I was for more or less the last 25 years. I never had such issues. Maybe I was just lucky?
Maybe you were. :-) And maybe this is not for you or me (I haven't contacted their sales, yet) it's not for everyone.
Personally, I have always been annoyed that the BIOS is clunky and every change requires a reboot, taking several minutes. As computers got faster over the years, this has gotten worse, not better. At the core of cloud economics is elasticity: don't pay for a service that you don't use. Wouldn't it be great to power down an idle server, knowing that it can be switched on seconds before you actually need it?
> Wouldn't it be great to power down an idle server, knowing that it can be switched on seconds before you actually need it?
considering you would still need to boot the VMs then, once the Oxide system is up, I’m not sure if this is such a big win.
And at a certain scale you’d probably have something like multiple systems and VMware vMotion or alternatives anyway. So if the ESXi host (for example) takes a while to boot, I wouldn’t care too much.
And, economics of elasticity - you’d still have to buy the Oxide server, even if it’s idle.
> considering you would still need to boot the VMs then, once the Oxide system is up, I’m not sure if this is such a big win.
To be honoust, I'm using containers most of the time these days but even the full blown windows VMs I'm orchestrating boot in less than 20s, assuming the hypervisor is operational. I think that's about on par with public cloud, no?
> [...] vMotion [...] ESXi.
Is VMware still a thing? Started with virsh, kvm/qemu a decade ago and never looked back.
> And, economics of elasticity - you’d still have to buy the Oxide server, even if it’s idle.
That's a big part of the equation indeed. This is where hyperscalers have an advantage that Oxide at some point in the future might enjoy as well. Interesting to see how much of that they will be willing to share with their customers...
Re VMware, it’s certainly still a thing in enterprise environments. Can kvm do things like live migration in the meantime? For me it’s the other way round, haven’t looked into that for a while ;)
How do you mean Oxide might have that advantage as well in the future? As I understand, you have to buy hardware from them?
Ah yes live migration, off course. We design "ephemeral" applications that scale horizontally and use load balancer to migrate. With 99% of traffic serviced from CDN cache updates and migrations have a very different set of challenges.
As to your question, I meant to say that as volumes and scales economies increase they can source materials far cheaper than regular shops. Possibly similar to AWS, gcs, Azure, akamai etc. It would be nice if they were able and willing to translate some of those scales economies into prices commensurate with comparable public cloud instances.
If you want more insight into all of the things that normally run on "PC architecture" - the 2.5 other kernels/operating systems running underneath the one you think you're running - https://www.youtube.com/watch?v=mUTx61t443A
Every PC has millions of lines of firmware code that often fails and causes problems. Case and point, pretty much all hyperscalers rip out all the traditional vender firmware and replace it with their own, often partially open source.
The BMC is often a huge problem, its a shifty architecture and extremely unsafe. Meta is paying for u-bmc development to have something a bit better.
Doing things like rack attestation of a whole racks worth of firmware if stacked PC is incredibly hard and so many companies simply don't do it. And doing it with the switch as well is even harder.
Sometimes the firmware runs during operations and takes over your computer causing strange bugs, see SMM. If there is a bug anywhere in that stack, there are 10 layers of closed source vendors that don't care about you.
Costumers don't care if its PC or not, but they do care if the machine is stable, the firmware is bug free and the computer is safe and not infected by viruses. Not being a PC enables that.
I imagine they are aware that this isn't a solution for many customers. A John Deere tractor makes a poor minivan. This isn't for you. That's fine. It's not for me either. That's ok. I don't need to poo-poo their efforts and sit and moan about how it's not for me.
The name is "Oxide Computer Company." But since "0x" is often used for hexidecimal numbers, and "0x1de" happens to be one, we play around with it from time to time.
Yeah, I read ... just "wow". I hoped something more affordable, or even something more flexible like "you get the full rack, but you are enabled to use only x % of the capacity for a lower entry price".
> Cloud computing is the future of all computing infrastructure.
I absolutely dread that future. I want all my computation back to be local, doesn't matter to me if I own the cloud computer or not as long as I also don't own the environment it is embedded in. If the connection to it isn't mine and neither is its source of power then I don't really control "my" cloud computer.
I work at a Telco that needs the teams to be able to create VMs, networks etc.. But due to legal constraints we need to host it on prem.
For those companies this would probably be a cost effective solution since now we spend a lot of resources managing varying Openstack installations with different hardware from HP / Dell etc...
feels a lot like those vblock /converged infrastructure in a box things they used to push. Turnkey virtualization environment - I guess the advantage/? here is a open source stack.
I discovered their podcast a month ago and binged through the backlog -- it's one of the best technology podcasts of all time! Really like the way these folks think and communicate!
just fooling around with their demo web console ..
"A disk cannot be added or attached unless the instance is creating or stopped."
"A network interface cannot be created or edited unless the instance is stopped."
well, i just wanted to get a feeling what experience it would be using their web console and it seems they still have some limitations in place that have long been solved by other "hyperconverged vm-deploy" players.
Want to spin up not only 1 but 20 instances via the console? Nope.
Any kind of guest agent in the virtual machine? For direct interop?
How is the guest filesystem freezed during snapshot creation?
Checkpoints for incremental vm backup?
They seem to use raw images, not qcow2? Why?
You are being negative though. You could have phrased that as "they are still new, but I hope they can catch up", which doesn't imply they messed up by being late to the game.
Disruptors never have feature parity to established players in their first offering.
their whole "AWS is bad because THEY are engaging in rentier capitalism, we are good because we sell you the tools to let YOU engage in rentier capitalism" shtick rings empty to me
the vast majority of organizations buying this product will use it to build SaaS. Now comes the part where you pretend otherwise ("but hospitals/universities need this too!"), press me for more evidence ("how do you know that's what people buy it for?"), or feign ignorance on the topic ("what's inherently rentier capitalist about SaaS?").
Nope, I appreciate the elaboration. Neither of our first two customers are SaaS companies, but there's no point in arguing about it: I have no idea if most of our customers eventually will or not. Thanks for elaborating, I at least grok what you mean now.
I regret missing the Open Source Firmware Conference--I really wanted to meet them and learn some more on what's the story here.
I know it's a brilliant team, and since ZFS/OpenSolaris/Dtrace I've known about Bryan. The product is a nicely looking product too. One thing I don't get is the target market. Who is it? Banks? Cloud vendors? Insurance companies? I think there's a part of the computer business that I don't get yet.
One guy in one of the previous HN threads here said that it's convenient to run 1 RFP for 1 box costing $1M-$2M vs running 30 RFPs for isolated components to put on the rack. Ok, I can see that. That's some argument.
But I know that Supermicro has rack assembly service. https://www.supermicro.com/en/products/rack and I'm sure Dells and HPEs have the same. Does it mean that it's impossible to call them and tell them: "Excuse me, I want 2 racks, 15 systems each, top of the line AMD CPU with RAM, NVMe storage, all maxed out, and whatever fastest Juniper switches you can find. Put me VMWare vFusion on it. Ship this thing to Infomart in Texas. I have $2M here with your name on it". They won't do this for me?
Its either this, or Oxide is going to e.g.: Bank of America, and along with RFP the bank is asking: "Show me complete supply chain records, including the source code to your boot loader, drivers, and any firmware that any component on this motherboard is running". And maybe that's ... impossible these days?
I'm the startup CTO, so not the consumer base, but after DHH started posting his thing about cloud exit, I decided to explore this, and I walked the floor of 3 DCs (in SV/TX). People who walk you around the DC don't appear to care about cables, fans or noise etc. If it's longer than 1hr on the floor, they'll put Airpods in and they're done. I've seen cages that look unified, pristine, with love and affection put into cable layout, and I've seen cages that look like a total mess. It's your cage--you only go there if things break.
My last guess is that maybe Oxide rack will end up being sort of what an Apple Macbook is among cheap HP/Acer laptops from Walmart. And it'll be a shiny toy of bold bearded IT dudes who work for GEICO IT by day, but by night they scavenge eBay for used server deals and get excited about the idea of running their own private rack in their basement. They can't afford it at home, but at work they'll want their own Oxide. If you're reading this, do know I know who you are.
> One thing I don't get is the target market. Who is it?
Fortune 1000, government. Large organizations that want to own their own hardware, yet want the cloud experience for deploying their software to it. First two customers to have received racks are Idaho National Lab, a Department of Energy laboratory, and a global finances firm. I think that gives a characterization to at least part of the market.
> They won't do this for me?
You are correct that they will do that for you. There are big differences though. From a hardware level, the largest one is that what you'll get in that case are individual 1U servers, in a rack, built from a bunch of reference designs, pieced together from various different organizations. We designed this as a whole rack. From scratch. This has a number of benefits, like for example, we use 80mm fans, at a very low RPM. Our fans draw less power and operate more quietly than the usual fans you'd get in that case. (I know you said people don't care about noise, and it's true that it's not a feature of the product to sell, just an interesting aspect of the design process: we didn't set out to make the rack quiet, it just is, thanks to other decisions that are more meaningful, like cooling efficiently) On the software side, we have written an enormous amount of software from scratch, designed for this specific hardware. Including a control plane, so you can think of the whole rack as a pool of resources, not as individual servers you manage yourself. And since we have done all of this in-house, we can take responsibility for the full quality of the product. If you have a firmware bug, it's not "oh sorry, we'll file a ticket with our firmware vendor and let you know when that's sorted," we will fix it ourselves. Everything is integrated and works together, because we built it for purpose that way, not because we installed a bunch of things from different organizations, ran some test, and said "looks good to me."
I think a lot of commenters here have little experience with hyper-converged infrastructure if they don't see this is different. This looks, by far, the simplest way to run workloads outside of the big cloud providers (AWS, Azure, GCP).
It seems they only support up to virtual machines but this is still miles ahead of VMWare, Nutanix, or CISCO ACI to my eyes. This honestly looks simple enough that a developer/DevOps team could manage it. With the other 'hyperconverged' infrastructure offerings you will still need a dedicated team of trained ops professionals to manage it.
I have a lot of experience with hyper-converged infrastructure. "Miles ahead" how? Oxide's control plane barely competes on basic functionality that OpenStack had 13 years ago, let alone VMware which is miles ahead of OpenStack. The hope is that the simplicity in hardware and network/storage/compute architecture will drive dividends as they improve their control plane software.
I don't discount that it's a great achievement towards simplicity in the racking approach for rack-scale use cases (having lived through Dell VxRack's nonsense), but let's not kid ourselves that any DevOps team could manage this - do they understand BGP peering? Three phase power requirements? Cooling? ZFS? How about basic maintenance migrations ala Vmotion or DRS? (kidding! they can't do it)
If you don't think Oxide will require an ops team, you may be wish projecting.
This point is understated. One of the main reasons cloud is so attractive to executives is the smaller devops footprint compared to a traditional on-premise deployment.
A solution as lightweight on operations as Oxide challenges this premise at its core and assuming the capex is not outrageously high, it might suddenly get very attractive.
It may be a limited market. You need to be large enough for a private cloud to make sense, but above a certain size it also should be a no-brainer to have a competent ops team making you less dependent on a singular infrastructure provider.
Maybe 37signals type businesses, SMB tech startups with a serious, stable, non-high growth business model and a tech savvy team. If you're selling $500k boxes you don't need to sell a TON of them to make a hundred million dollar business. Unless their VCs expect Oxide to be a billion dollar company with a broader market.
Perhaps this changes the calculus. If you don't need as much of an ops team to go private cloud. Then you might not need to be as large before it makes sense.
By a private cloud making sense, I mean the lower bound for being an Oxide customer, i.e. having the budget to buy the infrastructure with enough headroom for usage spikes and expected near-term growth. The reasons why you otherwise would go public cloud when you’re still small. The reduced need for an ops team with Oxide was already factored in.
So uber did private cloud, then went to public cloud recently because the balance was not worth it ( and they probably had enough scale to negotiate a good price, like netflix ) . If they were on something like oxide which reduced infra management costs a lot staffing wise and negotiated a bulk deal, would’ve they gone the public cloud route?
> This honestly looks simple enough that a developer/DevOps team could manage it. With the other 'hyperconverged' infrastructure offerings you will still need a dedicated team of trained ops professionals to manage it.
That seems like the best explanation of the value here I've heard so far! I could definitely see a number of SMBs whose use cases are expensive in AWS/etc and would be a good fit for a few racks in a datacenter, but who don't want the cost of a team to manage them. If this significantly lowers the skill threshold necessary to run servers in a datacenter, I could see that being a huge value prop.
Salary is that high in the US because we have no social net or price caps here. Your on your own for everything. Healthcare, retirement, overpriced homes, out of control rent, etc
Overpriced homes, out of control rent, retirement are extremely problematic in a lot of European countries. And in the US, healthcare is usually covered by the company (talking about big tech corps).
So the extremely high salary is still a net positive compared to EU
With this in mind, it's just a good salary. Until they have competition squeezing them on margins it looks like an OK approach. At some point their board most likely won't agree with paying some people below market rates for important roles and other more fungible roles paying 2-3x market rates but while they can keep doing it, it's great free marketing.
Wouldn't an employee making that $201k salary in another country be responsible for paying all the taxes to support the socialized healthcare, etc? In other words the take home pay may be significantly less.
Other taxes (GST~VAT 15%, city rates, high petrol excise, etcetera) will easily take another $10k. Interest on your home is not deductible (NZ has very few deductables). People earning six figures will often pay for private health insurance and medical fees on top of the socialised healthcare - maybe another few thousand.
retirement is covered by social security
healthcare by insurance (offered by the state if you can't afford it yourself)
homes, etc is true though and we just need to build more inventory IMO. but there are tons of areas with affordable homes, they just aren't near the big cities like NYC or SF or LA
You can not live on Social Security, no way. If you don't have a job your not getting healthcare and even if your job provides healthcare. Its too expensive to actually use. Also you need to be living below poverty wages before a state will give you healthcare.
While that may be true, it's also unfortunately true that stock in startups has a high probability of ending up worthless. It's important to compare real money now, not just theoretical money in future.
I doubt my Aspen vacation home developer will accept payment in startup stock.
That's probably [EDIT: decidedly] on the low side for the bay area for rock star devs, and Oxide has lots of rock star devs. I haven't looked but I assume they pay bonuses, probably differential bonuses.
Hi Steve, bit of a tangent, I stumbled upon your blog from intro section and went down the rabbit hole. Wanted to ask if you have any recommendations on documentaries. TIA!
Like Steve, I also took a pay cut (from around 400-600k a year TC at a FAANG) to work at Oxide. I'm very passionate about what we do and was attracted to the values-oriented culture.
Exactly, man! That's the way it should be done. It's not a company's business to decide how much your life should cost. It's its business to reward you for the value you provide, and that value is not tethered to your location, so neither should be your salary!
This really is one of these things where every employee in a cheap cost of living area will say "Exactly, man!" and people in an expensive area will see it differently because it might cap them lower than what they could get.
I always find that a very naive point of view, of course I would want to earn a US tech salary while living somewhere in the country side, who wouldn't. But I'm also aware that this is not how the world works in reality, there's different tax systems, different expense costs and we don't live in a global one-market world.
I find the strategy of defining different "zones", like most of the remote first / salary transparency companies much more realistic.
> of course I would want to earn a US tech salary while living somewhere in the country side
It's not about what you want, it's about knowing your value. If your work is worth a SF salary then that's what you should be getting.
Moving from Idaho to SF doesn't magically make you more productive. The company knows it's still getting more value from you than what you're being paid. They just want to keep more of that value for themselves whenever possible.
Have some respect for yourself and know your worth
Being in SF makes the market for your labor more competitive. If I'm living on a ranch in rural Idaho, I would interact with very few people on a given day, and most of them would be the same people I interacted with yesterday. In SF, I'd be interacting with far more people, and far more new people, with a much higher probability of those people working in tech, some subset of whom will be willing to offer me a job.
> Moving from Idaho to SF doesn't magically make you more productive.
If your entire world consists of staying home and interacting remotely with a company, then you are correct: Location doesn't change anything.
However, moving to a high-energy city with a high density of experienced engineers and tech companies can increase your rate of learning, career advancement, and experience much more rapidly than living in a smaller city. You have to actually branch out and interact with local companies and people, but it does happen.
But this is all beside the point. Hiring is a labor market. Developers who live in SF have more high-paid job options to choose from than someone living in Idaho. As a result, you need to bid more to get them into your company. Hence, the higher salary.
The discussion about cost of living misses this point. The real reason developers from places like SF get paid more is because if you don't pay them wages that are competitive with their local companies, they're just going to walk away and take any number of higher paying jobs they have access to.
I am so confused by this. I demand more in the Bay as Bay area landlords and their Nimby pals are exploitative jerks and California taxes and fees add up quick. Why is that cleanly separated from the discussion? I see it as more “I will demand more here than other places” vs. “I know my worth and it’s exactly X”
> This really is one of these things where every employee in a cheap cost of living area will say "Exactly, man!" and people in an expensive area will see it differently because it might cap them lower than what they could get.
Well the co-founders live in the Silicon Valley area, with their physical HQ being in Emeryville:
If you owned a home in the Bay Area prior to 2020 and refinanced down to a sub 3% mortgage 200k is plenty. If you’re trying to buy now on that salary it will be challenging.
Early stage VC-backed startup compensation is very different from later stage companies or bootstrapped companies. It's very common for founders or early employees to receive a lower salary and receive equity instead.
I don't know how you're coming to that conclusion. 200k's not going to buy you a large house, but it'll comfortably pay rent/food/savings for a 2-3 bedroom house or apartment for a family of 2-4 in all but the very most expensive parts of the Bay.
For mortgages, you'd need to be looking in the cheaper parts of the Bay, but that still means "dense, boring suburb" as opposed to "crime-ridden slum".
Assuming interchangeable human resources there are basically 2 options, either the company is extracting more value from everyone than the salaries they pay, in that case paying fairly would eat into the company's bottom line. Poor CXOs would not turn extra profits by keeping less fortunate employees on low salaries but just the regular one. Or the highest paid employees are not pulling their weight and their salaries are already subsidized by the rest of the company, which is also not quite fair.
Nevertheless, I agree everyone looks at this problem from their own POV, however it should not be the norm to provide equal compensation for equal work.
I understand! That's what I suspected, but wasn't sure. I tried understanding it for a minute then thought I'll just ask you! :) haha
Also, thank you for the links. I'll read this because I'm interested in knowing more about thinking behind this. My instinct is in this direction regarding salaries, but I want to develop more clarity. :)
If that's your blog, well done! I am totally on board with everything you say there. I've been thinking this way for years, thanks for articulating some more reasons why this is really important! Love your work! :)
Hmmm...I actually had to look up what crab mentality meant but I don't see how it applies here. From my quick lookup, crab mentality apparently refers to a reactionary state of mind where a person wants to sabotage another's success even when it doesn't directly impact their own.
In the case of everyone making the same salary, you could certainly still sabotage someone's success but I don't see how having the same salary makes this type of mentality _more_ likely than at a company with a more typical salary distribution.
My concern with everyone having the same salaries is that, potentially, employees have less motivation to excel in their individual work and are more likely to do the minimum to just stay in good standing with their employer and not get fired. Maybe a company can offset the lack of direct financial motivation with more of a team motivation that the financial success of the company as a whole results in financial reward for the individual or some other way of recognizing individual success inside the company.
I am doubtful this flat salary structure will result in a more successful company overall but I do think it's good to try new things. And yes this has probably been tried a number of times before but maybe not exactly like this. Or maybe some other external variables have changed w.r.t. other attempts in the past and this time it works. The typical salary structures we see in US companies today are the result of a large number of trials and errors and learning.
Ah I think you misunderstand. I think employees in the west being pissed that someone in a third world country makes the same salary as them is crab mentality.
I totally understand why companies want to pay less though. It's massive cost savings and it makes sense for them to hire for less money.
Replace "somewhere in the country side" with "some other place where the cost of living is lower than the highest one in the country" if it makes you happier.
My employer takes cost of living into account by multiplying your salary by the numbeo factor for where you live. Base salaries are all calculated for Cologne (HO) and then adjusted by the relative cost of living factor for your locale. (we're entirely remote)
I am personally not a fan of this: early in my career, a colleague moved from the Bay Area to Boston to try to save his marriage (which sadly didn't work) -- and our employer adjusted his salary down accordingly. As you might imagine, the difference between the Bay Area and Boston is minimal -- they adjusted his salary down something like 4% -- but I just recall how dispiriting it was for him to have this massive, stressful (expensive!) life transition exacerbated needlessly. Employers don't factor in other elements of cost-of-living (can you imagine giving someone a pay bump when a kid goes to college -- or a reduction when they graduate?!), and I don't believe geography should be factored in either: pay people for their work, not their ZIP code.
Whilst I completely understand your viewpoint, I have a more nuanced take on it (understandably, as I work there). Firstly, we don't pay peanuts. We have never taken outside funding; it has always been a company goal to grow organically. As such, we don't have as much cash to go throwing around like VC-funded startups. However, as I said we still pay well; yes I could make more by taking a job in London (I'm in the UK), but then I sacrifice so many intangible things. Working remotely has allowed me to be present whilst my 18 month-old grows up, and a 4 day work-week has helped even more too. Unlimited holidays, very flexible working hours and being treated like an adult about how you organise them, a flat company structure, a company which genuinely cares about employees and their wellbeing - all of these things add up. Salary isn't everything. Whilst no job will ever be perfect, this company is by far the best place I have ever (and will likely ever) work. Employee satisfaction is always very high and salary is rarely even a factor (let alone the main reason) if people decide to move on.
Just like Wikipedia and many other information sources. Don't trust one source blindly, do your research and you'll be fine. It gives a good enough general indication. A comparison like this will always depend on too many factors to make it applicable to everyone in any case.
Your perspective on egalitarian salaries is intriguing, but it's important to remember you can't speak for others regarding their intentions or their whys, or what it is for them. You can't pretend to define that for someone who is not you.
While you argue it's naive and cite different tax systems and costs, companies are successfully adopting this approach even in high-cost areas like the Valley. It might be hard to argue that the world doesn't work like that in reality, given that it’s already happening.
While it may not be widespread, it doesn’t mean it’s without merit or unrealistic. After all, remember that today's 'unrealistic' could be tomorrow's norm! It's tempting to think that people in less expensive areas would be the main proponents of a uniform salary, but the reality is nuanced. Preferences are likely influenced by a variety of factors, not just cost of living.
Your later points about the differences between compensation at different company stages are well taken, however it could be difficult to assert how much this dynamic affects preferences given the practice is not limited to early stage companies and equity vests often fail to yield returns.
In addition, your suggestion that only people in poor places want egalitarian salaries, could be seen as disrespectful of other people, because it seems to ignores the totality of an individual while preferring to try to reduce them to simplistic motivations. In that way, it’s also considered abusive. And can also be seen as disrespectful of others experience, and maybe arrogant: "Only people in poor areas want such naive, unrealistic salaries."
Looking deeper, this aspect of your comment, combined with its narrow focus on a single explanation, might be interpreted as your attempt to express your personal frustrations at your own salary performance, or justify and rationalize why you may not be making more. This might occur because you may find it easier to view something you don't have as unrealistic and naive, rather than the result of choices you could change.
In short, while you mention that egalitarian salaries and enthusiastic support of them is naive and unrealistic, it could be argued that the view espoused in your comment is naive and unrealistic because: it lack awareness of complex dynamics; ignores the totality of an individual while preferring to try to reduce them to simplistic motivations, and dismisses real practices as unrealistic, which might also be seen as out of touch. Overall, your views unfortunately could be interpreted as narrow and an expression of personal frustration, instead of a reflection of underlying real dynamics.
To conclude, while it's likely there's some truth to the correlation you propose, it's also likely true that even if some correlation exists, location is not the only factor at play. People may have various salary preferences, independent of their location, just as the value they provide is also independent. Finally, indeed, your view could be expressed more respectfully of others.
Anywho, it's understandable you may have that perspective, given what your background might be. Yet it's always good to remember that you can adapt your view over time, can grow and can include more data to expand that awareness of reality which you value! :)
Except if everyone gets paid the same, you're not being rewarded for the value you provide. I don't think salary should be tied to location, but it should be tied to experience, ability, and effort.
You raise an important point. There's certainly multiple ways to look at it.
The egalitarian way where a business can divide the share of revenue that is allocated for salary equally, no matter the role. This makes sense from a philosophy of everybody being a team and contributing equally to the results of the company. It can foster an esprit de corps and and a sense of fairness. On the negative side it could encourage companies to have more burdensome measures of fairness and contribution, and lead to resentment towards colleagues who don't pull their weight.
Then there's the other method, where a value-based salary is allocated to each employee taking into account their experience, ability and effort. Crucially, however, this salary is not adjusted for location. That's the case to which I was speaking, specifically, even tho the type used by 0xide is clearly the egalitarian one.
> I don't think salary should be tied to location, but it should be tied to experience, ability, and effort.
That's the labor theory of value (see: Smith, Marx), which in theory sounds meritocratic but it can't really be measured or assessed.
In reality compensation either becomes a function of power, social currency and negotiation skills, which is the general norm in professions, or you have an institutionalized, perhaps even democratic process to determine salaries. Both of these variants generate overhead and are only approximations to what anyone would see as fair.
The variant here where everyone gets the same, generous piece of a pie seems refreshingly simple and honest. I would also assume that it attracts the right kind of people, who are intrinsically motivated (at least after the threshold of a very high level of comfort is reached.)
Saying that people should be paid according to experience, ability, and effort is absolutely not the labor theory of value.
The idea that contribution "can't really be measured" is a cop-out. Contribution can't be measured perfectly but it can be estimated with some accuracy by people who are involved in day-to-day work. "Some accuracy" is really all that's required: as long as contribution is correlated with compensation to some extent, you have a functioning meritocracy.
> The variant here where everyone gets the same, generous piece of a pie seems refreshingly simple and honest.
I bet it works great if you have a small team, are extremely picky about hiring, and quickly fire bad hires. Otherwise it will be awful.
The problem is that in the end, power, negotiation and social currency often dominate over merit (such as effort, experience etc.) when it comes to compensation.
Even if you actually do measure and agree on metrics, then the measurement can easily become the goal for those who are not intrinsically motivated. Work ethic can't be taught by dangling carrots in front of people, because acquiring the carrot becomes the goal instead of moving the cart. This can be detrimental in a highly collaborative workplace.
Having a flat, generous salary might solve this problem, because you filter out carrot hunters and get cart movers.
> I bet it works great if you have a small team, are extremely picky about hiring, and quickly fire bad hires. Otherwise it will be awful.
Finding the right people to work with is difficult regardless. The same worker can be miserable in one place and flourish in another.
> Contribution can't be measured perfectly but it can be estimated with some accuracy by people who are involved in day-to-day work
This is handwaving in the extreme. Anyone involved in software development knows that a single line of code can be critical to success or failure, as can a blob of 100k LOC, so product-quantity metrics are of almost no use. The "estimate" you're talking about generally comes down to general "feelings" about who works hard, which have repeatedly been shown to be poor metrics for actual contributions.
Life is full of handwaving. In almost any workplace, it's very simple to know who's doing the work (and it's usually a shockingly small number of people). It's the idea that we can reduce this to a mathematical formula (the opposite of handwaving) that's odd.
> The "estimate" you're talking about generally comes down to general "feelings" about who works hard, which have repeatedly been shown to be poor metrics for actual contributions.
How has this been "shown"? Anyway, you're begging the question that there's some way to determine "actual contributions" that we can compare to "feelings".
If you actually work with a group of people on a daily basis and can't rank order them in terms of usefulness, I find that astonishing. And remember, rankings don't have to be perfect, they just have to more accurate that random.
> And remember, rankings don't have to be perfect, they just have to more accurate that random.
No, they have to better than both random and "everyone is, on average, and over an extended period of time contributing roughly the same". That's quite a challenge.
Do you go to customers and ask them which features provide the most value to them, and then follow the code back to the people who implemented them? Do you go to the customers who paid the most, and repeat the question to them only?
We're not talking about some award-prize ceremony speech in which we acknowledge that Dmitri and Aneka led the work to get version 8.0 which has been a huge success. We're talking about actual salaries, which are presumably linked in some way to actual sales, and I'm insisting that connecting individual developer efforts to the sales numbers is extraordinarily hard. "More" and "less" are not enough to come up with actual numbers.
> No, they have to better than both random and "everyone is, on average, and over an extended period of time contributing roughly the same". That's quite a challenge.
This is something virtually all functional companies do when they decide raises. The fact that they don't do it perfectly isn't a huge problem, they just need to be more right than wrong.
> Do you go to customers and ask them which features provide the most value to them, and then follow the code back to the people who implemented them? Do you go to the customers who paid the most, and repeat the question to them only?
Does productivity equal sales? I don't think so. If someone does a good job implementing a feature that doesn't drive sales, that should count toward their productivity. Equally, imagine a task that could be assigned to anyone that drive sales: it doesn't make sense to reward the person who happened to be assigned this task when anyone could have done it.
You're demanding way too much here because you're unwilling to get "handwavy" and instead want some rigorous way to quantify productivity. Instead, embrace subjectivity! Imagine you're in charge and ask yourself questions like:
1. If I need to organize a meeting to address some problem that needs to be solved as soon as possible, who would I invite to the meeting?
2. If someone tells me he plans to quit, how much would I be willing to offer to convince him to stay?
3. If someone quits, how hard are they to replace? In terms of hiring a replacement and/or transferring their responsbilities to someone else.
I suppose there are some workplaces where it's genuinely hard to rank people. But my sense is that they're rare, small, and careful about hiring. Everywhere I've worked, this is not the case and I'm fairly sure this is the norm.
The 2nd and 3rd questions you pose are all about sales.
If your company brings in $10M a year, and somebody plans to quit (or has quit), how much will your sales drop (immediately, and over a period of time). That's the answer to how much you can afford to offer them to stay, and that's how much you should offer their replacement. Suppose that says drop by $1M and you can be satisfied that the drop is 100% a consequence of the departed (or soon to depart) developer - that's how much value they bring to the company, and by the logic of capitalism (which I don't play by, btw), you should pay them some amount less than that.
The problem is: you can't determine the value before they quit, and you can't be sure that their replacement will provide that value after they are hired.
Look, I understand that in an organization of any size, there are likely to be slackers that feel like a deadweight, and others who feel like the contribute far more than the average.
The question is: does tying salary to this perception actually bring the benefits you think it does (which are intimately connected to the notion of incentive) ? There's some good evidence that for developers and other "head-based" employees, it does not, and that flat pay scales create an environment in which you get different kinds of benefits.
I've worked exclusively in a distributed FLOSS project for the last quarter century, so in many respects, I'm not well positioned to talk about what happens inside traditional corporations.
That is not the labor theory of value. The whole point of LTV, at least in Marxian economics, is that workers can't be payed according to their socially necessary labor, because a surplus labor is extracted.
Besides, LTV as a theory is meant to be a description of the world as Smith, Marx, etc. see it, not a prescription for how things should be done.
You're right that it is descriptive and not normative.
But the underlying belief of paying someone according to their effort, is very much based on the same premise.
What I'm saying is that nobody is _really_ paid according to their effort, experience etc. because those things cannot be reasonably measured.
The typical process of determining compensation is based on negotiation and power. In some places the process is more democratized and rules based. Both of these are only to some degree related to actual effort, experience and so on. This discrepancy becomes larger the more people are involved as well.
Location independence is not the interesting part of Oxide’s compensation strategy. It’s that everyone makes exactly the same. There are no negotiations, no levels, no promotions, and consequently no promo packets or promo projects. This is extremely enticing to FAANG types who are tired of a certain kind of bullshit, at the cost of a certain level of ambition.
I wonder if they offer equity. Hierarchies always form, they tried this with Gore Associates in Arizona (totally flat company structure) and has major problems.
Yeah this I’ve seen everywhere, but more related to the company size. In a 50 employee company, it’s hard to pull that off. As organizations get larger, you get this as inevitable part of human nature. If the salary table is flat, then highly ambitious people will ask for more skin-in-the-game.
The skin-in-the-game bit can be promotions, equity, stock compensation, etc.
I don't understand why you would compare total compensation at Oxide with base salary at other employers? Total compensation is the relevant figure across the board, and senior (as in, experienced, not some specific levels.fyi tier) engineers often make significantly more than $201,000.
Oxide equity, like for all private companies, isn't worth anything until and if they go public; it has no cash value. For total compensation purposes, it's $0.
> Oxide equity, like for all private companies, isn't worth anything until and if they go public
Open AI just had a tender offer for employees this month. It's not public. Clearly the equity is worth something even though the company is not public.
Some companies facilitate secondary transactions for their employees.
There are ways many employee friendly companies provide early liquidity without IPO.
That is only a problem if you insist on hiring people from such expensive areas. Given Oxide hires remotely, I doubt this is an actual problem for them.
Hence the word "concentrated". Your snark seems misdirected.
I live in flyover country, and there are no shortage of talented people here, but I wouldn't dispute that the per-capita software developer talent in SF is going to be higher.
I think it's mostly societal perception. For whatever reasons, a lot of people seem to think of professionals in SF and NYC as smarter or more capable than professionals from Boise or Salt Lake City. Just like a lot of people assume those who got into Harvard or Stanford as 18yr olds are smarter or more capable than those with degrees from the large public universities.
user: sunshowers has been replying in this thread and mentioned going from FAANG-> Oxide and taking what seemed like about a 60% paycut.
Yes it is somewhat of a stumbling block, but I'm betting they are getting extremely high quality employees who are doing it for the passion and not just for the money.
We've got designers, me included, and a few others on the team who aren't engineers. I also hail from what people like to call a "third-world country". So far it's working great, and we're hiring more folks from all over the world, not just the USA.
I always had a feeling that Bryan and people around kinda took the tech first approach and it never really came to a proper fruition. I think the ambition behind the engineering marvels they did over time was certainly bigger. Dtrace, Fishworks, Joyent, then this... so why is that? Perhaps the fierce refusal of joining the mainstream side (Linux)? Fingers crossed, but...
You've been downvoted to oblivion, but let me add a personal rebuttal for anyone else that finds themselves here: I don't agree at all about "not coming to proper fruition": DTrace was an absurdly outsized success by any metric; Fishworks was if anything too commercially successful (we did nearly $100M of business our first year!); Joyent represented a successful exit to Samsung, etc. I have been extraordinarily blessed to be in the right place at the right time with the right folks several times over -- but never more so than at Oxide: our ambition is absurdly ambitious, but it has never felt more attainable!
All of that said: thank you for keeping your fingers crossed. ;)
Let's start with their beliefs:
1. Cloud computing is the future of all computing infrastructure.
Yeah, no. There's still gonna be things like mobile phones which will do some stuff locally. And probably also desktops. And probably also on-prem servers.
Which brings us to their belief no. 2:
The computer that runs the cloud should be able to be purchased and not merely rented.
If you buy a computer and run virtualized loads on it, then you're not doing cloud computing. As simple as that.
Basically, as others have commented already, this is just something like a blade server system or a hyperconverged system like for example Nutanix is offering. The only real difference seems to be the open source approach.
They sell servers, but as a finished product. Not as a cobbled together mess of third party stuff where the vendor keeps shrugging if there is an integration problem. They integrated it. It comes with all the features they expect you to want if you wanted to build your own cloud.
Also, they wrote the software. And it's all open source. So no "sorry but the third party vendor dropped support for the bios". You get the source code. Even if Oxide goes bust, you can still salvage things in a pinch.
Ironically this looks like the realization of Richard Stallman's dream where users can help each other if something doesn't work.