On one meeting we had a typical discussion with ops guys:
- "why wouldn't we optimise our hardware utilisation by doing things a, b, and c."
- "hardware is crap cheap these days. If you need more capacity, just throw more servers at that"
- "is $24k a month in new servers crap cheap by your measure?"
- "comparatively to the amount of how much money these servers will make the same month, it is crap cheap. It is just a little less than an annual cost of mid-tier software dev in Russian office. We account only 12% increase in our revenue due to algorithmic improvements and almost 80 to more traffic we handle. A new server pays back the same month, and you and other devs pay off only in 2 years"
This pretty much summarises the viewpoint of a typical "big dot com ops manager" on hardware these days
However, there's a class of companies which don't buy servers individually but by the datacenter (or large parts of one).
If your company is building custom ASIC parts, FPGAs or one of the main customers of an entire chip architecture (eg Power9) then the math is different.
This is somewhat true. However, raising gross margin by cutting depreciation is a good thing. And throw in "green" for karma points. Using hardware that takes less space and power with more performance will get most good 'big dot com ops managers' happier.
Except a new architecture brings with it non-trivial software compat and operational overhead. If you are 100% in control of your software the former is manageable, but how many of us really are?
The trick here is to make the new platform attractive enough it survives the stupid test, i.e. I'd be stupid not to try this out! That implies either very cheap or very fast or.. both. I presented on this at Linaro Connect USA in 2014: https://www.slideshare.net/mobile/linaroorg/leg-keynotekiko-...
I wonder if people will actually get documentation for these, unlike many of their other devices. I'd hate to see server kerenels end up as much a wreck as phone kernels.
I can't really go into details because of NDAs (oh the irony) but these machines will boot fully open source and upstream kernels out of the box. We are very much trying to avoid the problems encountered with phone-grade ARM chips.
Will there be datasheets and other programming documentation available? ARM core documentation is, but everything else on the SoC is also important. Just because there is open-source code for it doesn't mean it's been documented, because the code only shows "how" and not "why".
Qualcomm is notoriously closed with docs, about the same as Broadcom. Intel is starting to close up, but is still more open because of the x86/PC legacy.
I don't know (and if I did know, I couldn't say at present).
However the points raised in this subthread are absolutely correct. Having upstream kernel support is vital so that any distro boots on any h/w and so that security fixes can be applied. Having good quality drivers is important, and documentation very helpful too.
Red Hat is taking the high road here and working with server manufacturers to make sure that upstream kernels work painlessly on ARM server hardware. We don't think that ARM in the datacenter can be a success otherwise.
If Qualcomm thinks they can pull the same crap with proprietary modules in the server space and living 'off tree' they're in for a big surprise
Let's see what happens when a sysadmin can't apply a security fix or a needed upgrade to the kernel because Qualcomm's "proprietary driver" has not been updated
In my opinion, the way for ARM to succeed in the server space is:
1) One quarter the price for equivalent performance for server task (e.g. 500 USD for 48-core Qualcomm vs 2000 USD for 16-core (32-thread) Intel). E.g. the problem of 48-core Cavium ARM chip was poor performance, that could be fixed by Qualcomm, with better IPC (big enough L3 cache, and 3 or 4 OooE).
2) One quarter the power usage
3) Reliable and affordable motherboards
4) Devices at similar price vs Intel ecosystem (high performance Ethernet, Infiniband, PCI-E SDD, etc.)
"One quarter the price" ... "One quarter the power usage" ...
The first is very unrealistic because this is a market and ARM chip makers aren't Santa Clause, the second seems very, very unrealistic, because Intel already has the most efficient "big iron"-ish CPUs (just compare SAP benchmarks vs. power usage of Intel-based vs. POWER-based systems, for instance), and have been under continuous pressure for more than a decade to make things more efficient (because energy use and costs following from it are a major cost factor for all of Intel's biggest server customers). That alone doesn't mean that it's not possible to be better, of course, but it's a strong hint that if no one else managed to get close to Intel there that it maybe is a hard thing to pull off. In any case I think a four-fold efficiency increase is expecting way too much.
10 % better power efficiency at the same performance level per unit would already signal a huge win for ARM.
No, actually he is exactly right. 10% better would made no difference to the typical buyer, and to complicate any TCO guesstimate would need to take into account managing a new arch in a deployment, which comes with its own hard to measure overheads.
Define "typical buyer". 10-15% instant energy savings would be one heck of a reason for FB/Google/Amazon/Microsoft to veer towards ARM for their clouds, especially if the vendor can plausibly show that they have more potential there.
These are servers, not hypedisruptionmarkets. Not once where there 400 % efficiency increases in a single generation of anything.
> 10-15% instant energy savings would be one heck of a reason for FB/Google/Amazon/Microsoft to veer towards ARM for their clouds
I'm not sure the upfront effort/cost would be worth it for that savings alone, especially when Intel will just come along and say they'll have that savings in an x86 chip in 9 months without any software dev required.
What this could do for the big DC companies though, is provide them a lever to keep Intel on that path. That lever alone might be worth all the dev effort required to support 2 architectures in a DC for some customers - and even if it's not worth it empirically, it may make them feel like it is.
What modifications are required to be made to the linux kernel/tcp stack to take advantage of 48+ cores, in order to achieve a more linear scalability?
Are there any real-life experiences? Would a different TCP stack such as mTCP[1] suffice?
> What modifications are required to be made to the linux kernel/tcp stack to take advantage of 48+ cores, in order to achieve a more linear scalability?
Yes, if you want to run TCP/IP on it. But if you have bucks for it, it takes just a little bit more cash to get rid of packet IO altogether, and the network stack overhead along with it.
Cavium is about on par with an old e5-2670 that is about $90 on ebay. So while at some future point arm server chips will likely be a good value Cavium's ThunderX is def. not it.
There's no reason to think this will be significantly faster than that. ARM evolved as a low power architecture, back when Intel just didn't have any low power CPUs. ARM is a lot faster these days, but only because architecture complexity, and thus power usage, has increased. There is no free lunch here. ARM isn't super efficient at all, it's just low power.
This chip has applications, but probably not in compute-heavy data centers. Networking gear maybe? Give each port in a switch an ARM core?
I read somewhere that ARM was growing fast because it hadn't been as thoroughly developed as x86 in the past. So right now ARM manufacturers were catching the lowest hanging fruit, so to speak.
Is ARM going to hit a limit soon like x86 has?
Also in the comments of the article someone says that the x86 architecture is outdated vs SOC architecture.
Is this true? Does this mean we'll see SOC on desktop in the near future?
Yes and no. ARM is growing fast, it is growing fast because there are lots of manufactures making ARM cpus.
First there is diversity, while Intel or Atmel might announce 3 new CPUs with 2 or 3 variants each in a year, you have 10 manufacturers each with several new offerings so that it many many different CPUs.
The ARM "tax" (licensing fee) is lower then the minimum Intel "margin" (what they demand over the cost to produce) so the device maker's margins are better.
ARM is more open about its various interconnects so it is easy^h^h^h^hpossible to add features to an ARM CPU in a custom SoC. And to delete features if you don't need them. The Cortex M4 cpus without Floating Point is a good example, try to buy a Pentium processor without a built in FPU.
The thing that kept Intel in the driver's seat was software, and it was why (in my opinion) Itanium never had a chance over AMD64. Software, software, software. And once open source broke the server OS monopoly and smartphones became the computer for the masses, ARM was a good bet.
Is there a CPU saturation limit? That is an interesting question.
Yes, the specs of Centriq and Vulcan are probably similar enough to Skylake that there's no catching up left for them to do. At best they could advance at a similar pace to Intel/IBM.
When it comes to the desktop, Intel is driven more by business concerns than technology. Putting the southbridge on the processor doesn't have that much advantage, but AFAIK Zen can run by itself with no external southbridge to some extent.
Well, Vulcan is now kaput, Seattle seems disowned and X-Gene is up for grabs, so only Qualcomm and Cavium remain as contenders in the server SoC space. I predict they still have a solid shot but it will require a lot of uncomfortable adjustments along the way.
Moore's law states transistor count doubles approx every two years. So if that keeps up either you'll see more shrinking of computer size, utilization of the new transistors per square inch, or both.
On one meeting we had a typical discussion with ops guys:
- "why wouldn't we optimise our hardware utilisation by doing things a, b, and c."
- "hardware is crap cheap these days. If you need more capacity, just throw more servers at that"
- "is $24k a month in new servers crap cheap by your measure?"
- "comparatively to the amount of how much money these servers will make the same month, it is crap cheap. It is just a little less than an annual cost of mid-tier software dev in Russian office. We account only 12% increase in our revenue due to algorithmic improvements and almost 80 to more traffic we handle. A new server pays back the same month, and you and other devs pay off only in 2 years"
This pretty much summarises the viewpoint of a typical "big dot com ops manager" on hardware these days