I love reading articles like this:
"So, for example, a cache miss on Jesper's 3GHz processor takes about 32ns to resolve. It thus only takes two misses to wipe out the entire time budget for processing a packet."
Then I go back to adding another layer of abstraction to my bloated Java code and die a little inside.
Actually guys like him are making our life easier and so we don't have to worry about a lot of stuff like this. Abstractions are a necessity, don't feel bad. There is no way a small team using Java/Scala can accomplish what they usually do in the time frame they do it, without those abstractions. Computers are there to ease our lives after all.
Indeed, there has been a large amount of "bypass the kernel" campaigns in the last couple of months. Robert Graham's 2013 Shmoocon talk has a great introduction into the whys and hows of this movement.
Facebook had a job posting that showed up on HN for a position to help speed up Linux's networking stack. While I doubt these improvements will surpass the kernel bypassing model, I'm glad a developer has decided to tackle this head on and help overall efficiency.
Look no further than Solar Flare's Open Onload[1], Mellanox's VMA[2], Myricom's DBL[3], Intel's DPDK[4], etc to see that kernel bypass tech is used in 10G+ networking all the time.
It is super fast, but also lower latency. Many of the features used in lower latency environments like busy polling have been slowly making their way upstream[5]. I think Linux will get pretty close to the userspace kernel bypass stuff, and this is coming from a guy who works in a low latency environment for $REAL_JOB.
There is a lot more than bypass the kernel here. You need special dma hardware that will preload cpu cache lines and stuff like that. Maybe even specialized cpu cache that can be pinned
TOE isn't supported under Linux and it was only a win if your TCP stack was slow (and by "your" I mean Windows). TSO is enabled by default in Linux and is indeed an easy win.
Too bad improvements in network technology haven't found their way to consumer level. It's probably related to the stagnant broadband speeds, last mile bandwidth improvements slowed to a crawl many years ago and concurrently device connectivity actually moved to a slower networking tech, wi-fi. Now people are happy to use wi-fi for home desktop computers since their last mile net connection is so slow anyway.
It's 10 years since motherboard integrated 1G became commonplace in regular PCs, same for 10G is nowhere in sight...
10 gig still isn't even commonly available on server motherboards, because of power / space / cost. There also aren't many copper 10 gig top-of-rack switches, just the Cisco Nexus 3064T and Arista 7050T come to mind. Juniper doesn't even have one.
It's easier for a lot of places to use twinax with 10 gig SFP+ switches rather than going copper 10 gig. That is definitely not going to trickle down to the consumer level.
It will probably be another 1-2 years before 10 gig is ubiquitous at the server level, and another 2-3 years after that before it is commonly on consumer equipment. Or maybe it never will be, and things will go in another direction.
> There also aren't many copper 10 gig top-of-rack switches, just the Cisco Nexus 3064T and Arista 7050T come to mind. Juniper doesn't even have one.
I might be missing your definition of tor. The juniper QFX5100 series has the 48T which does 48x 10GBASE-T plus 6x QSFP. the 5100-96S does 96x SFP+ and 8x QSFP. There are plenty of other cheap merchant silicon platforms that look similar. Personlly Im happy with DAC on SFP+ ports.
Oh yeah, I always forget about the QFX series. Seems like that would do the job.
List prices are utter nonsense for switches, but the QFX does come in above the other two I mentioned. Perhaps because of its fibre-channely nature that no one (I hope) cares about.
Still, any of these would work if you get the right deal. I could see an advantage for 10g copper if I had mixed racks where not all of my hosts needed 10g on the server side, but that's a big premium to pay over 1g TOR if you aren't using lots of 10 gig ports.
For me, I just use copper on 1 gig racks and DAC on 10 gig racks. So far, so good.
Last mile speeds for end-users aren't that important, most networking interaction they require goes to the internet anyway. Internet providers in the USA trying to push 5Mbit to be classified as "broadband" should tell you enough... I'm used to full speed 100Mbit internet (actually 160, but my router only has a 100Mbit connection), but for average useage - the difference between 50 and 100Mbit is rarely noticeable - unless you download big files.
Once 4k or 8k streaming becomes more common, sure then you'd like to have full Gbit where max bitrate would be 340Mbit - but for even decent full-hd streaming - nobody needs more than 50Mbit (in real throughput that is).
Higher speed things are more important on servers that have to handle more and more connections, and the end of this horizontal expansion is not in sight. Everybody starts using the internet for more and more things, and this all has to end up on servers that have to handle connections from millions of clients.
10 gigabit over Cat6A cable is also power hungry and very tricky: It's approaching the limits of what you can do over that sort of cable. Plus the frequencies involved make the adapter design a lot more troublesome. I suppose we'll start seeing cheap 10 gig knockoff nics eventually, but it's not even everywhere in the datacenter, never mind the home.
yep. Fiber is the way to go when you need 10Gbit. I was looking in to doing my place with Cat6 or 6a. Not only is 6a incredibly thick (thus, more difficult to run and probably terminate) but you need special tools and knowledge just to make sure the wiring is correct and there are no RF issues going on. It becomes way more about RF crosstalk and science, and less about simple wire connectivity.
Yeah, they exist in a select few workstation class Xeon boards. Intel announced those (server-targeted) chips in 2012, they take time to show up even in this high end niche. As there haven't been any announcements about mass market parts, it would be a small miracle if they showed up this year.
So, a local ISP has recently released 10Gbit fiber to home (for a tiny subset of the City).
Would any consumers be able to use this, or would it only make sense with a commercial switch (probably for a shared connection for an apartment complex or similar?)?
What would I have to do at home to take advantage of it?
Yea, with the speed of SSDs increasing so quickly the speed at which a desktop computer can transfer a file and store it has greatly exceeded what gigabit can provide. At the SME level 10G is still too expensive to justify at this point for regular workstations.
Am I the only one that thinks we need to start at re-evaluation of BSD sockets first? I know Apple tried and gave up, but it just seems like so many of the building block pieces we use everyday could really use a major polish or some good competition.
This article is mostly about the lower parts of network stack like QoS and talking to the NIC; the user API is kind of orthogonal but equally important. There have been several research projects about improved networking APIs; my favorite is IX which gets line-rate performance while retaining kernel/user protection. https://www.usenix.org/conference/osdi14/technical-sessions/...
The first relevant-seeming presentation is from 2009[1] and doesn't really get into the pitfalls of low-latency switching/handling like this article does.
I'm definitely interested to see how other operating systems handle this, though. In particular: Windows (is networking in user-mode?) and Solaris-likes.
Why are we still using 1500 byte packets at 100G again? Seems like there won't be any tricks left to make 1000G work. Does that count as technical debt?
Pretty much everything has supported 9k jumbos for over a decade. The internets mostly 1.5k MTU, but you normally arent doing multi gigabit streams over public connectivity. The other argument is TSO. Your kernels probably writing a 64K "packet" to the NIC driver. When segmentation etc is handled by the hardware why do you care about the MSS? On the network device side the SerDes are the issue. And were already running parallel lanes there; 40 is 4x 10 lanes and 100 is 4x 25 lanes. Why not 10x 100 in a couple years?
Ok... So looking at IFG as 96 bits or 12 bytes of "overhead" thats 0.8% or 0.13% for 1.5k and 9k frames. Why do I care about 0.67% of throughput? And pretty much all silicon in over a decade does line rate at 1k anyways. Or if its latency a hypothetical higher clocked lane would be something like 1ns instead of ~3ns per frame? Thats the difference between 2 clock cycles and 6 cycles of latency.
So what is your shorter/faster ifg buying in practice.
I probably couldn't improve the actual code myself, but I do understand what is being talked about conceptually. I enjoyed the article and learned a bit from it, thus I upvoted.
40G has been out for a few years but it's fairly expensive since it uses four lanes. 25G will be the best option if you need something faster than 10G IMO.
To expand on that: On a switch chip today, like the common Trident 2, you have 10 and 40 gig interfaces. The 40 gig are just four lanes bonded together (10 being one). These 25 gig products runs each lane at 25G instead of 10, so you get a 25G port in the same density you used to have 10, 50G at double the density of 40, and 100gbit/s at the old 40 gig density.
I think this is largely being driven by the server folk, who want to connect at 25G instead of 10.
And to expand on that slightly, it is not just about density, but also latency. Since 40G is clocked the same as 10G (as it is 4x 10G lanes), upgrading from 10G infrastructure to 40G will not improve overall latency Whereas 25G is clocked faster and will have lower overall latency. [Overall meaning the first packet won't arrive relatively sooner, but the second one will.]
There may be latency improvements inherent from design improvements within newer packet processing / switching ASICs themselves, just like Intel's tick/tock. For example, NASDAQ offers a 10G handoff to their 40G infrastructure which is 5-9us faster because of topology and equipment upgrades [1].
The nice thing about 40G is that it is compatible with 10G so you can selectively upgrade your switches to 40G but keep your 10G NICs, using these neato QSFP+->SFP+ breakout cables [2].
The atom server boards released last year actually have 4x 2.5g lanes. As far as ive seen everyone just uses 4x 1g serdes on them instead of the hybrid 10.
Then I go back to adding another layer of abstraction to my bloated Java code and die a little inside.