These big switch boxes typically end up being about thermal management and this box looks like thermal design was an afterthought. Also I'm not sure the power entry design is really all that smart. Generally speaking if you're in the market for a 640Gb/s switch (or 3.8Tb/s switch), does your data center really not have access to 48V power? The AC/DC conversion wastes power and space.
The QSFP's are "spaced for optimal airflow." However this spacing seems to neglect cooling the QSFPs themselves. Belly-to-belly mounting of modules is usually the most thermally challenging way to arrange them. The heat dissipated by the QSFP's is generally directed towards the top of the module. By placing open air channels between modules, they have effectively ensured that little to no air flows over the QSFP heat sinks (which is not shown). So there is probably a limitation on which reach codes are supported. My guess is that because of the thermal limitations of this design, it's not truly non-blocking in all reach configurations.
It's pretty common here (nl), you have to ask for it but it is usually available. I'd advise against it because you're deviating from the bulk but if you need it you can usually get it, especially at the larger colocation facilities.
This is true, however: 48V circuits are generally priced 'per-amp' just like 120V or 240V circuits. That translates into much higher cost-per-watt and with the efficiency of AC switching supplies being well over 90% you're probably better off from an economics point of view of getting the highest voltage supply that you can get and then to step down in your rack mounted devices.
Hey Nolan, I love Cumulus' approach and I bet you guys will go very far!
I thought about adopting Cumulus for a large telco project, but for the moment it doesn't seem a good fit as we do lots of openflow, L3 and custom application development.
We don't do OpenFlow, but L3 and custom apps are our main focus.
That said, often times the subset of things that can be done w/ OpenFlow in actual existing hardware are things we can do natively. Feel free to email me if you want to discuss!
There are 1 gig 48 port switches on Cumulus's HCL, they're around $2500 and $700/yr for Cumulus support. Unlike Cisco or Juniper, you must maintain support in order to continue using the product (legally). I can't imagine getting enough utility from a switch for this to make sense for home use.
It is substantially better pricing than list for comparable Juniper / Cisco / etc. equipment. However, Cumulus has no truly low end 1 gig switch (single power supply, limited L3 capability), and you can absolutely negotiate Juniper / Cisco / etc. down to be close to or even below the pricing of the Cumulus solution. That is tougher for 10 gig or 40 gig equipment, which is where the value proposition kicks in for Cumulus.
Maybe Cumulus's approach is enough of a value add for it to make sense to pay a premium, but everyone I talk to is interested in cost savings first and better manageability a distant second. You'll still have to have Juniper / Cisco / etc. in your life to an extent, Cumulus doesn't do routers and they don't have a full range of switch models.
All of these things are based on the Broadcom Trident II chip. If you want it cheap, don't go to juniper - get one from Quanta. You can have a 32x40G switch for $6,000.
On a similar note, I just recently became aware of Cumulus[1] Debian based switches (but the good bits are closed source) from among others edge-core[2] (via a presentation by PaaS-provider http://zetta.io) -- eg:
To clarify, the only part that is closed is "switchd", which is a userspace program that watches the kernel data structures (route tables, neighbor tables, bridges, ports, vlans, etc) and programs the hardware to match. It links against proprietary silicon vendor SDKs, and programs registers whose description were given to us under NDA.
Without this part, everything works the same, but is of course not hardware accelerated. So the 100% open source parts of Cumulus Linux would still make a great Network OS for a router/switch VM.
We don't yet have an official VM version, but that is something we will have in the future.
What is the flexibility with "open" switches? To get linerate switching, I'm guessing you're still limited by the hardware? Is the benefit that you can more easily setup routing tables (instead of depending on the switch vendor's capabilities), vlans, etc. just by creating them in userspace then pushing them over to the hardware part?
Or can you actually get fairly low level, like implementing your own algorithms for channel bonding? A while back I wanted to do some L7 inspection, but could only get like 10G per server, and we had 40G coming in. EtherChannel didn't acceptably balance out the traffic. Doing so would have required dealing with one of the network processor vendors and all that mess. Would an open switch platform make this a straightforward exercise?
You are limited by the hardware, and what our code supports programming into it.
The big advantages are reusing config management tools like puppet/chef/ansible/etc, and monitoring tools like collectd/graphite/nagios/etc.
Also, it is super easy to run services on the switches. For example, you can easily run isc-dhcpd on each ToR, instead of DHCP relaying back to one mega DHCP server. Distributing services like this scales better, and reduces the blast radius of service failure.
I've been experimenting with the idea of a transparent caching TFTP proxy server running on the top of rack switch, to make PXE scale better to large clusters.
The important thing is that anyone who has the know-how to write a transparent caching TFTP proxy server for Linux can just go ahead and do that on a Cumulus Linux switch! You don't need to come to us and convince us that it is a good idea and then wait for us to actually implement it. Compare that to asking for features from a traditional switch vendor...
We've been loving Cumulus + Quanta for 10Gb and 40Gb, in that it's more manageable than Cisco (for our environment) and a fraction of the cost. We end up using it at 1Gb too, but it's just a price match there, instead of a win.
While I wasn't aware you could get them that cheap, I think I'd still look into second hand infiniband for home use. Allows the use of copper for short (pc-to-pc, point-to-point) distances. While optic is absolutely cool, as far as I can tell a single 1m patch cable will cost ~80 USD -- quite a lot considering the cost of the NIC... And even in IBM's sales brochure for their optical switch, infiband comes out a little ahead:
40GbE uses approximately the same copper cables as Infiniband. There's probably more used IB equipment floating around than used 40GbE, but otherwise I'd go with Ethernet.
Ah, of course. I was misled by wikipedia[1], but it should've been obvious that the same connection could be used for both inifinband and ethernet. So, indeed, there are copper interconnects:
Actually seems the price is finally coming down a bit (compared to what I remember these used to cost, years ago -- but maybe I've just upped my budget ;-).
"The Quad Small Form-factor Pluggable (QSFP) is a compact, hot-pluggable transceiver used for data communications applications. It interfaces networking hardware to a fiber optic cable."
But it also interfaces hardware to copper cables, as I gather.
The cable you're looking for is called a direct connect cable. "Fake" optical modules on both sides, plugs it together with a permanently wired copper cable.
DAC is a common term for this. Direct Attached Copper.
It is a SFP+ plug on each end (but without all the optical magic), connected with twin-ax cable, which is like coax but with 2 signal paths, one for each direction.
This is rather cool, and I'm glad to see some work being done in the networking gear space utilizing open designs and firmware like this. Unfortunately, unless I find myself needing to network a datacenter in the near future, it's not immediately useful to me.
That said, I do hope other developments in network gear that will be useful in other markets emerge from this effort.
From my perspective, there's a gap in network gear between the unmanaged, low port-count switches in plastic enclosures, targeted to home and small office consumers, and the lower tiers of Cisco's catalog, targeted toward top-of-rack or wiring-closet-of-a-larger-building type uses. I would love to see a managed switch with say, 8-24 ports, supporting features such 802.11Q VLANs. I would love to be able segment my network at home so different devices with different performance and security needs aren't all stepping on each other's toes. And I'd like the firmware and hardware designs to be open source, so it can readily patched when bugs are found, and easily adapted to new use cases.
I realize that I'm an outlier and that my needs are not common, or there'd probably be equipment on the market that met them. But it is my hope that as a result of Facebook's work here, and similar efforts, that building such a device will become feasible.
I've the exact same needs, and by occasion got hold of cisco SG300-10 switch, which I think fits the description - so I thought to share my experience, in case it is useful.
I use it for the home lab, where I have a one-armed router serving multiple VLANs, and I have 3x MacMinis running Linux as a "server farm" (The latter I use because they are quite a good gear power-management wise, scaling from ~18W at idle up to ~250W when all cores are busy, and because they are very very quiet, which is handy when the "lab" is next to the bedroom).
The biggest complain I have about this box is that the only way to manage is the Web UI, and especially the 802.1q configuration is a bit unintuitive (though I just learned the firmware is actually upgradable to something with decent IOS CLI, so I will try it out and update here the impressions).
Otherwise, needing just a very simple L2 switching and 802.1q trunking at gigabit speeds, and fanless operation - I am pretty happy with it.
8 ports works well in my setup (the main segmentation/trunking is really in the lab, the rest is either wireless, or directly connected to the "border router").
EDIT: the upgrade to the latest firmware indeed unearthed the checkboxes to enable telnet/ssh, as well as quite a few new features, comparable if not more than the "bigger brothers". What's pleasant is a quite comprehensive IPv6 support.
The specs on this look very nice indeed. I'll be taking a closer look at this.
My main hesitation is the proprietary firmware. Now I'm not going to disagree with anyone arguing that Cisco knows what they're doing and is competent at putting together firmware for the hardware they sell. Nor will I disagree with anyone arguing that open source is not a magical talisman ensuring quality. No, my main concerns are a) timeliness of critical updates, and useful lifetime of the hardware vs support lifetime.
Being open source cannot prevent bugs, but once found, fixes tend to become available quickly. Also, I have found that the useful lifetime of computing and networking hardware tends to exceed the period of time the vendor will offer support for it. I have gigabit ethernet switches I bought years ago that still work just fine, even though they're no longer sold. That's what I love about OpenWRT. The hardware my home router uses is discontinued, but still does the job just fine, and I can still get updates when I need to.
That said, I think I will be checking the SG300 out. Thanks for the recommendation.
I've sent the software download link in the other comment - so far the history shows quite regular software updates for this box. But I share your concern and can not say much about this box besides what I can infer from the software publish history.
+1 on the OpenWRT. Building a custom package that allows you to get a $20 specialised networked appliance is a breeze. That platform absolutely rocks.
It's a normal checkbox in the gui "Enable SSH" / "Enable telnet", so I would suppose it is.
But further tinkering revealed it seems to be a quite-close approximation of IOS, but not the same IOS you'd get on the "older brethen" boxes.
It's about 95% the same, with differences in small details - the format of the output, the behavior on "more" prompt, the look of that prompt, the way the certs are stored, etc.
Nonetheless should be close enough to be usable in a geek home environment.
One caveat I noticed is the ssh seems to not work when connecting from Ubuntu 14.04, works fine from OS X. When I have time, I'll debug it further. (I very rarely do any changes on it, the CLI was more an unexpected bonus I wanted to check out rather than a real need).
I also wanted something in the category you're describing and after shopping around a bit I stumbled across Mikrotik's products. I bought one of their 24 port smart switches and it was exactly what I was looking for. Not a lot more expensive than some of the nicer consumer-targeted gear, but it has way more functionality.
Looks like a very nice product (along with most of the other products). It's a little disappointing that they seem to do only the very minimum wrt the GPL:
I mean, sure, asking for 45 USD for a CD with the source is technically complying with the GPL -- but it does seem a bit strange in this day and age. Not to mention that for the source code to be useful, one would hope one could build a working routerOS image from it -- and it doesn't appear that the CD will enable a user to build a running image -- and therefore not facilitate changing the product.
But apart from that, my first thought was -- can this thing run BSD -- because pf is quite a bit friendlier than iptables (even if the latter has gotten a lot better lately).
Does anyone know if there are any recommended alternatives to soekris for running a bsd switch/router (preferably running at ~gigabit speeds) ?
Not sure if it can run BSD proper. It should be able to in theory, there's nothing preventing you from flashing your own OS image, but I can't find that anybody has actually done it. There is a facility for virtualization as well. Not sure how desirable it'd be to run on bare metal though, as it uses dedicated hardware for routing that I think is proprietary, so it wouldn't work. If I was going to try it, I'd probably pick up one of their cheaper products first to test it out on (most of the lower end stuff is similar internally).
I'm with you about the GPL though. It's my biggest complaint about it. Apart from the proprietary hardware (which OK, I guess I can forgive it), they definitely seem to be playing it pretty loose with the terms of the GPL.
Pf is indeed nice and it'd be cool to run OpenBSD on it. That said, as I mentioned I really like their configuration tools and they make iptables actually quite easy to configure.
Your best bet for a BSD router is probably to pick up a cheapish computer and put in one of the Intel or HP quad-port ethernet cards and then plug into a dumb switch. They can be had for surprisingly reasonable prices, I saw some on Amazon for ~$80 (the quad-port cards, that is).
At home, I replaced my Cisco ASA 5505 (which replaced a Cisco 1811) with a RouterMaxx 1106 [0] running OpenBSD (from CompactFlash), though it's certainly not cheap.
That's the exact one I have. It's definitely not a "plug and play" router, but I love it. My only real beef is that most everything on it requires a much larger level of network administration knowledge than any other hardware.
I'm also scared that I'm going to open up my home intranet to the world every time I tweak some of the advanced settings.
Not trying to dissuade anyone from using them, but it's light years away from a DD-WRT-based router.
OTOH, the ability to run VMs on your router is magical. I have yet to do that, but I'm itching to do something like putting the unifi management tools on it.
Yeah, it definitely isn't something I'd recommend for people who don't know what they are doing. It's feature set is more comparable to enterprise hardware like Cisco or Juniper. Their terminal configuration utilities are pretty easy to use, but I actually really like their webui. It exposes all the functionality (which is a daunting amount) but is clean and very responsive.
That said, RouterOS pretty well documented and isn't hard as long as you know the basics. Mikrotik actually has a pretty vibrant little community around their wiki.
Yeah it can. The firewall is standard iptables, so you can do whatever you want. It's also got DNS, http cache, SOCKS proxy and approximately 1 trillion other things that you may find handy as an all-in-one home/small business network device. Needless to say, it's maybe not the best idea security-wise to run all that stuff on one device, but I'm not terribly concerned.
I've heard people say it's not really powerful enough CPU wise to cope with a ton of rules (>hundreds), but I have a fairly involved firewall config on mine and it's no trouble.
Funny story, we use a firewall rule to punish roommates for not doing their share of the chores. If they get too far overdue on chores, we have an iptables rule to randomly drop a certain percentage of packets to their machine.
There are some good cheap managed switches out there. I have a Linksys SFE2000-something 24 port PoE switch for my PoE devices (security cams, IP phone, other stuff) and a TP-Link [something] 16 port switch. Both support basic VLANs, SNMP, etc.
I have 1 2port VLAN that connects my cable modem to my router on the TP-Link. From that switch I have 1 port cables to a machine with wireshark, and I can configure that port to monitor various VLANs for whatever reason.
I run MRTG for several things, including basic traffic graphing. I display the graphs from the router uplink port and a couple of other key ports in a window on the VMS that also has security cameras on it. From that monitor I can keep on eye on key things (cameras, Internet I/O, some home automation stuff).
Anyway, I haven't found an affordable "perfect" switch for home stuff, but there are a lot of cheap, decent managed switches that give you a lot more flexibility beyond "everything on 1 network".
My home net is essentially segmented into Primary LAN, Security Devices, Guest LAN (mostly just a wifi bridge) and LAB LAN.
Have you found an OpenWRT firmware equivalent for these devices? A lack of an open-source firmware for such a switch has been one of my hangups. I'm leery of the firmware that would come with such a device, especially given the history of the firmware of home routers being implemented poorly (bufferbloat, obsolete versions of software with known vulnerabilities, a lack of upgrades available from the vendor because they've EOL-ed support on the device to make room for the new shiny version -- which is also shipping with vulnerabilities).
I think what would be ideal is something in the vein of the Linksys SFE2000, with an open-source firmware akin to OpenWRT. AFAIK, that, or something approaching that isn't available. However, I would love to be wrong in that regard.
Got a used 24 port 2960 on ebay 10/100 ports for ~$65. Maybe doesn't mean the "open" portion, but the rest is definitely satisfied. Also, bought it 5~ years ago and still runs beautifully (Noisy fan though).
This isn't really suitable for the purposes you're describing, but it has a niche for firewalls and routers that may interest someone.
I've been using PC-Engines Alix[1] boards for this purpose since 2008. They are 1-3 ports depending on model, but with VLAN tagging a few ports can do a lot of routing or other traffic processing. I run OpenBSD on mine, but they should work well with anything that can run headless.
Wow. That switching fabric is so much smaller than the HIPPI crossbar backplanes we used in the 100baseT/pre-GigE switches that I used to write firmware for... almost 20 years ago now. Of course, complexity means higher cost, so it's obvious why nobody bought our stuff. Hopefully, this open hardware project will have better luck!
/* it's easy to forget just how many iterations of Moore's Law have happened since the mid-90s */
I get the feeling this is a bit like Bugatti building the Veyron, its super cool and impressive but the vast majority of us will never be able to or have a need to drive it.
It is still fun watch this switch go around the proverbial track, but I'm happy knowing that I'll never have to configure, build and test a switch of this complexity unless I really absolutely have to, with my largest caveat being AWS disappearing from the face of the earth.
You're not the target market. You shouldn't be making an analogy with a Veyron, which remains a luxury car for an individual. You should rather be comparing it with a firetruck, a vehicle custom-built for a purpose you as an individual you will never need.
I currently work in the network hardware industry (think Cisco, Juniper, etc). Our boxes sell in the 6-figure price range, each [1]. We sell to your ISP, wireless carrier, datacenter constructor. We're the competition this kind of box is aiming at.
[1] it's highly specific hardware and software for a low-volume market. Individual chips used in the hardware can cost multiple thousands of dollars each.
Good points. I doubt if they are aiming at any competition. Their stated goal is to reduce the cost of the hardware for themselves by standardizing it. Their hope is that the other big guys (amazon, google etc) will pick this solution up driving up adoption and lowering the cost. If this happens, Cisco, Juniper et al gets disrupted and that's just a side effect.
Right, but we don't care about past chips. Moore's law applies to the whole market at the same time, so today's special network chips are still much better at what they do than today's general-purpose chips.
It's kind of stupid to say that today's general-purpose hardware can do what yesterday's special hardware could do at a faction of the cost, because the specialist market has also moved on and wants the performance of today's special hardware.
One of the great things about computing is that stuff which was insanely high-end N years ago is high-end of affordable now and will be commodity in another N years.
And N years after that you get that functionality for free along with your new TV.
I'm not entirely sure what I'd do with 1.28 Tb/s of switching power.
Though, 30 years ago, in Neuromancer, William Gibson wrote about the city of Los Angeles transferring megabytes (yes, whole megabytes) of data each second, so he probably couldn't have imagined what someone would do with the gigabit switch on my desk.
If the people who brought me HDCP[1] are the same people that will give me a networking switch inside my television, maybe I will consider using this Facebook switch after all.
The only datapoint I have is that it's in the SF bay area, since I am acquainted with someone who got poached from Amazon specifically to work on this and had to relocate from Seattle to there.
Thanks and sent you an email.
For anybody who is looking for similar details on the team who are behind these products, just FYI - there is not much info on FB site or on the internet.
If I get any info later, will report back.
I just want a fully open source switch that's ready for enterprise, from the h/w to s/w. Linux preferred but bsd based is acceptable.
I've been a "cisco guy" since 2001 or so, but I am so tired of them. Licensing fees kill budgets that could be used on other things, and you end up surrounded by consultants that only ever touch Lozedoze systems insisting that "nobody ever got fired for buying Cisco". Smartnet is a must for some equipment, yes, but I'm so ready for a paradigm shift in networking.
I've really been watching ubiquiti and their switching/routing products, they seem very promising but not quite prod ready. I am impressed with Dells open switches too.
Edit: The Microtik stuff being linked elsewhere here is looking pretty awesome too. Not quite FOSS but still.
As someone who runs a 6509 (almost fully populated with line cards and specialized controllers) as my HOME core switch (with a second one on the way for full redundancy), an open modular switch is very cool.
(I'm currently hacking on OpenFlow -> NETCONF bridging, to bring typical SDN capabilities to legacy Cisco environments). Hence my rather..... extensive home network.
This is very similar to what Cisco is doing with Nexus and the "fabric extender" TOR replacement kit. I'll have to see if Facebook has any of this stuff in GIT and stand it up in a VM environment and play with it (I already do a bunch of OpenFlow stuff on OpenWRT and am looking at implementing an open southbound API on FPGA on the parallela board).
Good thing it is not targeted at people looking to buy a 4x1G NIC, but at people who want a nicely integrated FPGA dev plattform with 4x1G and fast connection to a PC.
FWIW, all Arista hardware is AWESOME. They run a lightly modified very minimal linux operating systems that is very clearly (not even really hidden) Fedora. I got the privilege / opportunity to reverse engineer one of these to see if it was hackable (ie: add custom hardware into them or run apps directly on the switches).
Pretty much the entirety of the Arista userspace is python with all of the magic really in their ASICS. Truly, I've not seen many better engineered pieces of equipment. At the time (I did this > 4 years ago), the Arista switches had dual core AMD Athlons with a few G of ram. They were willing to put more RAM in one, or let us upgrade it. For a so called "vendor product" Arista clearly gets it. They have some really solid Linux guys working for them. When you get the pleasure of using their python apis or pull one apart, it shows that they know what they're doing. It isn't completely insane to think of running apps directly on the switches when they are simply x86_64 Fedora boxes with some fancy asics and a lot of interfaces.
1) low individual component cost (which is great when you need 100's of them for a datacenter build out)
2) no frills, high throughput non-blocking backplane which are normally only available from the enterprise grade network vendors at top dollar per individual component, and come with a mountain of features and bugs you DONT NEED
3) software defined networking stack: if you can imagine a scenario/feature that would improve your life, nothing but development costs will get in your way. contrast with enterprise vendors, which requires explaining the problem to begin with, and dangling a wad of money explaining how it will be worth their while to develop it
This looks great. They also have the advantage that James Hamilton mentioned AMZ got from building their own hardware - the get to do a cleansheet s/w design and not have to support all the gazillion options that Cisco does.
Am I the only one who read the title and thought it referred to a 'pushbutton' style electronics hardware switch? I would have been quite excited to see that!
The QSFP's are "spaced for optimal airflow." However this spacing seems to neglect cooling the QSFPs themselves. Belly-to-belly mounting of modules is usually the most thermally challenging way to arrange them. The heat dissipated by the QSFP's is generally directed towards the top of the module. By placing open air channels between modules, they have effectively ensured that little to no air flows over the QSFP heat sinks (which is not shown). So there is probably a limitation on which reach codes are supported. My guess is that because of the thermal limitations of this design, it's not truly non-blocking in all reach configurations.