For anyone interested in this kind of thing, I am designing a board to hold two Raspberry Pi CM4 modules connected to a gigabit switch with 3 extra RJ45 jacks.
My hope is to make a somewhat minimal board for CM4 clustering. Eventually I will make a version with 4 CM4s and a single RJ45 jack, but I started with two CM4s and three jacks so I could test the ethernet switch independently of a CM4. And it turns out this board has some interesting uses too. I will break out both PCIe ports for the CM4s, so this board could have two intel wifi adapters plus three ethernet ports. It seems like that would make it a useful general purpose network thing. For example you could put a long range parabolic antenna on one wifi adapter and an omni antenna on the other and make it into a remote network access point (like a Ubiquiti Nanobeam connected to an AP). It should support VLANs etc as I will connect to the management interface on the switch via SPI or I2C.
However it is one of many projects I am working on, so I can't give a firm completion date. Microchip tells me they will ship me the ICs I need in a month, so it would be nice to have layout done by then. But we'll see.
Don't forget to check the errata for that switch and apply the recommended workarounds. That fixed some link stability issues I witnessed on boards with KSZ9567.
The comments on that Banana Pi board point that the SoC doesn't have any network specific features i.e. HW NAT, WiFi offloading and so it would be bad for a router.
That seems like a reasonable criticism and if so it could apply to RPi CM4 as well, AFAIK they don't have network specific accelerators. Are you planning to address this with some additional co-processors?
If you end-up adding network specific accelerators, Please do work with OpenWrt team as lack of HW accelerator support (WiFi offloading) unlike proprietary firmware is a major pain point.
> The comments on that Banana Pi board point that the SoC doesn't have any network specific features i.e. HW NAT, WiFi offloading and so it would be bad for a router.
Does this really matter? It only has 1Gbps Ethernet, so shouldnt software NAT be fine? I have a significantly less powerful router with no hardware acceleration, and its never been an issue to get maximum speed, though my connection is only 100Mbit (over-provisioned to be about 20% faster).
I've seen the lack of HW NAT impacting performance of Wireless Network over Wired on some aftermarket firmwares for routers, OpenWrt does a decent job of software offloading but as with any SW acceleration the price is paid by the CPU.
Even with OpenWrt there are forks which can make use of HW NAT and give proprietary firmware level performance but are closed source!
Ah interesting. I don't see mention of NAT or address translation at all in the datasheet for the switch I am using. Since the original plan was just to mate CM4s with a switch I hadn't considered any additional network hardware.
This seems like a Rev2 thing. I could get the basic system up and running without additional complexity. But I really like the idea of forks of my work so once I've proven the basic layout I'd be happy to collaborate with the community on improvements.
Do you have any more information on that type of hardware or suggestions on how to learn more? Maybe I should get on the OpenWRT mailing list and introduce myself.
I'm going to be making a video about how I turned an Asus PN50 into an AP using Linux NAT but tbh the performance is pretty bad (but good enough for my home office)... It's also possible I'm configuring it wrongly. The general principles should be the same as RPi and even the net configuration is likely to be identical.
Rpis are notorious for just dying though. You may want to consider something more robust. Don't know if it's SSD corruption or what.
My understanding is that the issues with pi's dying is related to SD cards. But the Raspberry Pi Compute Module 4 (CM4) uses industrial eMMC memory which should have a much better lifetime.
> Maybe I should get on the OpenWRT mailing list and introduce myself.
I agree. OpenWrt's HW NAT support seems to be limited to mt7621 (i.e. At least upstream version).
Unless the SoC is purpose built for network computation I don't think they feature HW NAT, But open-source drivers for the SoC's which has HW acceleration is a bigger problem.
I think aiming for a good SW offloading is a more reasonable goal to start with, Which OpenWrt seems to be doing with MIPS based CPUs.
Interesting. The mt7621 is significantly more advanced than the basic switch chip I am using so far. I will continue with my current design and still try to reach out to the OpenWrt community for comment.
The CM3s and CM4s are so hard to find. The regular pis are still available, albeit at a higher price. Given that the CMs were intended for industrial/commercial integration, it seems like they would have a priority. They are mostly the same components.
Well the big issue with the CM4 is that there are 32 possible variants. As far as I can tell, a couple CM4 variants tend to be in stock at least in the USA last I checked. Only seeing one variant in stock right now at PiShop but it's there:
The thing about industrial customers is that they are willing to buy whatever variant is available now for testing, spend a year doing design and integration, and then switch to the correct module for final system tests. I'm doing this with a project at work right now.
I could speculate that some hobbyist customers who might buy a raspberry pi would just as soon buy something else if they're just trying to learn electronics. So I would think commercial customers would be more resilient to delays than the hobby customers (as long as one or two variants are in stock to test).
And releasing 32 different variants of a new design during a global chip shortage seems reason enough for delays.
Interesting. You mean support the use of the SYNC_IN and SYNC_OUT pins? I could break those out. Looks like they are still working on support, but I could break out the pins.
https://github.com/raspberrypi/linux/issues/4151
I own 2 of the original version of this board. Banana pi has a small community and little support. I was suckered in by the good specs but it's a serious hassle to even boot and documentation is just plain bad. Banana pi boards are for the very experienced soc developer. Limited OS, limited documentation, limited support. Limited success.
I've been running OpenWrt on my original r2 as my primary router and it has been rock solid. (both snapshot builds and now builds from the 21.02 branch). The default builds for the r2 available for download from OpenWrt are very bare bones so I don't recommend that route.
The onboard mediatek wifi isn't usable but using a mini pcie ath10k supported wifi module has worked great.
It definitely wasn't easy initially tracking down info for getting things working. No argument there.
I'd be happy to share my OpenWrt build config for the r2 for anyone interested.
Same here. Except I'd also like to add in the extra problem of "stability" - or lack thereof.
I had them on ethernet. Disabled wifi, verified they're getting enough power. And between 18-30 hours of operation, all the nodes will go unresponsive. A serial cable shows the machines up, but they lose all network connectivity.
There's nothing in dmesg. The networks are just dead.
It's not worth enough for me to troubleshoot them. They're terrible hardware platforms, badly supported, and garbage to use.
the major problem with most of these devices is lack of operating system support.
at least raspbian is very close to stock debian.
if trying to DIY your own router and you are willing to spend just a little bit more money, you can save a great deal of hassle with non-standard weird SOC-vendor-specific linux problems...
use an ordinary small x86-64 motherboard, put a 4-port intel 1000BaseT card in it, and run regular debian, or pfsense, or openbsd or something.
I'm curious, why use an unofficial Debian build maintained by one developer[1] rather than Raspian?
Not knocking the Debian build, just wondering.
[1]: "This site is not an official Debian project. While the maintainer (Gunnar Wolf) is a Debian Developer, content herein provided should be considered unofficial. "
Raspian originally was a custom build of Debian to target the unusual hardware support of the 1st gen Pi (it had hardware float but not other bits needed for the armhf architecture, e.g. like having SSE but no PAE on x86, I can't remember the details though). Newer Pis are more standard (see "Are the images 32 or 64 bits?" at https://raspi.debian.net/faq/), and so Raspian isn't needed for them. So that's the difference between Debian and Raspian (there's likely other differences with extra things the RPi foundation has added, but YMMV if you find them useful or not).
For SoCs, you can either do the installation on the device (which can be painful), or build a chroot with the needed features and flash it to hardware (e.g. sdcard or some board storage), https://raspi.debian.net/ is providing pre-built chroots for you to flash (but if you wanted, you could use https://salsa.debian.org/raspi-team/image-specs to build/customise your own). There's lots of similar projects which build a Debian chroot for a specific SoC, either with custom kernels or with a specific feature-set (e.g. freedombox: https://www.freedomboxfoundation.org/). None of these are "official" Debian projects (I think debian installer a.k.a. d-i is the only official/supported way of doing a install), but many are run by Debian Developers scratching a specific itch.
FYI, if you look at https://salsa.debian.org/raspi-team/image-specs/-/commits/ma..., there appears to be a fair few people who've made changes, and a least a few of them I recognise as being DDs, so it's not just one person (Gunnar Wolf is probably the person the name's been allocated to, and he probably wrote the docs). If it's security support you're thinking about, I'd put those images as more likely to have a fix early than Raspian (as those images are stock Debian, not rebuilds).
After several tries, I just gave mine away, for the same reasons. There are so many boards that are a pleasure to work with, but this one was not worth the hassle.
I can recommend the replacement, a PC engines board with four Gigabit Lan ports running OpenWRT.
That's right, the Realtek RTL8367RB-VB-CG is an ethernet switch chip, and so it certainly will allow for full speed switching, entirely internally.
It's not nearly as rare as all that, your basic OpenWRT-running 5-port router has the same setup. But those cheap SOHO routers aren't going to do really fast routing, because they typically come with very slow CPUs.
This board's Rockchip RK3568 looks pretty capable though.
> It's not nearly as rare as all that, your basic OpenWRT-running 5-port router has the same setup.
IME, this kind of divided switching/routing setup became ubiquitous after gigabit became widely available. I remember that while buying my previous router (Cisco Linksys E4200), this architecture was advertised explicitly, and it was exclusive to E4200 in that generation of Cisco/Linksys routers.
The main firmware of E4200 was prone to crashing under some circumstances, so having a good network sans DNS was a good trade-off back then.
Both specs for MicroTik and Ubiquiti EdgerouterX are much inferior to BPI-R2 Pro, for example EdgerouterX (currently out of stock) only supports 256MB RAM while the latter can support up to 4GB RAM.
Another is that I believe RPI-R2 can support the mainline Linux kernel with normal distributions (e.g. Ubuntu), do MikroTik and EdgerouterX support that as well in addition to OpenWRT?
Are additional RAM gigabytes anything important for a switch? Especially one that delivers on the parameters?
Does BPI-R2 actually supports those distributions, or rather those distributions "can run" on that hardware? That's a difference. OpenWRT can be run on Mikrotik, yet Mikrotik doesn't support that. OpenBSD runs on Edgerouter, yet I doubt they support that. Will anyone that I will buy BPI-R2 from, fix my "normal distribution" if it fails to detect some hardware, or with a failed upgrade?
It's an awesome device but I think we are confusing a few important things here. Like the word "support" for a start.
Broadly speaking, I would not rely too much on support from manufacturers but rather by the communities when the projects are stable enough. Manufacturers have the tendency to slow or halt support in a few years because it costs them money, or to push newer products, while communities support usually lasts much longer.
For example, the NanoPi NEO Core official images all still use a 4.14 kernel (2017-2018), while the ones at Armbian use the current mainline 5.10.
> Are additional RAM gigabytes anything important for a switch? Especially one that delivers on the parameters?
For a switch, probably not. But it does open up a bunch of possibilities for some constrained environments.
I would use such a thing combined with a few other services aside from pure network switching. Off the top of my head, I'm thinking running pi-hole and Home Assistant.
Why? Because I live in a studio apartment, and it would be a single, small and quiet box. I'm currently doing this on a mini-ITX J3455, and while the setup is broadly OK, I'd love to be able to remove the switch I have lying around. Also, the integrated Realtek controllers suck, and I can't replace them with anything better (there's no PCIe port).
Of course, your other points absolutely matter, and are the reason why I'm still looking around for quiet x86 machines that I could expand.
I didn't quite determine if you have a solution or are looking for a solution, but a Raspberry Pi 4, 4GB with a passive fanless heatsink case can run Pi-Hole and Home Assistant without breaking a sweat. I use a FLIRC case.
I also want to do some routing on a gigabit link, so the RPI doesn't scratch that itch. I'm also not comfortable with the various horror stories I've heard about SD cards dying in Pis (although I've run one with home assistant for several years with no issues).
I'd also like to avoid running external drives, the goal being to have a self-contained unit. I'd rather have a larger unit instead of multiple small ones that require cabling (mostly for esthetic reasons) and various dongles.
Glad to hear this. I went checking [0], and there's still a speed bump for me: mine is the PoE version, and its switch ports are not supported yet. The PoE version is also an orphan with respect to OpenWRT.
If switch and PoE support can be worked out, that may a way forward for my router.
Yup. There's a lot of hardware offloading going on inside when running the stock firmware. Something like this board, or a dual-NIC mini-PC, and a surplus PoE switch, might be a better choice.
It won’t help with switching, but it is good for routing/fw + running real software.
I just run a Netgate appliance for that plus something less flaky than OpenWRT.
I also wonder what actual speeds you can get out of these NICs. So many cheap gigabit NICs aren’t really up to the task. And I’d guess that the switch plane runs at 1 gigabit total here.
Edgerouters run Debian, and you can install Linux or even BSD on them. But they have very slow CPUs and use a specialised NPU to handle forwarding of traffic, so if you do not run their OS its going to be very slow. This is the cheapest model, but the lite version can handle over a million packets per second, which requires a beefy CPU.
This! Having been down the home-rolled router rathole several times with PFSense etc, you often need a vastly more powerful generic CPU to match the performance of the dedicated NPUs in even entry-level dedicated networking gear, something I don't think people starting out at this often realize. This is why Ubiquiti and co get away with seemingly absurdly low RAM/CPU specs on devices like the Dream Machines etc.
I'd also remind anyone considering this that your family will probably not thank you for any outages you introduce to their browsing. My LAN is something that has to "just work" all the time and home-brew solutions will often require you to do more update/maintenance tasks than an off the shelf Mikrotik/Ubiquiti/<enterprise style vendor> solution.
You can still add neat features to your network like VPNs, PiHole, encrypted DNS, etc etc without replacing the core router or switch - I use a device much like this Banana Pi to run those services in some containers alongside my <enterprise style vendor> router/switch- I get all the cool new features I want; family get an internet connection that's always on and "just works" thanks to me not having to maintain the core router.
I am running Mikrotik hAP AC aka MikroTik RouterBOARD RB962UiGS-5HacT2HnT using OpenWRT on the home network with 700..900Mbps speeds already. Mikrotik has hardware switching and OpenWRT can use it.
Does that imply you cannot use this for a firewall solution? If it's a totally separate switch, wouldn't you be better off with a RPi and a cheap switch?
So far my experience with Banana Pi products has been of great hardware, at great price point, but miserable software to the point of frustration. The best help always comes from good samaritans on GH releasing experimental builds that somehow work, but you have to actively look for them. Anyone with better luck than me has pointers to solid B-PI resources?
This has been the same story with all ARM boards for at least a decade. They have promising specs, atrocious linux support, and it never improves no matter what. Get a x86 APU2 and just run mainline linux on it, they're the same price as the ARM board but have perfect compatibility with everything.
To be fair it mostly looks that way because ARM has ruled the market for at least a decade, and there's just so many of them. But yes, many vendors make no effort to mainline their code.
But some do, or pay those samaritans to, and as a result you can definitely get boards (including this kind of 5-port switches) that run completely supported on a mainline kernel.
My home gateway is a Turris Omnia, supported in mainline for years. I run a plain Debian on it, works great.
I have 3 Olimexino A20 Boards running on Debian stable without any external repos/software.
All hardware for an audio/web/django/email/xmpp/backup server is running fine for years.
I've been using one of these to run pfSense (and recently OPNSense) for years now. Has given me zero issues. Also easily handles acting as a VPN server. Less powerful devices tend to be bottlenecked on how fast the VPN connection is by the speed of the processor (I think AES-NI is helping a lot here).
There are quite a few on ebay if you're desperate for one. Probably won't find the newer 4-NIC models, though.
Ebay or a licensed reseller (if there are any) should be easier anyways. I remember them requiring a VAT ID for EU customers. I ordered a couple for a small business so that was no problem back then but private individuals are pretty much out of luck.
I'm all for having more choice, but you can get complete Intel Atom systems with Intel NICs, case, power supply, passive cooling etc. for not much more than this board.
With multiple NICs? Do you have a link? I've been looking around for such a thing for a while now, but the ones I see tend to be quite expensive (~500 €), and it's not always clear they have Intel NICs).
In my case, the best compromise I've found, is to accept having a bigger a computer and recycle some older EliteDesk / OptiPlex or similar, which usually are very quiet and can take PCIe network cards. The thinner ones use low-profile cards, which can accommodate 4 ports.
Look for Eglobal or Qotom on AliExpress. You can get these rebranded in the US on Amazon (I forget the name) with a warranty but at significant markup. Looking now it seems there are not so many Atom systems available, they are mostly Celerons and Core i3/i5 systems (which might be better, depending on what you want).
I was attracted by the specs of the previous version of this board, but the software side was a real mess. I screwed around with it quite a bit, but ended up giving up on them.
The company didn't have a reasonable, up to date kernel. Everything was out of date, and they were seemingly incapable of doing it. MediaTek people stepped in, because the board was getting press as a reference for how people could use their SOC. It was still not usable as a router. For example, while there is WiFi on the board, there is no Linux WiFi driver, only an Android driver. And other weird things that I forget right now.
Maybe things have improved, but definitely take care on this.
I like this idea of Other Fruit Pi boards designed for special, non-general computing applications like routers. Is the stock pi design "open" enough that skilled board designers can just plug-and-play and manufacture these kinds of custom boards, or does it take SeriousBusiness™ resources to do these things?
I'd be super interested in a Pi with a few SDRs built in. Currently I currently play with a Pi with a bunch of USB SDR dongles jammed into it but wouldn't it be nice to have all the RF hardware and connectors built into the main board?
Is this board intended to route packets using the CPU/Linux kernel or does the CPU act on a control plane on top of the two Ethernet controllers which talk to each other directly? Recently I set up a relatively powerful AMD "nuc" as a wifi AP and found my speed to be a dismal 7Mbps - enough for my office needs but I wonder if the cpu is liable to be a bottleneck (it's also possible I set something up very wrong)
Would this board be capable of switching AVB (IEEE 1722.1 specifically)? Existing switches are ridiculously priced, and this might be a great solution paired with some open source code for (home) music studios.
The particular switch chip used here (the RTL8367RB-VB-CG) is older than IEEE 1722.1. And even if it did support AVB/TSN it'd still be cheaper to get said hardware switch without attached SBC.
Remember for true AVB you're asking for network wide sync to the nanosecond with stream based traffic shaping and bandwidth reservations done in hardware accurate to those timings. It's all a real PITA to implement hence the price. That being said you can get a semi-reasonably priced switch like the MOTU AVB for <$500. If you need tons of ports the best way is going to be an older enterprise switch that supports AVB up for resale but beware you'll likely need the AVB license along with the hardware and it's an uncommon license.
Nice writeup! I've been meaning to replace my router with something more... controllable ever since I moved. Using a GigE USB feels messy, handling all those URBs I figure will tax the CPU more than I want.
Heaven would be any board that can run Debian and has 2 GigE NICs, and for psychological reasons $100 feels like a hard limit. Annoyingly hard niche to fill!
It is a really good board hardware wise. And mainline Linux support is coming along pretty quickly.
You can check the Quartz64 development wiki for updates for that board, which should carry over to other rk3566/rk3568 boards too. Right now the major drivers that have not been ported are the hdmi output and sink output driver.
The blocker for me buying one of these boards is that firmware sources have not been published, so devs are still using u-boot from the soc vendor (rockchip). It isn't terrible, but I have high standards now that my rk3328 and rk3399 devices run mainline u-boot with open source boot firmware.
Most people world wide do not have gigabit internet. The fact that you live some place that does does not invalidate this point. Furthermore most people can do almost everything they need with several mbps.
My hope is to make a somewhat minimal board for CM4 clustering. Eventually I will make a version with 4 CM4s and a single RJ45 jack, but I started with two CM4s and three jacks so I could test the ethernet switch independently of a CM4. And it turns out this board has some interesting uses too. I will break out both PCIe ports for the CM4s, so this board could have two intel wifi adapters plus three ethernet ports. It seems like that would make it a useful general purpose network thing. For example you could put a long range parabolic antenna on one wifi adapter and an omni antenna on the other and make it into a remote network access point (like a Ubiquiti Nanobeam connected to an AP). It should support VLANs etc as I will connect to the management interface on the switch via SPI or I2C.
However it is one of many projects I am working on, so I can't give a firm completion date. Microchip tells me they will ship me the ICs I need in a month, so it would be nice to have layout done by then. But we'll see.
Fork or star it on github to follow along. https://github.com/tlalexander/rpi-cm4-switch-board
Also if you have other thoughts about the design please share!