> Also, there are multiple u-boot binaries with separate environments. It could be done for redundancy or different modes. I’m not sure.
This is typically done in order to be able to safely update the u-boot bootloader, retaining the ability to boot the previous version if the upgrade fails. Also called A/B firmware update. The previous bootloader stage ("SBL" or "TZ", depending on secure boot mechanism) might check a flag in a configuration partition to see which loader should be started and whether a previous boot to that loader was attempted, and will revert back to the previously active one after an upgrade that fails to boot up, or fails to pass secure boot integrity checks.
A failure could for instance happen if the power drops while writing the update to the u-boot partition. Without such a mechanism, an update failure would brick the device. Alternatives are "never update the bootloader", or "hope for the best".
You can even boot and update your BIOS without a working processor. Did this over Christmas when the mainboard/CPU combo I got for a family member wasn't compatible. Just put in a USB drive with the correct file name and press a button on the board and it updated and I was good to go.
A little of topic, but just wanted to state how much better these things are than the old days where it was easy to brick a board.
Depressing to see a “reboot every 20 days” hack / workaround for whatever stuff doesn’t work correctly. OTOH the xfinity Arris modem I used to have rebooted almost daily whenever it felt a bit sad (I think if the BER crossed a threshold or something), so I suppose it’s an improvement if that’s your baseline.
This is very often due to upstream vendor issues. In the case of routers, even if you're the one designing the IC's, building all the layer 7 services, industrial design of the plastics, etc; there's likely still a vendor providing your radios and their drivers are almost always a black box for you.
Sometimes, while you go back and forth with the vendor in a hapless effort to repro anomalous bugs, the best thing for the customer is reboot-every-N-days instead of degraded-performance-forever.
It sucks, but its commonplace.
(N.B. I work in this space and its definitely been a pain point that we've had to accept to be able to make progress)
It wouldn't be a huge issue if the user had control on when the reboot would occur. I mean, for many users it would be much better to have a reboot at 4AM say every sunday morning than randomly and possibly in the middle of something important. An external timer power cycling the router at the desired moment before the 20th day can solve the problem as the counter would then restart, but it would have been trivial to implement it in software.
The norm in embedded world, particularly if the component giving you hard-to-debug issues is a "black box" where you have to deal with binary blob firmware - and almost everything dealing with WiFi or BT is closed source.
It is very likely that most Qualcomm customers like Starlink have access to the source code of the Qualcomm proprietary Wifi driver for the QCA wifi Access point SoCs. Some vendors also have access to the source code of the proprietary Wifi firmware running on the Tensilica CPUs inside the Wifi IP core of the SoC. (Linux runs on an ARM CPU in this SoC, the wifi IP cores are an extra realtime FW)
End consumers normally do not have access to some source code or any documentation about these chips, sometimes even the competitors are getting access to source code to integrate their solution better. I do not know whom they protect against, probably all people who did not sign a NDA.
These thinks are only shipped to end customers in binary only version to protect the IP from someone. Often it is pretty easy to tell the system it is now operating in a different country (e.g. setting in Web UI) and then it will not comply to the local radio regulatory requirements any more. From my experience it is not the FCC which really demands it, please blame the chip vendors for binary only driver first.
The worst hacker for a silicon vendor, where they normally protect most against, is some guy like the author of this blog post who analyses his device in much detail and tries to run own software on it which was not verified by the vendor first. Such software modifications could cause worse customer experience because it is performing worse (which is funny if they have to reboot the vendor software every 20 days) or could cause extra support effort.
In my experience, its exceptionally rare to have vendors release code like that, if for no other reason than opsec concerns. Obviously larger clients have much more leverage, but even in those situations its usually because they're getting a bespoke branch of whatever mainline firmware is produced for their other retail products (assuming its not from-scratch for a client that size in the first place).
> It is very likely that most Qualcomm customers like Starlink have access to the source code of the Qualcomm proprietary Wifi driver for the QCA wifi Access point SoCs
No. They are jus that hard to work with.
MTK, Qcom, BCom are all terrible with software support, no matter who you are.
Candela Technologies for example has access to the QCA wifi firmware source code for the QCA Wifi 5 AP chips. They provide custom builds here: https://www.candelatech.com/ath10k.php
I know of one other company.
I do not know if they also have access to the driver source code, but I assume so.
Silicon vendors often do not care about small customers, small is probably less than 250k units per year. If you are a customer which is expected to generate more than 5% of the revenue for the complete product line or the business unit then you get good support from these companies.
The proprietary Broadcom Wifi driver was leaked multiple times by some companies which did not run the script to clean a GPL tar, but did something on their own and forgot to remove the Boardcom wifi driver source code. Last time I saw this was already 8 years ago. The Broadcom proprietary Wifi driver also contained the source code of the firmware running on the ARM chip inside the Wifi IP, at least it did this some time ago.
For the Qualcomm 5G modem and the graphics driver this is probably different.
Unless you're Apple, I'd make an exception there - they absolutely require support from their vendors for all their non-standard stuff: AirPlay WiFi-based screen mirroring, Handoff, Thunderbolt-based Target Display Mode / Target Disk Mode.
Apple always demands to be in control of what is running on their devices, no matter what - even where it's pure software that's concerned, like with the NVidia fallout - and they (usually) insist on quality... there's a reason why Apple hardware feels so painless to interoperate, compared to the Windows or Linux world.
Woah, really in-depth write-up. It's sad that so many vendors are locking-up their Linux devices, preventing technically savy people from running their own software there (and also not releasing drivers to their hardware, for example it is really hard to find a 802.11ax router that is supported by OpenWrt).
> This app is written in Go, and it’s really hard to disassemble.
Shouldn't it be easier than an app written in C/C++? As far as I know Go embeds reflection metadata, including type information, which should help with decompiling. I have never tried though, so I might be wrong.
The MediaTek wifi AX platform is supported by OpenWrt master and also supported by upstream Linux including AX wifi. (MediaTek contributes a lot to upstream kernel)
* MT7622 SoC (2x Cortex-A53, including 4x4 ieee80211n wifi integrated in MT7622 and supported by upstream kernel too)
* MT7915 Wifi 6 (ieee80211ax) 4x4
* MT7530 5 port 1G switch (2.5G uplink)
You can find this hardware in the following devices:
* Linksys E8450 / Belkin RT3200 (same device just different name and color of casing)
* Ubiquiti UniFi 6 LR
* Also some others
Unfortunately the Linksys E8450 is unavailable in my country (Poland) or is ridiculously expensive. I settled on a Totolink X5000R which supposedly is supported by OpenWRT and has a Mediatek MT7621A. I will receive it in 2 days and test it.
Currently the satellites don't have intersat links. There is a ground station in each geofenced region, which makes mobility difficult. I'd expect it to start working once the intersat links are functional.
I don't think that's the issue. Starlink won't even let you move from one cell to another even when the cells are served by the same ground station. And moving from one ground station to another is no harder than handing off from one cell tower to another, which I assume was solved 30 years ago. They just aren't allowing it.
I don’t understand why star link is country bound. Here in the eu you can’t take your starlink equipment with you when you travel to another eu country. It just won’t work. Even if it’s a neighboring country. And even when starlink is offered in said country. You need a country specific kit.
This is the only reason I didn’t get a starlink subscription. Seems like a bullshit limitation. Reminds me kinda of Tesla stink.
There's a couple reasons this is currently happening:
1) The Starlink Constellation only looks for your base station in it's allocated cell and isn't setup to scan and discover mobile stations
2) Right now there is no communication inside the constellation (they might have launched a few with the laser links but it's not turned on broadly) so the satellites are linking down to another base station that has access out to the wider internet. They need to provision access on those so they don't get overwhelmed.
3) The base station itself needs to know where it is well enough to hit the active satellite for it's cell. You might be able to do that without GPS but it is an issue you have to address.
Both of these are supposed to be solved eventually and they've said they are planning to allow roaming base stations eventually.
They need to have enough intra-constellation communication to route traffic out to some satellite that can see a base station. They said the "v0.9" satellites that were launching starting this year were all laser equipped and all sats launched this year (and presumably in the future?) would be similarly equipped.
SpaceX has to allocate bandwidth on the satellites to consumers. If consumers move, they will have overallocated bandwidth in some places, degrading network performance. They'll probably come out with a mobile product at a higher price point in the future.
Kind of doubtful that consumers moving their terminals around would be statistically significant compared to all the other factors that cause regional variations in aggregated bandwidth usage.
If it were offered as a mobile service it would be very attractive to people on the go so it would skew the number of mobile nodes higher than expected. I think base station discovery is another issue blocking mobile nodes from happening so an in between step could be to provide a way to easily move your base station and register it to a new cell.
>I don’t understand why star link is country bound.
First and primarily, the legal front: All regulated electromagnetic spectrum transmission is country-bound, since the regulators are by country. Private entities cannot legally transmit in any given country without authorization, and there is a fairly in-depth set of national and international legal frameworks for this that are pretty well respected. In principle SpaceX could negotiate the proper licenses and agreements internationally to allow roaming, and they certainly will eventually for certain applications like maritime and aircraft usage. But it's perfectly understandable in their bootstrapping phase of pure fixed point service (which is the easiest for a host of reasons) that they just put a blanket ban on it. Those who want mobile just have to wait.
Second, there are technical reasons. Starlink will always face some fundamental density limitations in how small a cell any sat can talk to and how much bandwidth there will be per cell and for the overall sat. And in its initial version, satellites also act purely as "bent pipes", mediating directly between a ground station which then talks to the internet at large and the end customer CPE. And SpaceX is being careful and deliberate about building up all the infra needed to scale its network, and wants to ensure that everyone joining gets a good experience. So they're also being fairly careful about allowing anyone out of their assigned cell.
Eventually a number of developments will alleviate this to some extent. Intersat optical links are now launching on new birds, and IIRC this year (maybe early next?) they'll have the in-space routing network up. That will dramatically increase their flexibility in routing traffic and dealing with chokepoints (as well as allowing Starlink usage out of range of ground stations, again critical for ships/aircraft). Simple increasing raw numbers of sats (and particularly VLEO v-band with Starship) will improve their density margins a bit as well. And then there is just plain old software infra stuff. Portal for customers to update addresses, backend to deal with handling allocations and looking for cell overload including across varying regulatory regimes, figuring out pricing at the margins and what to allocate for fixed vs mobile usage in a given area, letting people see where there is room and where there isn't, and putting a nice GUI over all of that is work that isn't needed for a 1.0 MVP but will be important to getting stuff to work right.
>Seems like a bullshit limitation. Reminds me kinda of Tesla stink.
No, this one at least is pretty reasonable. Easy to forget that Starlink is doing something very different, new, with more to come, and is very, very early. And still a chance it could all fail, they haven't actually hit their core minimum economic targets yet (that was what Elon's "bankruptcy" warning last month was about). It's much more capex intensive and opex intensive then traditional offerings. They've got an excellent product though and are bootstrapping it impressively but they've got to be very careful nonetheless. Plenty of fundamentally incredibly promising businesses have foundered on the shoals of scaling.
I like this router a lot, a few notes from using it for almost a year now:
1. It gets really hot. Like, it kinda makes the plastic routers seem well-ventilated.
2. Congestion is, sadly, a real concern here. Too many devices will definitely cause the router to choke a bit, and if you're going to share it with a family that's just something to be aware of.
3. Latency is fine wired, but a little shaky over WiFi. Especially recently; I'm not sure if this is a firmware issue or what, but there's some funky QoS issues that I've noticed lately where latency is wildly unpredictable. It almost shut down a Zoom interview the other day...
4. You get one LAN port. That's it.
Besides those gripes, it's fine, but it leaves a lot to improve on for the next iterations. Hopefully Starlink is taking notes here, because as nice of a device as it is, it does feel a little rushed for the purpose of getting it out the door.
If they put a few GB of RAM in the router, they could run a proxy in it to play video out of a cache of traffic broadcast to all terminals in range of a satellite and held by terminals that actually want it, obviating need to upload and download multiple logically-identical copies of content being watched (at possibly different points in the playback sequence) from multiple terminals in the area. This would reduce loading on the channels, enabling more effective bandwidth from the very limited available spectrum.
It would need adaptation at the application layer, but the data management needed is minimal. It would also need some attention at the central terminus / edge node serving the area, to identify duplicate requested streams and shift a corresponding stream to the broadcast channel. In turn, this would typically depend on cooperation from the original source of a stream, typically mediated by their contract with an edge cache service.
The extremum of usefulness for this facility would be when hundreds or thousands of terminals in an area are used to watch the same World Cup football match. In this case, the cache storage footprint would be minimal, as almost all viewers would be watching at the same point in the broadcast.
More commonly, a few blockbuster movies and local news might dominate broadcast traffic. But no advance planning or personal attention would be needed; any stream found being fed to more than one downlink could be shifted to broadcast according to the amount of bandwidth that would thereby be saved.
As laser inter-satellite links are added to the constellation, a single uplink stream could serve simultaneous broadcast from multiple satellites in a chain.
No need for any caching, just do proper multicast like "real" IPTV networks do. Other than that, I don't think that enough people would watch the same shows within a reasonable timeframe to justify the costs.
Multicast would save exactly zero Hz of spectrum bandwidth, which a moment's thought would reveal to you. Allow me encourage a habit of that.
Multicast routing uses, exclusively, point-to-point links. So, a packet could go up to the satellite once, but would need to be copied down to every terminal individually.
But internet video is in any case not distributed by multicast UDP. Even if it were, you still would need caching for viewers not watching identically the same frames at the same time, because a multicast router does not do time-shifting: all copies of each packet are queued to send immediately.
Of course the transport link would have to cooperate and open a group-encrypted channel for every multicast group a client is subscribed to. Basically how regular satellite TV works, but with dynamically allocated bandwidth.
You were right to point out multicast. Its a great way to do cache discovery if nothing else. Saying that its de facto not effective at saving spectrum bandwidth handwaves pretty much every last mile detail thats relevant in networks like these.
No, fuzz is 100% wrong. Multicast makes more-effective use of fiber, where each new fiber link branching off provides its own, private pool of bandwidth, added on to all the others.
Once you get to radio spectrum transmission, each pair of endpoints competes with every other pair in range, in a strictly limited pool of bandwidth. Adding another terminal takes away capacity from all the others. A multicast stream going to your satellite and then distributed to 100 terminals by multicast burns 100x the fixed available bandwidth as each terminal gets.
The only way to get any benefit from batching delivery over radio spectrum is via broadcast. For a packet network, you have the extra chore of making those broadcast packets actually useful to as many terminals as possible. In practice that requires caching of broadcast content in the terminal, so that when it is actually asked for it is already there.
>Once you get to radio spectrum transmission, each pair of endpoints competes with every other pair in range, in a strictly limited pool of bandwidth. Adding another terminal takes away capacity from all the others. A multicast stream going to your satellite and then distributed to 100 terminals by multicast burns 100x the fixed available bandwidth as each terminal gets.
Multicast is indeed problematic for wireless networks, even beyond what you describe (e.g. FEC, ACK storming, retransmits, etc). But if the phy layer is using OFDMA or similar, its entirely plausible to dedicate a subcarrier for multicast traffic. This doesn't solve the O(N) terminal problem you describe, but it mitigates the impact for concurrent transmission of unrelated unicast traffic while the multicast delivery is happening.
There's a fairly large academic body of work on this specific topic re: OFDMA multicast. And though I think your note on caching is a necessary component of this (a la Open Connect), I still think there is a role that multicast can play here.
FEC is not a problem, but the solution to a problem. FEC is "forward error correction", sending a few percent of redundant data, enough so the receiver can fix up corrupted packets on their own. (FEC was once a patent problem, but those have mostly expired.)
ACK storms aren't a thing with multicast; if FEC doesn't get you what you need, you wait a random time and send a NAK up the line, and the sender re-sends the requested bit. If the bit you were about to request shows up again before you get to asking, you discard the NAK you would have sent. So, if a lot of receivers miss the same packet, only one NAK usually goes back.
The Starlink transceivers are SDRs and extremely versatile. So, instead of playing games with modulation and subcarriers, it probably uses its whole band on each packet, effectively broadcasting all packets, and receivers just pick off whichever packets are meaningful to them. The ones actually intended for more than one terminal are the "broadcast" packets, differing only in how they are processed by the terminal.
The actually tricky thing is that broadcast streams cannot be like the regular end-to-end packet streams, where identical images are delivered in absolutely unlike packets. You need a higher-level negotiation identifying streams of interest by name and playback position, so that the uplink terminal knows when it is sending (or will have sent) the same stuff twice. If it decides some of its traffic should be broadcast instead, it can substitute packets from an equivalent, shared stream. The receiving terminal, meanwhile, is told that the stream it was getting is ended, and it should start extracting and caching content from the substituted broadcast stream.
The usual DRM nonsense will make this annoyingly more complicated: the participating broadcast receivers get a copy of a key to decrypt the broadcast traffic. Everybody else throws it away as indistinguishable from noise, like all traffic not meant for them.
> ACK storms aren't a thing with multicast. if FEC doesn't get you what you need, you wait a random time and send a NAK up the line, and the sender re-sends the requested bit
Interesting. This has not been my experience during my time so far in building a 802.11-over-mmWave ISP for the last 7 years. BUM traffic is routinely an issue, one that we measure frequently, and multicast ACK storming is absolutely a side effect of times when things are bad. This is usually because multicast, at least with 802.11, has to be sent at the lowest possible MCS because you don't know the rate capabilities of the STAs on the other side of the fence (also true for broadcasts) and as such the delivery is inherently unreliable. When things are bad, ACK storms are the measurable sine qua non of these events.
Its possible that, for Starlink, this isn't an issue for them because its not a conventional PMP infrastructure as you point out. They know the rates of the terminals on the other side and they're not using non-proprietary modulation.
FEC (or even network coding) can certainly help in these situations, but it has been far from a silver bullet in most testing I've been a part of.
> The Starlink transceivers are SDRs and extremely versatile. So, instead of playing games with modulation and subcarriers, it probably uses its whole band on each packet, effectively broadcasting all packets, and receivers just pick off whichever packets are meaningful to them.
Even if its an SDR, and presumably using something like phase-key shifting for modulation at the PHY layer, there is still MAC layer multiplexing it could use to offset the impact of multicast. E.g. in 802.11ax with MU-MIMO, you could dedicate an entire spatial stream to one kind of traffic, or alternatively you could dynamically put STAs into MU groups based the bulk of the traffic they're sending to blunt their impact on the shared spectrum. This is even more interesting in the SDR context where you can build a more adaptive algorithm for grouping (e.g. like using geospatial context to ensure minimal interference etc).
>The actually tricky thing is that broadcast streams cannot be like the regular end-to-end packet streams, where identical images are delivered in absolutely unlike packets. You need a higher-level negotiation identifying streams of interest by name and playback position, so that the uplink terminal knows when it is sending (or will have sent) the same stuff twice
I think this is where your point about the proprietary SDR is the most interesting (and maybe even for the DRM case as well). Would be interesting to see how they could handle this being in the middle of it all, without causing a major performance impact to end users. But thats mostly an implementation detail.
I should have said ACK storms aren't a thing if whoever specified the protocol you are on was not an idiot. If you don't have any choice about the protocol you must use, you must endure whatever horrors have been inflicted upon you.
Starlink will not have that problem, anyway. They could invent their own horror story, and then have no one to blame but themselves.
A more complicated modulation scheme could enable cheap hardware in the terminals to see only a fraction of traffic, which might be worth the RF complication. That probably depends on how much of the protocol work they are willing to do in the terminal's FPGA. Certainly the CPU cores in the terminal won't be up to sorting through the whole pile.
An FPGA configured to watch for a small number of packet headers of interest, and decrypt and deliver just those packets, ought to be able to relieve the CPU core enough.
You may want to look into Othernet, which does satellite broadcast data distribution to rasp pi based terminals. But they only have 2kbps to work with, so it’s largely the day’s news and some broadcast messages.
But it’s free and it works!
I understand some Canadian arctic cities want to do what you say because they depend on x$/gb satellite links, but not big enough for a Netflix or whatever appliance. But every time there’s an iPhone or windows update, everyone is downloading it independently. It would also make sense to load a box of SD cards with their supply flights and put them on-net. Dunno if a formal system exists for that.
Thanks for the Othernet mention. I just looked into the Wikipedia article and the company's web site, and it's exactly the sort of tech I enjoy.
Unfortunately, even though its receiver was supposed to enter production in November, it's still not available. Ordinarily I'd blame 'rona/supply chain/whatever, but according to Wikipedia, it failed to deliver a previous kickstarter.
When it's available, I'll buy one. But I won't be the first, in case this turns into vaporware.
The satellite based WiFi used on commercial aircraft do this, with some on-demand content cached on the aircraft. I think they also "tape-delay" some of the live stuff to deal with short fade-outs. Google around for DVB-S2 and DVB-S2X.
This is typically done in order to be able to safely update the u-boot bootloader, retaining the ability to boot the previous version if the upgrade fails. Also called A/B firmware update. The previous bootloader stage ("SBL" or "TZ", depending on secure boot mechanism) might check a flag in a configuration partition to see which loader should be started and whether a previous boot to that loader was attempted, and will revert back to the previously active one after an upgrade that fails to boot up, or fails to pass secure boot integrity checks.
A failure could for instance happen if the power drops while writing the update to the u-boot partition. Without such a mechanism, an update failure would brick the device. Alternatives are "never update the bootloader", or "hope for the best".