In short, it was remote bricked, by giving it commands to rotate a bit. After successfully executing those commands - no further commands could be received, as now the antennas are not facing earth anymore.
But luckily it automatically readjust itself to earth automatically every half year exactly for these events. So on 15.10 we will know, if it is really lost. In either case, the end of its mission is near anyway, because the nuclear batteries are near its end.
The Debian package installer once asked me (a long long time ago) whether I want to restart sshd after a glibc update, saying existing sessions wouldn’t be affected. That was a lie, apparently, because the SSH session I was updating the system died and the resulting SIGHUP killed the update process in a way that necessitated some recovery later.
More seriously, Mikrotik routers have a nice feature where they will rollback your config change if the connection you’re configuring one over stops responding to keepalives. Like a lot of Microtik features, it’s probably copied from some Serious Business network OS, but I wouldn’t know.
> Like a lot of Microtik features, it’s probably copied from some Serious Business network OS, but I wouldn’t know.
Not sure who came first, but OpenWRT does this if you make a breaking change in the webinterface, and connectivity is lost for 60 sec, it will rollback the changes.
it’s probably copied from some Serious Business network OS
I wouldn't know who came first, but it's a feature of JunOS (Juniper) as well: every config apply first applies the config, then waits for confirmation on the terminal where it was ran. If confirmation isn't given within X seconds, it reverts the config change.
lol, reminds me of the time I forgot I was on localhost, and needed to run a huge statistics processing task. I remembered to carefully use bg and disown so the program would keep running even after I disconnected. Then I sighed in relief and powered off my machine.
I was halfway into a multi hour render on my 486 when I accidentally switched it off. Luckily, I caught myself before I released the power button, so I jammed an eraser under my finger and taped it down so the button wouldn’t spring back and complete the toggle.
Had a node that I was connected to over wireguard. Wanted to reset the wireguard conf.
sudo wg-quick down wg0
Nice one, mate. Had to drive back to log in and bring up that interface. I still do this, FWIW, but now I use `at` to schedule "up" 1 min in future haha. So far so good though it's not smart :)
> FWIW, but now I use `at` to schedule "up" 1 min in future
Any time I’m ever doing a pfctl to change my router’s firewall rules I schedule a “revert to the n-1 rule set” job two minutes from now to avoid the trip to the basement & hunt for the serial cable.
I do +2 because once I was too close to the next minute, fat-fingered the update command & instinctively corrected it. But the change had reverted before I hit enter on the new update, so of course I blocked too much traffic and wedged my SSH connection, triggering the cursing and trip to the basement with the serial cable.
Luckily for me, I just had to go down the hall to the data center, and then reset it with the local terminal. Compared to you, I learned the lesson on the cheap, but you got the bonus of a nice get out of jail free card. Neat CYA trick that I will keep in mind.
Heh, this reminds me of my ~10 year old self. I had unknowingly infected my family PC with some form of malware, but no knowledge of AV software.
I guessed, incorrectly, that perhaps the OS developers would have been smart enough to disallow users from deleting OS owned/managed files. So I went around finding and deleting everything that looked remotely official, including C:\Windows\System32. To my surprise, it worked.
Until I rebooted :(
I recently had this hacker feeling too. I wanted to help my friend diagnose some internet issue at her place and gave her a Raspberry Pi that she could connect to her router. It had Tailscale preinstalled, so as soon as it booted, I was able to access it without her having to open ports in her router. However, I managed to mess up some network config that caused Tailscale to stop working and lost the remote access. Since I knew pretty much exactly what I did wrong, I was able to have her connect a USB keyboard to the Pi and send her exactly what to type in order to fix it, and it worked! (For some reason, she doesn't have single HDMI cable so she couldn't connect it to a monitor.)
>For some reason, she doesn't have single HDMI cable so she couldn't connect it to a monitor.
LOL. Outside of computer dorks what not, it's not very common for people to have a large selection of cables to go digging through. Maybe a drawer full of decrypt USB cables that came as chargers to a phone, but most people I know lose those cables and are constantly buying replacements. So expecting the average person to have a box full of random cables suggests to me you might be living in a bit of a bubble.
i find that it’s actually fairly common for people to have accumulated several loose and random cables over time. even my mother, who is not particularly a technologically inclined individual, still had several hdmi cables and different usb cables. every single device (not just phones but things like flashlights, wireless accessories, fans, etc…) people buy that needs to be charged generally comes with a charging cable, and there are ever more usb chargeable devices coming on the market nowadays.
hell, even homeless people such as myself tend to have giant collections of random cables, even when they don’t actually have anything that requires charging.
this is actually the hardest problem in computer science (develop a general intelligence capable of predicting when a command will shut down your session)
I should point at that astronomical navigation is a remarkable skill that was developed and turned into routine calculations in a relatively short period of time. The first order calculations are based on star imaging and used a Kalman filter,which had been invented just a few years before (https://en.wikipedia.org/wiki/Kalman_filter#History) along with a star catalog (list of known star locations relative to the sun/earth) and direct observation by astronauts. I think a sextant was useful.
Second order calculations use careful analysis of the signal pattern in telemetry data- IIRC you can see a slow stretch of the phase which can be used to estimate distance and velocity with high accuracy.
Voyager, along with Apollo, stand as the finest examples of human engineering done yet- we got a bunch of people to the moon and back, and built a probe that still operates 50 years later... farther than anything else humans have launched... I'd be lucky if I can deploy my web app once a week.
Logged in to comment that this particular comment made me laugh. It really puts what I personally do to shame, many times over. Thanks for the laugh. Cheers.
So other people talked about how it does track, but there's another thing to note here.
"The high-gain antenna has a beamwidth of 0.5° for X-band, and 2.3° for S-band."
At 130-150 AU, the earth is always within about 0.4° of the sun. Since commands are sent on S-band, pointing directly at the sun gets a pretty good signal.
If I can trust arbitrary estimates on stack exchange, the deep space network transmitters will be about equal brightness to the sun at frequencies near 2GHz when the 20signal is 10 Hertz wide. That would put the transmission rate in the ballpark of 10 bits per second, and the real number is 16 bits per second, so that seems to work out.
Being drowned out is harder than you might think. The maximum data rate of a weak signal is 1.4 x [bandwidth] x [signal-to-noise ratio]. If you transmit across a 200MHz band, and your signal is a million times weaker than the noise, you can do hundreds of bits per second.
As long as there's not too much light pollution. Fortunately for the dung beetles, their habitat isn't very urban. However, it's the little examples like this that make me a light pollution dork.
Basically the probe knows where it is because it knows where it isn't. By subtracting where it is from where it isn't, or where it isn't from where it is (whichever is greater), it obtains a difference, or deviation. The guidance subsystem uses deviations to generate corrective commands to drive the probe from a position where it is to a position where it isn't, and arriving at a position where it wasn't, it now is. Consequently, the position where it is, is now the position that it wasn't, and it follows that the position that it was, is now the position that it isn't.
Wait, i don't understand. I was under the assumption that this text was a joke, but now I'm seeing it in reference to air force training materials? Is it a joke there as well or did someone actually write this text seriously, and plan for it's use as intelligible instruction?
There's oodles of references to this online but nothing really I've found so far explaining whether it was ever intended to be taken seriously in the first place. It's hard to imagine anyone doing so.
Yes, it's a joke. It's not complete nonsense, though, unlike the Turboencabulator. There's a real explanation buried inside there. ("Where it isn't" is the target position, so it's looking at the deviation between where it currently is, and where it should be. The next time step, "where it wasn't" is the old target position, and so forth.)
It's not really hypothetical: losing communication with stuff in space is a very common failure mode and a huge amount of the system design is focused on making it as unlikely as possible (generally the radio system gets a huge priority in almost everything and there are a lot of failsafes built at every level to make it possible to reestablish communication if anything disrupts it).
I was amused to learn that if modern satellites lose contact with earth, they go into "safe mode": pointing towards sun, solar panels fully deployed, everything else except telemetry, radio, and temperature management disabled, waiting for further instructions.
https://en.wikipedia.org/wiki/Safe_mode_in_spacecraft
Imagine deploying a billion dollar piece of hardware and hoping that it has enough intelligence to keep itself from burning up before you can reestablish
contact!
Aerospace has a very high quality standard compared to other industries.
Lots of formal processes capture what would otherwise be informal design decisions elsewhere. In this case, they probably have reams of pages detailing a failure mode effects analysis (FMEA). One mode is “oops, we sent the wrong command” and the document would define the specific design mitigation(s) for that outcome until it reaches an accepted risk threshold.
As far as I’m aware, no NASA standards call for FMEDA. It doesn’t mean a project manager couldn’t levy it, but it’s not often that a contractor adds additional requirements to a gov funded build.
FMEA relies on a really smart person anticipating all the different combinations of failures worth exploring (NxM), not just N or M.
Some failures are fairly common, and individual failures might be fairly inert but have more serious consequences if they are cascaded with another specific failure.. for example, cruise control enable + failure of steering wheel control pad _and_ previously undetected failure of brake sensor/brake light circuit = cruise control stuck ON. Actually, this failure is inert if the cruise control is OFF when it happens. Contrived example but you get the idea ...
I have seen a lot of FMEDA (and other tool) use lately to combat concerns with cascading failure, but not sure what's currently standard at NASA or how they deal with this. I would think cascading failure would be their expected scenario on a 10+ year unmanned mission.
NASA STDs, handbooks, guidebooks, NPDs and NPRs are all open-source. They don’t mention FMEDA, and they don’t generally have a detectability column in their FMEA. IMO they are a little outdated
I've done for NASA what they were calling FMECA and FTA for a subsystem. They had a lot of freedom to tailor the analysis to the situation, and the end result didn't quite match anything established. We addressed detection in some of the FMECA columns which are not traditionally for detection; and events in some of the FTA. It was a contortion of terminology and format to modernize and maximize the value of the analysis given their limited resources and the bureaucracy of what they were allowed/required to do on paper.
Here's how I would describe the possible analysis approaches in broad terms, avoiding terminology that NASA does not officially use.
- Start from the hazard of being pointed in the wrong direction and work backwards to identify the causes, forming a tree.
- Start from the event of commanding the wrong direction and work forwards to identify mitigations or the lack thereof, also forming a tree.
- Start from looking at a component or subsystem, list all the ways it can fail without regard for the application. Then consider the application and work up towards the causes/events.
- Close any gaps between the top-down and bottom-up approaches.
Yes, what you're describing is two different approaches for safety analysis. According to the NASA software engineering handbook [1]
"Software Fault Tree Analysis (SFTA) is a top-down approach to failure analysis which begins with thinking about potential failures or malfunctions (What could go wrong?) and then thinking through all the possible ways that such a failure or malfunction could occur. Fault Tree Analysis (FTA), is often used by the hardware teams to identify potential hazards that might be caused by failures in hardware components or systems, but with the SFTA, the software isn’t considered the hazard, but it can be a cause or contributor when considered in the context of the system."
"The Software Failure Modes and Effects Analysis (SFMEA) is a bottom up approach where each component is examined and all the possible ways it can fail are listed. Each possible failure is traced through the system to see what effect it might have on the system and to determine if it results in a hazardous state. Then the likelihood of the failure and the severity of the system failure can be considered."
But, to the earlier post, these are driven by hard requirements; specifically adherence to NASA STD 7150.2 and NPR 7150.2. Developers/contractors can tailor/waive them with pre-approval but, in general, they tend to go in the direction of less requirements, not more. This may all be moot because I think Voyager pre-dates any of those requirement documents and I'm not sure what existed in the late 1970s.
The D aspect of the FMEA I worked on was motivated by a reliability requirement, not by 7150.2. 70's NASA was using FTA and FMEA but avoiding putting numbers on top-level analysis. I imagine they did whatever ad-hoc analysis they thought was necessary for such a highly publicized mission even if it wasn't a separate deliverable.
Edit: The comment you deleted right before I could reply was good! I think people would enjoy and benefit from your description of how the process works if you're willing to repost it.
As you noted the reliability requirement did in fact flow down from an engineering requirement which is why they exceeded the minimum FMEA standards. There's no official guidance on where and how exactly to track that information so they put it in the usual place but in an unusual way. The lack of a standard during Voyager's time probably impacted the visibility of the work more than the substance.
The Voyager that's flying now is not necessarily the Voyager that was launched.
The hardware is the same, but they've updated, patched, and rewritten the software that's running in it throughout the years.
I'm not suggesting that the failsafe mode wasn't originally considered, and implemented, but simply that it doesn't have to be the case. They could have made changes to it over time.
It wasn't a solution for this specific problem. Spacecraft orientations are going to drift over time, periodically rehoming is the simplest way of dealing with it. That it doesn't care whether the orientation drift was natural or artificial is just a bonus.
Actually there's a couple of work arounds for this problem as they anticipated it all along. My father was Director of Operations at Tidbinbilla deep space tracking station which ran most of the comms to Voyager 1.
I am paraphrasing what he said as a non-technical person: Voyager has both a dish receiver, and a pole antenna. The dish is the usual mechanism for comms but in an emergency such as this they would send commands to the other antenna. To do this they would turn the main tracking station dish up to max, and send a "TURN AROUND!" signal out.
But prior to that they had to alert the local electricity grid, and the local air traffic control to not have any planes flying over at the time!
I guess the Voyagers are too far away for this manoeuvre now.
Oh man that reminds me a lot to Kerbal Space Program, those times I lost communication because of a wrong turn and the antenna/solar panel faced the wrong way
I like to think at least a few NASA engineers come to meetings with some brilliant ideas....that were cooked up at the 1am mark of a 7 hour weekend KSP binge.
A bad case of planet between antenna and space center! Is orbit stable? If so, launch and maneuver into position a huge network of relays until connectivity is restored. If not, watch while probe crashes and burns onto the celestial body. An automatic realignment function would be nice...
I exaggerated a bit, it only happened with solar panels.
But it does gives you a similar feeling when you schedule a maneuver on a point where the planet/moon blocks your signal, and you only realize after it's too late
> In short, it was remote bricked, by giving it commands to rotate a bit. After successfully executing those commands - no further commands could be received, as now the antennas are not facing earth anymore.
I did this exact thing in the small last night - wanted to work on fixing a faulty switch, so my wife and I get on the intercom system on our landline phone so she can tell me when the correct breaker is off.
And of course, breaker #1 is the one that controls the intercom, severing our connection.
The text and link I provided mention it as well, but I am now not sure, if giving 15.10 as a date was maybe confusing for non europeans (or non germans, I am a bit lost who uses what date format)...
It avoids confusing the Americans who otherwise put the month in the wrong place.
It avoids being ambiguous for everyone who may otherwise be worried that it was written by an American with the month in the wrong place, when the day is less than 13.
2023-10-09 is the 9th of October and it's clear to everyone regardless.
It also has the benefit of sorting chronologically if sorted "by name" when used in a filename as it's largest unit on the left, smallest on the right.
It was confusing to me. Took me a while to realize it was a date and then had to deduce what it represented.
Frankly before your comment I wasn't going to complain because I saw the tantrum you threw when people corrected you on the usage of "bricked" but maybe next time spell the month to avoid ambiguity.
But luckily it automatically readjust itself to earth automatically every half year exactly for these events. So on 15.10 we will know, if it is really lost. In either case, the end of its mission is near anyway, because the nuclear batteries are near its end.
edit: Nasa has a blog post on this https://blogs.nasa.gov/sunspot/2023/07/28/mission-update-voy...