Hacker News new | past | comments | ask | show | jobs | submit login
NASA mistakenly severs communication to Voyager 2 (theregister.com)
490 points by belter on July 31, 2023 | hide | past | favorite | 446 comments



In short, it was remote bricked, by giving it commands to rotate a bit. After successfully executing those commands - no further commands could be received, as now the antennas are not facing earth anymore.

But luckily it automatically readjust itself to earth automatically every half year exactly for these events. So on 15.10 we will know, if it is really lost. In either case, the end of its mission is near anyway, because the nuclear batteries are near its end.

edit: Nasa has a blog post on this https://blogs.nasa.gov/sunspot/2023/07/28/mission-update-voy...


All: if you want to argue about what "bricked" means, please do that at https://news.ycombinator.com/item?id=36946612, not here. But also consider: "Please don't pick the most provocative thing in an article or post to complain about in the thread. Find something interesting to respond to instead." - https://news.ycombinator.com/newsguidelines.html


reminds me of the time I forgot i was on a remote connection, and could not figure out why the thing quit responding when i typed eth0 down


The Debian package installer once asked me (a long long time ago) whether I want to restart sshd after a glibc update, saying existing sessions wouldn’t be affected. That was a lie, apparently, because the SSH session I was updating the system died and the resulting SIGHUP killed the update process in a way that necessitated some recovery later.

More seriously, Mikrotik routers have a nice feature where they will rollback your config change if the connection you’re configuring one over stops responding to keepalives. Like a lot of Microtik features, it’s probably copied from some Serious Business network OS, but I wouldn’t know.


> Like a lot of Microtik features, it’s probably copied from some Serious Business network OS, but I wouldn’t know.

Not sure who came first, but OpenWRT does this if you make a breaking change in the webinterface, and connectivity is lost for 60 sec, it will rollback the changes.


it’s probably copied from some Serious Business network OS

I wouldn't know who came first, but it's a feature of JunOS (Juniper) as well: every config apply first applies the config, then waits for confirmation on the terminal where it was ran. If confirmation isn't given within X seconds, it reverts the config change.


That only applies when you do "commit confirmed".


Yes, the Safe Mode button. But you have to remember to press it before you start configuring the router and then exit Safe Mode when you're done.


Or the time I typed `reboot` and my workstation didn't reboot, because that xterm was the department file server.


lol, reminds me of the time I forgot I was on localhost, and needed to run a huge statistics processing task. I remembered to carefully use bg and disown so the program would keep running even after I disconnected. Then I sighed in relief and powered off my machine.


I was halfway into a multi hour render on my 486 when I accidentally switched it off. Luckily, I caught myself before I released the power button, so I jammed an eraser under my finger and taped it down so the button wouldn’t spring back and complete the toggle.


Had a node that I was connected to over wireguard. Wanted to reset the wireguard conf.

    sudo wg-quick down wg0
Nice one, mate. Had to drive back to log in and bring up that interface. I still do this, FWIW, but now I use `at` to schedule "up" 1 min in future haha. So far so good though it's not smart :)


> FWIW, but now I use `at` to schedule "up" 1 min in future

Any time I’m ever doing a pfctl to change my router’s firewall rules I schedule a “revert to the n-1 rule set” job two minutes from now to avoid the trip to the basement & hunt for the serial cable.

I do +2 because once I was too close to the next minute, fat-fingered the update command & instinctively corrected it. But the change had reverted before I hit enter on the new update, so of course I blocked too much traffic and wedged my SSH connection, triggering the cursing and trip to the basement with the serial cable.


Luckily for me, I just had to go down the hall to the data center, and then reset it with the local terminal. Compared to you, I learned the lesson on the cheap, but you got the bonus of a nice get out of jail free card. Neat CYA trick that I will keep in mind.


There's a feature in many business data centers called "remote hands". Mainly to avoid the drive/flight to admin a remote machine.


Heh, this reminds me of my ~10 year old self. I had unknowingly infected my family PC with some form of malware, but no knowledge of AV software. I guessed, incorrectly, that perhaps the OS developers would have been smart enough to disallow users from deleting OS owned/managed files. So I went around finding and deleting everything that looked remotely official, including C:\Windows\System32. To my surprise, it worked. Until I rebooted :(


or when I was futzing with display configs on a linux install, accidentally disabled my screen, and had to restore it blind


That’s such an absolute hacker feeling, I was honestly surprised I got it to work, back then. Thanks for reminding me of that!


I recently had this hacker feeling too. I wanted to help my friend diagnose some internet issue at her place and gave her a Raspberry Pi that she could connect to her router. It had Tailscale preinstalled, so as soon as it booted, I was able to access it without her having to open ports in her router. However, I managed to mess up some network config that caused Tailscale to stop working and lost the remote access. Since I knew pretty much exactly what I did wrong, I was able to have her connect a USB keyboard to the Pi and send her exactly what to type in order to fix it, and it worked! (For some reason, she doesn't have single HDMI cable so she couldn't connect it to a monitor.)


>For some reason, she doesn't have single HDMI cable so she couldn't connect it to a monitor.

LOL. Outside of computer dorks what not, it's not very common for people to have a large selection of cables to go digging through. Maybe a drawer full of decrypt USB cables that came as chargers to a phone, but most people I know lose those cables and are constantly buying replacements. So expecting the average person to have a box full of random cables suggests to me you might be living in a bit of a bubble.


i find that it’s actually fairly common for people to have accumulated several loose and random cables over time. even my mother, who is not particularly a technologically inclined individual, still had several hdmi cables and different usb cables. every single device (not just phones but things like flashlights, wireless accessories, fans, etc…) people buy that needs to be charged generally comes with a charging cable, and there are ever more usb chargeable devices coming on the market nowadays.

hell, even homeless people such as myself tend to have giant collections of random cables, even when they don’t actually have anything that requires charging.


I got my first laptop (win95 at the time) at a garage sale super cheap because the owner throught it was broken.

He'd set the resolution to 1024x768 on the 800x600 panel, once I fixed that I was good to go!


this is actually the hardest problem in computer science (develop a general intelligence capable of predicting when a command will shut down your session)


This is like some weird version of the halting problem


usually I do something like:

    # ifconfig eth0 down; ifconfig eth0 up
that said, I have done this:

    # reboot
...on the wrong system


How the heck does it know where earth is?

That's some impressive science there, not like there is a deep-space GPS.

Does it look for the sun and figure out from there?


I should point at that astronomical navigation is a remarkable skill that was developed and turned into routine calculations in a relatively short period of time. The first order calculations are based on star imaging and used a Kalman filter,which had been invented just a few years before (https://en.wikipedia.org/wiki/Kalman_filter#History) along with a star catalog (list of known star locations relative to the sun/earth) and direct observation by astronauts. I think a sextant was useful.

Second order calculations use careful analysis of the signal pattern in telemetry data- IIRC you can see a slow stretch of the phase which can be used to estimate distance and velocity with high accuracy.

Voyager, along with Apollo, stand as the finest examples of human engineering done yet- we got a bunch of people to the moon and back, and built a probe that still operates 50 years later... farther than anything else humans have launched... I'd be lucky if I can deploy my web app once a week.


yeah, it really puts my god awful pile of terraform in perspective.


Logged in to comment that this particular comment made me laugh. It really puts what I personally do to shame, many times over. Thanks for the laugh. Cheers.


So other people talked about how it does track, but there's another thing to note here.

"The high-gain antenna has a beamwidth of 0.5° for X-band, and 2.3° for S-band."

At 130-150 AU, the earth is always within about 0.4° of the sun. Since commands are sent on S-band, pointing directly at the sun gets a pretty good signal.


Is that signal not drowned out by the sun? Or are these bands where the sun doesn't do anything?


If I can trust arbitrary estimates on stack exchange, the deep space network transmitters will be about equal brightness to the sun at frequencies near 2GHz when the 20signal is 10 Hertz wide. That would put the transmission rate in the ballpark of 10 bits per second, and the real number is 16 bits per second, so that seems to work out.

Being drowned out is harder than you might think. The maximum data rate of a weak signal is 1.4 x [bandwidth] x [signal-to-noise ratio]. If you transmit across a 200MHz band, and your signal is a million times weaker than the noise, you can do hundreds of bits per second.


I assume star tracking -- wikipedia seems to confirm

"... and celestial referencing instruments (Sun sensor/Canopus Star Tracker) to maintain pointing of the high-gain antenna toward Earth"

https://en.wikipedia.org/wiki/Voyager_2


Sorry to self-reply, but this Q&A on "Space Overflow" about this specific star tracker is great:

https://space.stackexchange.com/questions/43803/how-did-the-...


Cool finding!


Dung beetles do this too.


that's nuts!

"- African dung beetles orient to the starry sky to move along straight paths

- The beetles do not orientate to the individual stars, but to the Milky Way"

https://www.cell.com/current-biology/fulltext/S0960-9822(12)...

https://www.science.org/content/article/dung-beetles-navigat...


So you can see astrological events do affect us.


s/log/nom/


As long as there's not too much light pollution. Fortunately for the dung beetles, their habitat isn't very urban. However, it's the little examples like this that make me a light pollution dork.


Basically the probe knows where it is because it knows where it isn't. By subtracting where it is from where it isn't, or where it isn't from where it is (whichever is greater), it obtains a difference, or deviation. The guidance subsystem uses deviations to generate corrective commands to drive the probe from a position where it is to a position where it isn't, and arriving at a position where it wasn't, it now is. Consequently, the position where it is, is now the position that it wasn't, and it follows that the position that it was, is now the position that it isn't.


I've heard this as a sample in a song, but can't recall which.


I believe it is a Meat Beat Manifesto track, though I'm not sure which one.


Ok, what is this quote from?



Wait, i don't understand. I was under the assumption that this text was a joke, but now I'm seeing it in reference to air force training materials? Is it a joke there as well or did someone actually write this text seriously, and plan for it's use as intelligible instruction?


There's oodles of references to this online but nothing really I've found so far explaining whether it was ever intended to be taken seriously in the first place. It's hard to imagine anyone doing so.


It's apparently from the 50s, as seen here: https://archive.org/details/sim_electronics-now_1959-03_30_3...


Yes, it's a joke. It's not complete nonsense, though, unlike the Turboencabulator. There's a real explanation buried inside there. ("Where it isn't" is the target position, so it's looking at the deviation between where it currently is, and where it should be. The next time step, "where it wasn't" is the old target position, and so forth.)


That's the voice. :-D


David Tennant's narration of the new W1A series in which the BBC launch a syncapatasatellite?


it probably has both gyroscopes and star-charts for navigation.


Amazing that someone thought up a solution to a hypothetical problem 46 years ago, then fired it 30 billion km away


It's not really hypothetical: losing communication with stuff in space is a very common failure mode and a huge amount of the system design is focused on making it as unlikely as possible (generally the radio system gets a huge priority in almost everything and there are a lot of failsafes built at every level to make it possible to reestablish communication if anything disrupts it).


Indeed. Voyager 2 has in fact been listening via its backup receiver since 1978.


I was amused to learn that if modern satellites lose contact with earth, they go into "safe mode": pointing towards sun, solar panels fully deployed, everything else except telemetry, radio, and temperature management disabled, waiting for further instructions. https://en.wikipedia.org/wiki/Safe_mode_in_spacecraft

Imagine deploying a billion dollar piece of hardware and hoping that it has enough intelligence to keep itself from burning up before you can reestablish contact!


Aerospace has a very high quality standard compared to other industries.

Lots of formal processes capture what would otherwise be informal design decisions elsewhere. In this case, they probably have reams of pages detailing a failure mode effects analysis (FMEA). One mode is “oops, we sent the wrong command” and the document would define the specific design mitigation(s) for that outcome until it reaches an accepted risk threshold.


FMEDA probably. And in recent times, fault tree analysis seems to be better for complex systems.


As far as I’m aware, no NASA standards call for FMEDA. It doesn’t mean a project manager couldn’t levy it, but it’s not often that a contractor adds additional requirements to a gov funded build.


FMEA relies on a really smart person anticipating all the different combinations of failures worth exploring (NxM), not just N or M.

Some failures are fairly common, and individual failures might be fairly inert but have more serious consequences if they are cascaded with another specific failure.. for example, cruise control enable + failure of steering wheel control pad _and_ previously undetected failure of brake sensor/brake light circuit = cruise control stuck ON. Actually, this failure is inert if the cruise control is OFF when it happens. Contrived example but you get the idea ...

I have seen a lot of FMEDA (and other tool) use lately to combat concerns with cascading failure, but not sure what's currently standard at NASA or how they deal with this. I would think cascading failure would be their expected scenario on a 10+ year unmanned mission.


NASA STDs, handbooks, guidebooks, NPDs and NPRs are all open-source. They don’t mention FMEDA, and they don’t generally have a detectability column in their FMEA. IMO they are a little outdated


I've done for NASA what they were calling FMECA and FTA for a subsystem. They had a lot of freedom to tailor the analysis to the situation, and the end result didn't quite match anything established. We addressed detection in some of the FMECA columns which are not traditionally for detection; and events in some of the FTA. It was a contortion of terminology and format to modernize and maximize the value of the analysis given their limited resources and the bureaucracy of what they were allowed/required to do on paper.

Here's how I would describe the possible analysis approaches in broad terms, avoiding terminology that NASA does not officially use.

- Start from the hazard of being pointed in the wrong direction and work backwards to identify the causes, forming a tree.

- Start from the event of commanding the wrong direction and work forwards to identify mitigations or the lack thereof, also forming a tree.

- Start from looking at a component or subsystem, list all the ways it can fail without regard for the application. Then consider the application and work up towards the causes/events.

- Close any gaps between the top-down and bottom-up approaches.


Yes, what you're describing is two different approaches for safety analysis. According to the NASA software engineering handbook [1]

"Software Fault Tree Analysis (SFTA) is a top-down approach to failure analysis which begins with thinking about potential failures or malfunctions (What could go wrong?) and then thinking through all the possible ways that such a failure or malfunction could occur. Fault Tree Analysis (FTA), is often used by the hardware teams to identify potential hazards that might be caused by failures in hardware components or systems, but with the SFTA, the software isn’t considered the hazard, but it can be a cause or contributor when considered in the context of the system."

"The Software Failure Modes and Effects Analysis (SFMEA) is a bottom up approach where each component is examined and all the possible ways it can fail are listed. Each possible failure is traced through the system to see what effect it might have on the system and to determine if it results in a hazardous state. Then the likelihood of the failure and the severity of the system failure can be considered."

But, to the earlier post, these are driven by hard requirements; specifically adherence to NASA STD 7150.2 and NPR 7150.2. Developers/contractors can tailor/waive them with pre-approval but, in general, they tend to go in the direction of less requirements, not more. This may all be moot because I think Voyager pre-dates any of those requirement documents and I'm not sure what existed in the late 1970s.

[1] https://swehb.nasa.gov/


The D aspect of the FMEA I worked on was motivated by a reliability requirement, not by 7150.2. 70's NASA was using FTA and FMEA but avoiding putting numbers on top-level analysis. I imagine they did whatever ad-hoc analysis they thought was necessary for such a highly publicized mission even if it wasn't a separate deliverable.

Edit: The comment you deleted right before I could reply was good! I think people would enjoy and benefit from your description of how the process works if you're willing to repost it.

As you noted the reliability requirement did in fact flow down from an engineering requirement which is why they exceeded the minimum FMEA standards. There's no official guidance on where and how exactly to track that information so they put it in the usual place but in an unusual way. The lack of a standard during Voyager's time probably impacted the visibility of the work more than the substance.


This thread was a good read, thanks.


The Voyager that's flying now is not necessarily the Voyager that was launched.

The hardware is the same, but they've updated, patched, and rewritten the software that's running in it throughout the years.

I'm not suggesting that the failsafe mode wasn't originally considered, and implemented, but simply that it doesn't have to be the case. They could have made changes to it over time.


It’s possible to update the Voyager FSW?


It wasn't a solution for this specific problem. Spacecraft orientations are going to drift over time, periodically rehoming is the simplest way of dealing with it. That it doesn't care whether the orientation drift was natural or artificial is just a bonus.


Sometimes we don’t give enough credit to previous generations.


I only give credit to previous generations. Firm believer that we only understand in retrospect.


Actually there's a couple of work arounds for this problem as they anticipated it all along. My father was Director of Operations at Tidbinbilla deep space tracking station which ran most of the comms to Voyager 1.

I am paraphrasing what he said as a non-technical person: Voyager has both a dish receiver, and a pole antenna. The dish is the usual mechanism for comms but in an emergency such as this they would send commands to the other antenna. To do this they would turn the main tracking station dish up to max, and send a "TURN AROUND!" signal out.

But prior to that they had to alert the local electricity grid, and the local air traffic control to not have any planes flying over at the time!

I guess the Voyagers are too far away for this manoeuvre now.


Oh man that reminds me a lot to Kerbal Space Program, those times I lost communication because of a wrong turn and the antenna/solar panel faced the wrong way


I like to think at least a few NASA engineers come to meetings with some brilliant ideas....that were cooked up at the 1am mark of a 7 hour weekend KSP binge.


You have to agree that "skip to morning" button really works.


A bad case of planet between antenna and space center! Is orbit stable? If so, launch and maneuver into position a huge network of relays until connectivity is restored. If not, watch while probe crashes and burns onto the celestial body. An automatic realignment function would be nice...


How? What mods do you use? I played pretty much without mods, so if you could also mention your whole mods' stack would be cool.

In vanilla, all antennas are omnidirectional and as long as there is a working solar panel on a probe, it can be used as a relay sat.


I exaggerated a bit, it only happened with solar panels.

But it does gives you a similar feeling when you schedule a maneuver on a point where the planet/moon blocks your signal, and you only realize after it's too late


> because the nuclear batteries are near its end.

and we are charging our phones daily....



So they don't have a simulator they run these commands on first?


> In short, it was remote bricked, by giving it commands to rotate a bit. After successfully executing those commands - no further commands could be received, as now the antennas are not facing earth anymore.

I did this exact thing in the small last night - wanted to work on fixing a faulty switch, so my wife and I get on the intercom system on our landline phone so she can tell me when the correct breaker is off.

And of course, breaker #1 is the one that controls the intercom, severing our connection.


Perhaps a better design would be to realign the antenna automatically if it hasn't received any signal from Earth after a week or whatever.


We can certainly do that with Voyager 3!


Who is directly in charge of the program?


who and when was this automatic reset on 15.10 added?


This link from NASA mentions the October 15 date:

https://www.jpl.nasa.gov/news/nasa-mission-update-voyager-2-...


The text and link I provided mention it as well, but I am now not sure, if giving 15.10 as a date was maybe confusing for non europeans (or non germans, I am a bit lost who uses what date format)...


2023-10-15 would be pretty universal.

ISO 8601 works for everyone.

It avoids confusing the Americans who otherwise put the month in the wrong place.

It avoids being ambiguous for everyone who may otherwise be worried that it was written by an American with the month in the wrong place, when the day is less than 13.

2023-10-09 is the 9th of October and it's clear to everyone regardless.

It also has the benefit of sorting chronologically if sorted "by name" when used in a filename as it's largest unit on the left, smallest on the right.


It was confusing to me. Took me a while to realize it was a date and then had to deduce what it represented.

Frankly before your comment I wasn't going to complain because I saw the tantrum you threw when people corrected you on the usage of "bricked" but maybe next time spell the month to avoid ambiguity.


I used to do remote work in firewalls quite often, and after locking myself out once or twice, I came up with a new habit: before making any changes I would schedule a reboot for 5min out which would revert any changes. That way if I locked myself out I could just wait for the reboot and get back in.


I did a similar thing in the early days of my career, but I actually caused an outage as a result.

In this instance, I was adding itables rules to a host. I wrote a script that add all the rules to enable expected network traffic, then set the default policy to DROP. Before running this script, I scheduled another script to be run which would delete all the rules I'd added. I did not remember to set the default policy to ALLOW.

The script runs, everything looks good. Five minutes later, pagers start going off.

Thankfully we were able to remotely power-cycle the host and didn't have to drive down to the datacenter in order to fix the issue.


Standard practice on Cisco routers, where I've worked, is to do "reload 5" before doing dangerous things.

On juniper, it's "commit confirmed".


or safe mode on mikrotik


‘sleep 300 && init 6’ was my go-to, but since then systemd has made firing init 6 unreliable (it won’t trigger a reboot locally if root has an open ssh session, at least on Ubuntu).


And then if it worked for those 5 min before the reboot, you'd redeploy the change 'for real', without a reboot?


My typical workflow was:

- Schedule the reboot

- do my changes

- Make sure everything was working properly

- Go get lunch

- Notice a bunch of pages and alarms about a firewall going offline

- Rush back to my office

- Login to the firewall

- Schedule the reboot

- Re apply the changes

- Test it again

- CANCEL THE FING REBOOT THIS TIME

- Eat my now cold lunch


This used to be my workflow as well. I did make a few improvements though

  - begin change control at 4:55pm on Friday before Christmas
  - schedule reboot
  - paste changes
  - make sure everything is working properly
  - leave security key on desk 
  - go to christmas party
  - firewall goes offline, pages go off
  - remotely log into firewall with phone
  - rush back to office to get security key
  - accidentally type init 1 hanging server
  - discount datacenter remote hands not picking up the  phone
  - rush to datacenter to power cycle server
  - :(


Yeah, there are different kinds of memory in firewalls. Like a running-config and a startup-config. If you just change the running-config and don't commit to the startup-config, when the reboot takes place it'll pull the config from the (non-modified) startup-config instead, reverting changes.


copy run start!


Kinda like changing display settings in Windows.

Changes will revert in 15 seconds....


This is clever, I like it.


"commit confirmed" from Juniper routers is much better


Mikrotik safe mode gets a 3/5 in comparison - it reverts changes you made if you lose connection to the router, so it does it’s job as an anti-lockout mechanism; but I much prefer the atomic nature of a confit commit on junos still.


Every time I read about space engineering, I'm amazed by how contingencies have contingencies. It's so much careful planning and rigor compared to my world. I can always re-compile, re-deploy and regularly realize that my job is not life or death.


Honestly, I'd say most engineering is like that outside of the software world. In the classic engineering disciplines with actual licensures at the end of the pipeline, the responsibility and ethics of this are ingrained into students from day 1. (Budget and importance of the application doesn't always allow for the indulgence of this though, at least to a point.)

This type of thinking also follows from decades of experience.

For some reason the software engineering world largely abandoned esteem and respect for all of the above.


It’s not the licensing or the ethics classes or the responsible thinking or the professors that causes some engineering disciplines to be more carful.

It’s the cost when something fucks up.

If I’m holding my phone near a cliff, and I rely on it for navigation and I’m hours from civilization, I’m a little more careful, not because I’m normally super careful. It’s because — in that specific scenario — losing my phone would cost me so much and the chance of it happening is much more likely.

Space companies spend a little extra because the cost is years of development and billions of dollar evaporating in a few seconds.

And there are software teams in certain industries that dot their I’s and cross their T’s as well.

Even on some dumb CRUD app, if it’s a critical piece of code that the rest of the software hinges upon, you spend a little extra time because the cost of fuck up is so major.

Or you’re launching a product and you have a sign up that will seed your user base, you damn well make sure it works.


I don't understand why this dig is constantly taken at software. Look at how many layers of fallbacks exist even on the average webapp written by junior devs. Optimistic rendering on form submissions, graceful degradation of features, falling back to last cached data, HTTP request retries with binomial exponential backoff and jitter, TCP packet retransmits, ECC corrections on servers, etc.

In cases where fault tolerance isn't as robust, it's for the same reasons as other disciplines you mentioned: budget and importance.


It's also completely untrue that the norm outside of software engineering, I think this perception comes because we only think of the big engineering projects like NASA or building projects, and forget how broad engineering is and can be. I worked for a company that mainly did electrical engineering, and there was plenty of happy-path work that just assumed the error cases would happen rarely or be handled somewhere else. It was also quite difficult to get good change control working, and automated testing was painful and irregular. (In fairness, automated testing was also a lot harder, but we could have worked harder on it and caught a lot more issues early on.)

My impression from friends working in other engineering disciplines is that software engineering works similarly to other fields: the more risk to human lives is involved, the more testing, redundancy, etc is involved.


OceanGate


They said engineering, not randomly hacking at stuff.


I think it comes down to to a couple of things that software doesn't have that most other disciplines do:

Standardisation - in the big 'E' Engineering world, there would be a recognised international standard for Web Apps that ensured/enforced that all Web Apps supported this functionality, or they would not be approved for use.

Another factor is Accountability. A senior Software 'Engineer' would have to take personal responsibility (liability, accountability) that the software product they are producing and/or overseeing met all these requirements and personally sign off that these standards have been met. If the product were to fail at any point and it was determined that the cause was negligence in following the standard, any damages sought (not common, but not unheard of) would ultimately find their way to the accountable individual and their insurance.

In cases where budgets/importance don't allow for this level of scrutiny, there would still be paperwork signed by the producer of the software and the client acknowledging deviation from the standard and waiving any recourse for doing so.


> Standardisation

there is totally standardization. At the building block level. TCP/IP, Protocols on top of that, language standards etc.

Web Apps are complex, why would there be a standard? Just like there's no standard for cars, other than for some of their components like wheels or headlights.


Aeroplanes are complex, but you can bet your life there are standards for those. And cars? Wow - I'm not sure which country you live in, but there are probably as many safety standards for road vehicles as there are for planes!

To continue the analogy from earlier - standards wouldn't mean all web applications would have to be designed, programmed and work exactly the same way, but it would mean that they would need to be formally tested (to an approved test plan), and to use your example, would need to demonstrate that each of those layers of fallbacks (as dictated by the standard and covered in the test plan) operate correctly in order to be certified.

If anything, I think software has a huge advantage over physical world engineering in that testing can be replicated at virtually no cost whenever a change is made to the design. I shudder to think how many cars get trashed in order to meet vehicle safety testing requirements.


you're confusing regulations with standards


No, I'm not.

Here is the Australian Standard for Caravan and light trailer towing components, Part 1: Towbars and towing brackets

https://store.standards.org.au/product/as-4177-1-2004

There are thousands of these documents covering everything to do with transport from the vehicles to the reflectivity of street signs.

The regulation (at least in my state) is that only engineers who are registered as Registered Engineers are permitted to carry out professional engineering services in this state.


There are absolutely very well-defined and detailed standards for cars defined at both a national and international level. They range from excruciating requirements around single components (eg. ESC) through to broad design requirements (like how wide a car).

See the Australian Design Rules (which happens to form the basis of most UNECE and Canadian transport regulations) if you want to see how detailed they are.


I agree with this 100%.


To be honest with ourselves, until we have standardized licensing/accreditation that is fully recognized, we aren't really engineers.

I would love to see a day when redundancy like this is just a standardized, accepted practice rather than a stand-up debate. Easier said than done of course.


You can have this now, just go work in healthcare tech or a bank. The trade off is no innovation, career boosts, professional accomplishments, or projects under $10M.

Clients who want NASA quality can have it if they bring NASA budgets and timelines.


Oof. Point well made. That I certainly don't see changing any time soon.


I did an engineering degree but I have to say, the ethics imparted on me were basically “be diligent and don’t build anything that harms people by accident” which… really ought to be, like, table stakes for living in society, right?


As you've stated the oath, it's certainly glib, but it's not table stakes because it's not a mere commitment to good intentions or a kind heart. Engineering ethics are not a commitment to good intentions. To take that pledge seriously, you need to be able to trace all your requirements and consequences in order to analyze, prevent and verify you've prevented potential danger without breaking what you've built. Most people in society would not succeed at this.


It ought to be but it's not. Plenty of sociopaths out there who think nothing of poisoning children with heavy metals in toys if it's cheaper. How many businesses neglected software system security, only to compromise the personal information of thousands, millions of people? It's not like there are consequences. As far as I know, no one was ever arrested.


>For some reason the software engineering world largely abandoned esteem and respect for all of the above.

it's a lower bar for entry, any kid can run a compiler -- it's harder to acquire a bulldozer and planning permits.

similarly if you look at 'diy' or 'garage' engineering you can find all sorts of hazardous/poorly-built/never-should-have-existed death traps. How many people have died in recent years from fractal burning?

it's still engineering -- they're building their own tools -- but it's within a realm (DIY/maker) that historically has undersold the dangers inherent with the things.

Why? Mostly because they're self-taught, mentorless, and without the direction within their education to be taught the importance of engineering rigor, similar to the kid given the compiler who starts making forkbombs and goofy dialogs to prank their friends.


> Honestly, I'd say most engineering is like that outside of the software world.

Add civil engineering to that nowadays - both buildings and roads.

Sure, there are regulations and licensing, but quite often the entity financing the whole thing cares little about such things.


> For some reason the software engineering world largely abandoned esteem and respect for all of the above.

That's what happens when there's no liability. Even if you're a billion dollar corporation, you can just slap some standard legal boilerplate disclaimer on the license agreement and absolve yourself of all responsibility even when hackers pwn the private information of your customers. They can't complain since technically the contract says there were no guarantees.

Some version if this legal boilerplate can be found in virtually all software licenses, including open source ones:

  THE SOFTWARE IS PROVIDED "AS IS",
  WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
  INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
  FITNESS FOR A PARTICULAR PURPOSE, TITLE AND NON-INFRINGEMENT.
  IN NO EVENT SHALL THE COPYRIGHT HOLDERS OR ANYONE DISTRIBUTING
  THE SOFTWARE BE LIABLE FOR ANY DAMAGES OR OTHER LIABILITY,
  WHETHER IN CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF
  OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
  IN THE SOFTWARE.
Start holding developers and especially corporations directly accountable and liable for any problems and I guarantee they'll start respecting it the very next second.


Move Fast And Break Things^TM

Jokes aside I think it's mostly a value/cost thing. NASA's software has different requirements and failure scenarios than most software developers (in this context I will not call them software engineers) have to care about. Verifiable correctness is harder to predict, and in most devs' roles it's easier to just try something and see what happens, rather than know what'll happen up front.


Errors in software rarely ever matter and even when they do, can usually be trivially corrected.


Software does not wear out like most physical components, but they often cause failure in interaction/coordinating between subsystems.

As the amount of coordination increases, the number of failure modes tends to grow quite fast. That's why software failures in physical, safety-critical systems are not trivially corrected. There are a lot of second order effects that need to be considered.


> Software does not wear out like most physical components.

It fails like buildings near fault lines, because the ground moves under them. Think broken dependencies, operating system obsolescence, et cetera.


I like this analogy. Although your example focused on software-centric coordination, I think it's important to also extend it to non-software systems.

An apropos and famous example is the Ariane 5 rocket mishap. The same validated software from the Ariane 4 was used, but the hardware design changed. Specifically, the velocity of the Ariane 5 exceeded that of its predecessor and exceeded the 16-bit variable used.


Honestly I can't imagine someone who hasn't been living under a rock for the last half century could say this. Just one example: Knight Capitol was the largest trader in U.S. equities, with a market share of 17.3% on NYSE and 16.9% on NASDAQ in 2012, right up until August 1, 2012, when it lost $460 million and 75% of its equity value because of a software error. What was left of it was acquired in December of that year.


Except when they do matter, like the Therac-25 deaths or those 737 MAX crashes.


If you're referring to MCAS in 737, the software itself wasn't the main problem; I'd say that the main problem was that it wasn't even a documented feature (let alone the engineering of the system itself).

The pilot couldn't even turn MCAS off originally. That's not a software thing, that's a "who the F designed this" thing.


> 737 MAX crashes

To imply this was a software bug is a pretty silly representation - the system was poorly engineered and didn't have proper contingencies for sensor disagreement. This is pretty clearly a design/engineering error with a software component.

Besides, the guy said "rarely ever matter" for a reason, not "explicitly never impact things"... Bit of a silly comment from you IMO


To view software in isolation is an equally silly representation. In the physical world, software is part of an overall system that needs to be considered holistically. Most major safety-critical mishaps are the result of several failures, often across different domains.

In the case of the 737MAX, the software was a design around a physical constraint; that doesn't mean the software doesn't matter. Most software is designed as a workaround of a certain physical or mental constraint.


It’s not life or death, but time spent dealing with errors - debugging, the direct effects, understanding full impact - isn’t a resource we can get back.


It's funny you say that, because designing systems that work extremely well, have contingencies upon contingencies, and can be relied upon (e.g. as a life-critical system) is so time consuming and (I imagine) mind numbingly boring (e.g. reviews upon reviews of white papers to ensure that the system spec is scientifically sound) that I'd guess time is the last thing you'd get back from writing NASA-style applications.


I find myself thinking about that a lot - mainly "how many more hours would have needed to be spent at stage A to avoid the hours being spent now to recover from problems our software is currently causing". And often if I'm honest with myself it's hard to see that the extra investment of time earlier on would have necessarily resulted in a net productivity gain. It would however likely be a less stressful way to work (building fire-proof code rather than putting fires out all the time), and rather more satisfying. As an engineer of any sort I think it's perfectly reasonable and justifiable to want to produce something of quality even if it takes longer and the consequences probably won't be that terrible if you just release the first thing you can slap together. Unfortunately others are almost entirely motivated by the (not entirely irrational) fear of what happens if you don't release something quickly enough.


> For some reason the software engineering world largely abandoned esteem and respect for all of the above.

The main contingency with most software is that you fix it.


I like when people mention that they're "computer doctors." I have some stressful migrations that require a lot of planning and could cost a significant amount money if botched but I can't imagine the additional stress of someones life being at my fingertips.


It's tricky.

Many moons ago when I was hands-on and stressed about migrations & config, my team lead at the time would say exactly the same thing - his wife is a doctor and her job is way more stressful - People die. And I bought into it as a relief for a while.

But... I work on a payroll system. My team does impact people. Mistakes can have important negative consequences to real live individuals - from stress invoked in trying to call help centre and fix their paycheques, to disconnected utilities if they don't get paid correctly/timely, to other downstream consequences.

Any number of other IT systems have significant consequences - e.g. airline ticket systems, airbnb bookings, etc. I feel the "nobody died" is a double-edged sword: it can help relieve people of the daily sense of artificial stress, urgency and grind that management may impose; but also builds a false dichotomy / unreasonably binary threshold on when our job matters / impacts ...


I think one of the greatest contributions launch window aerospace neurosurgeons make to society is the way they cause nobody else to ever feel stress in any way.



It’s so inspiring when you see how these things are just built to last.

quote: “In the past, engineers have compared keeping the probes operational to keeping an old car running. The tech is severely outdated, yet it keeps ticking over – a trend often seen in the spacecraft of past decades.”

At some point us humans will probably simply have forgotten how to maintain them.


You should read "The Machine Stops" by E. M. Forster. Also, "Pump Six" from "Pump Six and Other Stories" will also do fantastic job of diving into this "forgetting how to maintain them" reality.


Pump Six really nails that feeling of "this thing we don't really understand keeps filling the log with warnings we don't know what to do about, let's ignore them and pray it just keeps working."

Any similarities with the real world are surely coincidental.


Many apps have warning/errors that are undecipherable from the very beginning, let alone 20 years later.

Or only make sense when looking into source code that is long gone


The worst is when the log line is constructed in a way that makes it really hard to find the source. Source code file name and line number is ideal but a tag like on Android auffices.


And to say that people are paid to find traces of attacks in logs, while after 5 years, everyone ignores everything that’s in the logs.


The most incredible thing about The Machine Stops by E. M. Forster is how casually prescient it is - first published in 1909!

Instant messaging, video calls, the internet…


> "forgetting how to maintain them" reality

I serve the Omnissiah.


Awesome, thank you! Just as I was again running out of things to read.


Ringworld


Fire upon the deep, where space ships runs on a future version of unix and only one guy knows what the unix epoch means.


I really love that series. It's been a little bit since I last re-read them but there are certain concepts/ideas in them that I still think of from time to time.


I think the first half of the first book is the best. Much less interested in Tine's world


I don't recall that in that book. Maybe you're thinking of A Deepness in the Sky? I haven't read that one yet.


I think the reference is in Fire: It’s an offhand line about an ancient timekeeping system which the modern engineers mistakenly believe is calibrated to humanity’s first steps onto another celestial body.


As A Fire Upon the Deep is one of my favorite books (it's been a while since I've read it- my copy is currently on tour), I'd like to chime in and say I remember this reference, but I believe it's in A Deepness In the Sky, which goes more into Pham's backstory. It's definitely one of these two books though.


> Take the Traders’ method of timekeeping. The frame corrections were incredibly complex—and down at the very bottom of it was a little program that ran a counter. Second by second, the Qeng Ho counted from the instant that a human had first set foot on Old Earth’s moon. But if you looked at it still more closely. . .the starting instant was actually some hundred million seconds later, the 0-second of one of Humankind’s first computer operating systems.

- Chapter 17, A Deepness in the Sky


At a conference I met Vernor Vinge and told him my entire career was basically because I read his books in high school. He was very happy.


I loved that aspect of it - it's becoming more and more true as we build more and more frameworks/abstractions. Once we got to Kubernetes and some of the modern web frameworks, the notion of "Programmer-at-Arms", the one-in-thousands master developer who'd actually dig into the depths of these abstractions, made perfect sense!


Yes, that bit is in "Deepness".


Keep in mind if you start the Ringworld series there's also a tie in series that starts 200 years before Ringworld (Fleet of Worlds) and both end with the same last book. Niven and M. Learner wrote so many books...


I think the Man-Kzin Wars are also somehow related, but I'm not sure if it's technically in the same continuity or not.


Same universe, not sure if same characters. There are like 20 books, and I think some of them are community written.


Loved this series when I first read it and it will always hold a special place in my heart but I did reread a few months ago and the way Teela Brown (and some other women) is talked about/to left me feeling very uneasy.


> At some point us humans will probably simply have forgotten how to maintain them.

Nah, these systems are simple and incredibly well documented. A ton of people have operated them, too. They'll be fine.

I'd expect something like that to happen to a university cubesat lol.


We can only hope that because they're so well documented, we can work around any "dead hardware" or "dead media" issue. Like, I hope the Voyager manual doesn't say "see disk 2 for firmware", and disk 2 has turned to dust 10 years ago.


The Foundation series covers this as well though I can't really recommend the book series. I tried a re-read when the TV show came out and felt pretty icky with how women were portrayed in the books. Also they aren't as good I remember. The TV completely diverges from the books but in a good way IMHO. Normally that bothers me a lot but after rereading the first book again I think I prefer the TV show.


I loved the series as young teen but rereading the first book was a disappointment. I'll be checking out Foundation after I finish Silo.


Space opera is my favorite genre but I've failed to get through the Foundation series probably 20 times now so this may be terrible advice but it's recommended to not read them in publication order by Asimov himself.

This has a good breakdown: https://www.reddit.com/r/Asimov/wiki/seriesguide/

edit: Somehow I got Foundation mixed up with Banks' Culture series. I think I have gotten through most of Foundation if not all but I've had a hard time with the Culture series, there I usually start with Player of Games..


The Culture series is good but I've struggled with getting through it all as well. If you like space opera I can highly recommend the Honor Harrington series, the first book being On Basilisk Station [0]. This has held up for me for well over a decade and I've reread the entire series (~14 books IIRC) at least 4-5 times. I've heard it described as "Horatio Hornblower in space" but I never read that series so I can't speak to that.

[0] https://www.goodreads.com/book/show/35921.On_Basilisk_Statio...


Incidentally, On Basilisk Station is a free e-book at the publisher's website [0]. They also have an online HTML version [1]. So you can try the first book in the series to see how you like it before purchasing any of the others.

[0] https://www.baen.com/on-basilisk-station.html

[1] https://www.baen.com/readonline/index/read/sku/0743435710


I'm a big fan of the audiobooks for this series as well, the narrator does a very good job IMHO.


I accidentally read the 2nd book first, and going back to read the first afterwards was kind of a struggle. You could totally just skip the first and read the other two.


It will probably readjust. And power supply is expected to be dead ca. 2025 anyway.

OK, OK, if the Klingons find it now, then it'd be a shame not to get some measurements. (The cameras, however, are off since decades.)


They made some updates and expect it to go through 2026

https://voyager.jpl.nasa.gov/news/details.php?article_id=129


>> It will probably readjust. And power supply is expected to be dead ca. 2025 anyway.

> Are you both misunderstanding that? Your link says:

> Launched in 1977, the Voyager 2 spacecraft is more than 12 billion miles (20 billion kilometers) from Earth, using five science instruments to study interstellar space. To help keep those instruments operating despite a diminishing power supply, the aging spacecraft has begun using a small reservoir of backup power set aside as part of an onboard safety mechanism. The move will enable the mission to postpone shutting down a science instrument until 2026, rather than this year.

> Switching off a science instrument will not end the mission. After shutting off the one instrument in 2026, the probe will continue to operate four science instruments until the declining power supply requires another to be turned off. If Voyager 2 remains healthy, the engineering team anticipates the mission could potentially continue for years to come.

Going from 5 science instruments to 4 in 2026 is hardly "dead."


extra > on 2nd line


Let's just hope it doesn't make it to the machine planet.


That’s Voyager 6, not Voyager 2.


This seems like a multi-reference to me:

(1) Dune, (2) Turrican, others?


Most directly in the context of Star Trek (Klingons mentioned above), it will be a reference to Star Trek: The Motion Picture. (Which is about the hypothetical Voyager 6 probe's interesting history.)


Dave from EEVBlog just visited a facility communicating with Voyager 2:

https://www.youtube.com/watch?v=586Zn1ct-QA

https://www.youtube.com/watch?v=vUvzgZt1Vug

There's a part 3 with a tour of the complex.


I was fortunate to have the opportunity to visit Goldstone, up in the California desert on Fort Irwin. It's not open to the public very often.

I got to visit most everything there, including the 70m telescope. It was just a cool space tech nerd day of tours, presentations, and sunshine.

The dichotomy of the 70m antenna is interesting is that it broadcasts 450 kilowatts of power out into space, but has to receive and decode, "as small as 1 billionth of 1 billionth of 1 watt" signals from the space craft.

One of the reasons its on a military base is to restrict the airspace above it so that they don't accidentally cook some aircraft that happens to overfly the antenna when it's transmitting.

It's truly astonishing they're able to pull that off, frankly.


450 kilowatts? Is this the most powerful transmitter on earth? Where does it source the electricity for this?


In the general category of “most powerful radio transmitter”, it doesn’t even rank in the top 5. The biggest are 1MW+.

Many industrial sites use 450kW or more (see any bauxite processing plant) so are coupled with a power plant or have agreements for dedicated generation at nearby plants.


It's a re-upload from 2017, just in case anyone else thinks they are going crazy like me.


Why would someone reupload their own old videos?


"NOTE: This video is a re-release from the EEVblog Discover channel from 2017, to hopefully find a new audience."

Worked on me, I guess :-)


> "the antenna on the spacecraft had been pointing two degrees away from the Earth [...] left it without the ability to receive commands or transmit data [...] NASA reckons the situation is temporary [...]"

I wonder how it's temporary. Does the probe have a re-targeting function? The answer is in the original statement:

> "Voyager 2 is programmed to reset its orientation multiple times each year to keep its antenna pointing at Earth; the next reset will occur on Oct. 15, which should enable communication to resume. The mission team expects Voyager 2 to remain on its planned trajectory during the quiet period."


I wonder why the reorientation is so infrequent? Is it a long process or a strain on hardware that you wouldn't want it to happen every day or even every month?


Reorienting requires using a reaction wheel or propellent to move yourself.

Propellent is finite, so you want to use it as rarely as possible.

A reaction wheel is by itself infinite (assuming it doesn't break), but eventually it saturates and you need to desaturate it, which basically means spinning the wheel the other way while spending propellent to maintain position.

All of this is to say, reorientation is an expensive process especially if refueling isn't an option.


Reaction wheels only saturate by absorbing external acceleration, though.

You can reorient as much as you want with a reaction wheel, and the only cost is electricity.


>reorientation is an expensive process

That’s true, but a failsafe automatic reorientation mode after two weeks with no communication from Earth might be a useful feature


On the timescale of decades, does a 2 week vs a 10 week waiting period make much of a difference?


>On the timescale of decades, does a 2 week vs a 10 week waiting period make much of a difference?

I agree. It probably doesn’t make much difference now. There are very unlikely to be any important data that will be lost. The mitigation process they already implemented seems prudent and sufficient

During earlier parts of the mission, like planetary encounters, it would make a difference. Even a two week wait could be much too long in those circumstances.


Presumably during encounters the probe was in a different control scheme to keep its instruments pointed at the planet, as opposed to keeping its antenna pointed at Earth.


It costs fuel to reorientate


In addition to the points made by sibling comments, there is always a chance something going wrong in the reorientation, so you do not want to do this more than necessary


> it is programmed to recalibrate its position a few times a year. October 15 is the next scheduled reset.

Curious to know how this recalibration actually works. Any explainer that anyone can point to would be appreciated. Thanks!


Good video on the topic - there is a sun sensor on the dish - looks for the brightest object and orients to face it. https://www.youtube.com/watch?v=NbsHgE89qO4&t=340s


For all the distance Voyager has covered, our Sun is still the brightest object in its view? That’s incredible


Approx one light-day out. The nearest star is 4.2 light-years out.

Doesn't even matter if voyager is heading towards it or not, it's still crazy far away. Voyager is still on our doorstep as far as interstellar distances go.


The space between stars is truly immense. The sun is still 2,000 times closer to Voyager 2 than the next star


How bright does the Sun appear compared to other stars at a distance of 32Bn km?

Here is a photo from Voyager 1 at a distance of 4Bn miles:

https://photojournal.jpl.nasa.gov/catalog/PIA00450


Trying to see if I can work this out.

Voyager 2 is 160.7 AU.

Light falls off in brightness to the distance squared. So the sun will be 160.7^2 = 25824.5 times fainter for Voyager 2 than it is from Earth. (Since Earth is at 1AU)

The apparent magnitude of the sun from Earth is -26.72. Each step in magnitude is multiplying by 2.512. (2.512^5 = 100, so 5 steps of magnitude is a factor of 100).

log2.512(25824.5) = 11.0295.

11.0295 + -26.72 = -15.6905.

The apparent magnitude of the full moon is only −12.74 (lower is brighter). So for Voyager 2 the sun is still several times brighter than we see the moon. The sun is still many many times brighter than the next brightest star in the sky, Sirius, which has an apparent magnitude of −1.46.

Sources: Voyager 2 distance is https://voyager.jpl.nasa.gov/mission/status/ all else is Wikipedia.


Superb! Thanks for the link.



Excellent link - thanks

I skimmed though the Voyager document and it seems to have very good coverage of overall telecommunications system.

For the topic of the periodic calibration the following is all I could spot

>> Four 7-hour and two 0.5-hour attitude control calibration maneuvers are performed per spacecraft every year, each requiring 70-m station downlink coverage to ensure uninterrupted downlink telemetry.

While this is interesting in itself, it merely states the schedule but doesn't satisfy my curiosity about the exact mechanism used to do the recalibration.

Thanks nevertheless. Interesting reads here.


The Voyager Neptune Travel Guide mentions things like the Canopus Star Tracker and the Sun Sensor.

https://ntrs.nasa.gov/citations/19900004096


Thanks!


Not a rocket scientist; but I have tuned in a TV. I imagine it is simply programed to turn a few degrees then turn back to wherever it saw the strongest signal from earth.


Makes sense.

So I presume it uses its thrusters to impart a very small spin on one axis, and then on an orthogonal axis.

A mechanism records the signal strength as it sweeps all angles, and once the optimum direction is determined, the thrusters are fired in just the right way to counteract the spin and bring the craft to a halt at the optimum orientation.

Given this was programmed decades ago - the electromechanical system that does all this jugglery and runs reliably for so long would be a great case study for systems design.

Even the programming that ensures that this routine is triggered without fail every few months must also have gone through intense reliability testing.


doesn't even need thrusters if it has reaction wheels that are still operational


But....How does it know it's the earth?

It is billions km away.., is the earth that noisy compare to solar wind and cosmic rays?


Voyager has a star sensor that is meant to align with the Sun and Canopus (IIRC). If both are properly aligned it means it's pointed in the right direction. At its current distance the beam width is wide enough to cover all of Earth's orbit. So really it just needs to point at the Sun anymore.

The carrier signal from Earth is also powerful on a particular frequency and polarization. While there's definitely noise at the receiver it's looking for a very specific signal so can filter out everything it's not expecting. We do the same thing on the Earth side, filtering out noise to recover the very weak signal received from Voyager.


At 32 billion kilometers distance, 2° off target means the communication beam is missing Earth by about 1.1 billion kilometers.

EDIT: This is about 7.4AU. If Jupiter and Saturn were in a line from Earth right now, this distance from Earth would be about the halfway point between the two gas giants. So no, we also won't be launching a rocket to go catch the beam just to re-establish comms.


Hope the re-calibration works. Would be a sad way to lose contact after all these years.


True, but they only had enough power on-board for it to last until 2025 anyway, so it's already on its last legs.


The power needs are mainly for radio communications back home?

Or that includes navigation / propulsion / course corrections / reorientation also?

(There is not enough "solar power" that can be harvested at that distance I presume)

If we spread out the communications to be less frequent and say bring it down to essentially a heartbeat signal once a month ..would it prolong the service life. Mostly for emotional reasons at that point :)


Voyager 2 is approximately 20 billion km from the Sun

Earth is approximately 150 million km from the Sun

Sunlight intensity falls off with the square of distance (ignoring any additional small losses from space dust / scattering from gases etc), so twice the distance = a quarter the solar flux. At the Earth it's ~1361 watts per square meter.

Voyager 2 is approximately 133 times further from the Sun than Earth is, which means it receives optimistically 1361 / (133^2) = 0.07694 watts per square meter.

I found a JPL article [1] that says the RTG onboard Voyager produces 40% less power than it did at launch, and the Wikipedia article [2] says it produced 470W at launch, which means it makes ~280W now.

Wikipedia [3] suggests the solar panels available at the time of Voyager's launch in the late 1970s could convert ~10% of incoming solar power to electricity. Modern panels bring that up to 30% but the designers of Voyager did not have access to time travel.

So at present distance Voyager would need approximately 36000 square meters of solar panel to produce the same amount of power.

[1] https://www.jpl.nasa.gov/news/a-new-plan-for-keeping-nasas-o...

[2] https://en.wikipedia.org/wiki/Voyager_2#Power

[3] https://en.wikipedia.org/wiki/Solar_panels_on_spacecraft#His...


There have already been some questionable football field size comparisons in this discussion thread, but in this case the comparison might add intuition—that is about 7 American football fields worth of solar panels.


The 'battery' in this case is an RTG, so the amount of power drops whether it is used or not.


It might be possible to use the remaining electrical heaters as some kind of crude thermal battery (assuming they have any heaters still running, they already shut down the heaters for most of the scientific experiments).

Simply turn the heaters off before transmitting and keep any transmission periods short enough that the electronics don't get too cold.

JPL probably have a bunch of tricks like this ready for when power levels drop. That 2025 estimate is 10 years old and I'd be surprised if it's final.


They made some updates and expect it to go through 2026

https://voyager.jpl.nasa.gov/news/details.php?article_id=129


> The probe is currently around 32 billion kilometers from Earth, and gets 15km further away every second.

I beg anybody to rephrase it understandingly with using some units similar to football fields. Is it possible to launch a little cheap rocket with a transmitter just to correct Voyager's position?


It's about 3.5 trillion NFL football fields away. 15km/s is about 33,000 mph - more than 10x the speed of sound, and faster than a bullet. Does that help?

We are talking about distances that are so big, there is no comparison that makes sense. Nothing else IS that big. The numbers are literally "astronomical". If you're struggling to wrap your head around it, you're doing it right.

"Space is big. You just won't believe how vastly, hugely, mind-bogglingly big it is. I mean, you may think it's a long way down the road to the chemist's, but that's just peanuts to space." -- Douglas Adams


When you start getting beyond the "thousands of football fields" it starts becoming difficult to conceptualize. In this case, even though GP was asking for football fields, it may be easier to visualize it as flying nearly 800 thousand times around the entire equator of earth. And voyager is going around the equator roughly once every 45 minutes or so.

So to catch up, you would have to be faster. Let's say you were able to travel around the equator in 15 minutes, so you're gaining 30 minutes per equator. If my napkin math is right, it would take you roughly 45 years to catch up to voyager.


Your parent was not suggesting to catch it, but rather to launch a transmitter to intercept Voyager's radio beam as a relay. Unnecessary, but creative.


My math also didn't account for the fact that voyager would continue traveling in those 45 years you'd be trying to catch up, so it would actually take longer to catch up to it anyways.


> When you start getting beyond the "thousands of football fields" i

I feel like that line is somewhere between 5 and 15 for americans, and not "thousands". And probably at around "oh the handegg one, no, I have no idea how big one is in the first place" for rest of the world


As an American, I've never seen more than maybe 2-3 football fields next to each other. They're usually stand alone items so that is even rare. Imagining them in plural at all is something people likely do with a large degree of error is my guess, even for us American's that are familiar with the size of a single field. It's a awfully small unit for anything related to space. Even a kilometer which is ~11x as long as an American football field is a small unit for space.


For order of magnitude descriptions, American and Association Football fields can be treated as approximately equivalent lengths. (The former is slightly larger counting the endzones as part of the size, slightly smaller if not counting them.)


> 800 thousand times around the entire equator of earth

This probably wasn't your intention, but putting it in terms like this, for me anyway, actually drives home just how short a distance the Voyager probes have travelled.


I just wanted to make the distance something that could be understood and processed. Sounds like it worked!


You definitely did a good job. I'm an avid sci-fi reader, write it as a hobby, spend a not-insignificant amount of my free time reading up on space news, and even have a degree in mathematical physics; this is the first time in a long time that an analogical choice of units has had an impact on my perception like that. Well done!


> 15km/s is about 33,000 mph - more than 10x the speed of sound

Well, technically, 15km/s IS "more than 10x the speed of sound". An average car, is, TECHNICALLY, more than twice the size of a bicycle.


> Well, technically, 15km/s IS "more than 10x the speed of sound".

Technically, the speed of sound depends on the medium, and 15km/s is much slower than the speed of sound in interstellar space. (Which the sources I can find give at ~100km/s.)


honestly I was just shooting for easy round numbers. "More than 43x the speed of sound" doesn't have the same ring to it. And besides, as we all know "technically correct is the best kind of correct!" :)


So what was wrong about "40 times the speed of sound"?

Also, I don't particularly like the speed of sound for this comparison. Most people think of speed of sound as speed of sound at about sea level pressure, in gas composed of around 78% nitrogen and 21% oxygen and at roughly 25C temperatures. But the speed of sound is highly dependant on the medium and its temperature and pressure. There actually can be sound waves in space (pressure waves in interstellar gas resulting from various astronomic phenomena) and they propagate at very wide range of speeds, typically somewhere between 10 and 100km/s.

The main reason to use "speed of sound" is because important things change when objects travel at little below or above speed of sound in the medium they are in. But this is only useful in relation to the actual medium the object travels through.

One place where it trips people up is when they are talking high altitude airplanes or rocketry. They are talking about something traveling at "X Mach", or "X times the speed of sound" and then I try to figure out if they mean X in relation to the speed of sound up there or the speed of sound at sea level. Just a nightmare trying to use it to convey speeds even within confines of our atmosphere.


A nice feature of using the speed-of-sound as a measurement unit is that people know how difficult it is for aircraft to achieve it. So it makes it clear how much faster these things are going. We don’t have anything comparable between the speed-of-sound and the speed-of-light, do we? I suppose you could use escape-velocity, that isn’t something as many people know, but does I guess get you closer to the speeds in question.


> A nice feature of using the speed-of-sound as a measurement unit is that people know how difficult it is for aircraft to achieve it.

But it's not aircraft ? It's trivial for spacecraft to achieve it


There is nothing trivial about it. The only reason Voyagers are traveling so fast is we were very lucky at the time and got gravity boost from pretty much everything we could get gravity boost from.

But yeah, it is not comparable as the challenges for spacecraft and planes are completely different.


> The only reason Voyagers are traveling so fast is we were very lucky at the time

“Lucky”, only in the sense that (1) completing a large government project on time, and (2) not having some kind of disaster (particularly, at launch) screw up the mission require a certain degree of luck of luck on top of planning and execution (though, not relying completely on that luck is also why there were two Voyagers): we got all the gravity boosts because the mission was planned around an alignment that enabled it to do that and visiting each of the outer planets (which was really the main goal; the beyond the solar system part was gravy.)


"Lucky" because the planets LITERALLY aligned for this to work. This kind of alignment only happens very, very rarely.

The New Horizons probe was launched at much faster speed than Voyagers, actually beating the record of the absolute fastest launch in history, but because of not getting those gravity assists it will never overtake Voyagers.


Trivial seems the wrong word here. Picking your nose is trivial. Space travel is exceptional.


Well, getting to space is the hardest part, once you're there breaking speed of sound is trivial


I think Mach numbers are always given for the situation the aircraft is in at the time


That's the idea. But quick survey of people in my vicinity confirmed most people think about Mach numbers as just another unit for speed of sound.


Good description.. reminds me of Vernor Vince’s description in his novels.

We are truly lost in a “The Deep” … as in absolute nothingness


It’s crazy when you consider that the sun is 8 LIGHT MINUTES away from earth. Light can go around the entire planet hundreds, no, thousands of times in that same period. Space is huge. Incredibly huge.


... and it's not even close to being that far away from the solar system.


Okay, if you tossed a football in 1977, and you tossed it really hard, like with the force of 5,000 Joe Namaths, then the football would have traversed 350 billion football fields (that's 44 stadiums per human on Earth) and the football would be speeding across 164 more fields per second; that's 7,380 in the time I took to post this comment.

*Joe forces estimated


Once time travelers have conquered all the big challenges (i.e. kill Hitler, buy Apple in 1980, stop Skynet), they can go back and make sure Sir Isaac Namath discovers the law of gravity.

It's the 24th century version of jacking with Wikipedia.


People keep going back to kill Hitler. But the resulting future is a butterfly effect nightmare result. So, people keep going back to save him. That's why he had so many close calls.


The antenna is pointing two degrees off course, so you wouldn’t need to send a spacecraft all the way to catch up with Voyager 2 and fix it, you’d just need to launch a relay spacecraft to the nearest point that intersects the signal beam. If Voyager 2 is about 32 billion km away, that point would be only about 1 billion km away, assuming the signal is a straight line.

“Only”.

It’s probably not worth it.


You mean sending this antenna to space?

https://megous.com/dl/tmp/95ce96af5966be24.png

:)


It looks enough like an umbrella.


That sounds like too much, but turns out that 2 degrees is indeed about 1/30 radians.


> Is it possible.

Using current technology we could probably make an object go faster than that so yes, it would be able to catch up.

However, we'd probably just put better instruments on this new object and make that the priority.


> Using current technology we could probably make an object go faster than that so yes, it would be able to catch up

We could achieve slightly greater speed immediately after launch but we wouldn't be able to exploit the planetary gravity assists that accelerated the Voyager spacecraft.


It is one trillion baker's dozens times the height of 1 fl oz of 200 proof ethanol in a quarter inch glass tube heated to 100F.


Here's a 'wrong' but possibly helpful comparison, in the spirit of football fields:

32 billion kilometers is about 100 times the distance a satellite travels from earth to Mars. [1]

That Earth-Mars trip is estimated in the same article to take 4 months, so figure 400 months or 30+ years to shoot another satellite out to reach Voyager 2.

This is ignoring planetary slingshot math, the extra speed to 'catch' voyager 2, and surely lots of other details. Personally I find years and "mars" to be more intuitive in this case than trillions of football fields.

[1]https://mars.nasa.gov/mars2020/timeline/cruise/#:~:text=The%....


0.5003% of the maximum velocity of a sheep in a vacuum (1)

(1) https://www.theregister.com/Design/page/reg-standards-conver...


> I beg anybody to rephrase it understandingly with using some units similar to football fields. Is it possible to launch a little cheap rocket with a transmitter just to correct Voyager's position?

please tell me you are being sarcastic!


It's been travelling the width of the earth every 14 minutes for the last 47 years.

To reach the point 2 degrees from earth would take 1.64 years at that speed.

To reach that point before October 15th it would need to travel about 9x faster than falcon 9 second stage or almost twice as fast as the fastest spacecraft in history.

But it would need significant additional time and fuel to slow down such that it didn't immediately blow past that point and become useless, so it would need an even higher speed.


How to tell if somebody is an American


-


Then they would also use football fields (but think of soccer fields).


Only Americans call football ‘soccer’


> Only Americans call football ‘soccer’

Australians call it "soccer" too. Disambiguates it from Australian Rules, Rugby League and Rugby Union


The word “soccer” actually comes from England! From Wikipedia:

The term soccer comes from Oxford "-er" slang, which was prevalent at the University of Oxford in England from about 1875, and is thought to have been borrowed from the slang of Rugby School. Initially spelled assoccer, it was later reduced to the modern spelling.

“Football” almost always means soccer (association football) in the UK, but there are also things like rugby football and Gaelic football.

Edit to add: you need to disambiguate when other forms of football are popular (eg at Oxford university) but these days soccer is the most popular sport by a huge margin.


If you ignore much of the majority English-speaking world, then yes, that would be accurate.

https://brilliantmaps.com/football-vs-soccer/


Kind of tangential, but I've been watching a lot of original Star Trek recently, so I was curious about how far this was in lightyears, probably because of the Enterprise's proclivity to run in to Voyager.

If it's 15 billion miles away (sorry for my Freedom Units), it is 22 light-hours away, or 0.0026 light years away (unless my Google-fu is way off). If we could move at the speed of light, which we can't, it would still take nearly a day to get there. So if we were on the Enterprise moving at Warp 1, it would take a day to get there and reorient it back towards Earth. If we could move at Warp 10, we'd have already been there and fixed it.


22 hours is assuming you only wanted to reach that location (and not stop to visit the craft) and you had instant acceleration up to c for your velocity, which would be infinite G forces and turn your cells in to quantum foam.

More realistically, even if say we did have a light speed capable craft, we'd be limited by the forces for accel and decel to have a starting speed of 0 relative to Earth, and ending speed of 0 relative to Voyager. The limits would be determined by the withstanding capability of the bodies of the pilots, or the mechanical strength if it were another drone.

I don't feel like doing the math properly, so I asked GPT4 this question:

> let's say we wanted to move a human astronought in a spacecraft to the current position of Voyager 2 which is 22 light-hours away. We want to get there as fast as possible, and arrive at a co-moving frame with Voyager, which has a relative velocity of 32000 miles per hour moving away from the Earth. Assuming the astronaut can withstand a g-force maximum on their body of 6G, and using that to inform the maximum accelleration and deceleration for the voyage, how long does it take to reach the Voyager 2 co-moving frame?

Response:

Firstly, let's convert everything to consistent units. Here we'll use hours for time, miles for distance, and miles per hour^2 for acceleration.

1 G is approximately equal to 22 miles/hour^2. Therefore, 6G is approximately 132 miles/hour^2. The distance to Voyager 2 in miles is 22 light-hours * 670616629 miles/light-hour = approximately 14753561838 miles.

Assuming we accelerate at 6G until halfway, then decelerate at 6G for the second half, we can use the kinematic equation:

d = 0.5 * a * t^2

Solving for t when d equals half the total distance gives:

t = sqrt((2 * d) / a)

Substituting in the values:

t = sqrt((2 * (14753561838 miles / 2)) / 132 miles/hour^2) = 84331.5 hours

So, the total time to reach Voyager 2 while maintaining a maximum acceleration/deceleration of 6G is approximately 84331.5 hours * 2 = 168663 hours, or about 19.25 years.

This is a rough estimation and doesn't take into account the relative velocity of Voyager 2 or relativistic effects at high speeds.

It also assumes constant acceleration and deceleration, which wouldn't be feasible with today's technology due to fuel constraints.

-------------

So our 22 hour lightspeed trip would still take 19 years if it had a human onboard.

The biggest limitation with sci-fi travel isn't breaking relativity, it's finding a way to accelerate squishy meat-bags. This person will be living at an effective weight of nearly 1000lbs for 20 years, which will really mess them up.

For the sake of the longevity of our pilot, I asked to re-run the calculation limiting the on-board gravity to 1G. It now takes 114 years to reach Voyager's current position, which by then will be much farther. It turns out we will never catch up, and reaching it is impossible if we limit our craft to earth like conditions necessary for human health.


Stop making the chatbot do math.

And come on, you can sanity check those numbers.

"1 G is approximately equal to 22 miles/hour^2."

Think about this. That means it would take you an hour to reach 22mph.

The real speed you'll be going is 22 miles per second.

And sqrt(14753561838 / 132) is 10572, not 84331. That's not even close.

Also it used the Voyager 1 distance which is significantly different.

Once we fix all the numbers, each half of the trip is sqrt(20 billion km / 60m/s^2) which is a week. So two weeks total.

At 1G, each half is 16.4 days, so it takes a month total.


> 6G is approximately 132 miles/hour^2

Imagine stop-and-go traffic if this were true TT


Apparently 32 billion km is about 29.65 light hours, so to catch up we'd need a magical massless spacecraft to travel at the speed of light for a bit over a day to reach it. Hopefully that demonstrates how utterly infeasible it would be to reach it.

It's also near the end of its usable life so it wouldn't be worth it anyway.

And actually, according to https://voyager.jpl.nasa.gov/mission/status/ it's actually 19,936,472,690 km from Earth so I think like 20ish light hours or so.


> 32 billion kilometers > launch a little cheap rocket

My sides.


Not only that, it also has to locate a pretty small object whose position is not well known, and course corrections, if they would help, from Earth take 18 hours (round trips 36).


The number is wrong to begin with, Voyager 2 is about 20 billion kilometers from Earth [1] if I did not do the conversion incorrectly as NASA shows it in miles only.

[1] https://voyager.jpl.nasa.gov/mission/status/


Wolfram Alpha just told me that it's 800,000 laps around Earth's equator away. You can probably compare that to a very long airplane ride (about a 45 hour flight) done nearly a million times.

If that's not enough for human scale understanding, it's gone the same distance Earth goes in its orbit in 34 years.


About 30,000 AR15 lengths per second.


Since the muzzle velocity (1km/s) of an AR15 is about 1/15th the speed of Voyager (15km/s), if you had a matryoshka doll of AR15s that could fire other AR15s you would only need 15 nested AR15s to shoot a bullet as fast Voyager is travelling.


Note that this is actually how multi-stage rockets work.


In Russian, Konstantin Tsiolkovsky famously described the "tyranny of the Kalashnikov equation"


Are multi stage bullets a thing?


I don't think so, but there were gryojets, which were half way there. They were bullet-sized rockets fired from a gun with an unrifled barrel that accelerated as they flew, exceeding Mach 1 at terminal velocity.

Supposedly you could literally slap them out of the air if you were at the muzzle, when they had just begun accelerating.

They were in the James Bond movie You Only Live Twice.

https://en.wikipedia.org/wiki/Gyrojet


Not sure if you are trolling.


I'm think GP means we could launch a rocket to the place where Voyager thinks Earth is supposed to be (where its antenna is pointing towards) and fire off a signal to tell it to move.


>>and gets 15km further away every second

>>I beg anybody to rephrase it understandingly with using some units similar to football fields.

More like it can go from Earth to Moon in like 8 hours(or so).


For reference

- my quick calculation shows just over 7 hours 6 minutes based of google saying the moon is 384,400 km away.

- It took 3 days for Apollo 11 to reach the moon.


Its 3250000 australias away and gets 1 more australia away every 10 days.

Im not sure if thats what you wanted but australias per day is my new favourite unit.


15 km further away every second and 1 Australia every 10 days implies that it would take 10 days to cross Australia if you were going at 15 km per second, which from my understanding of travel options there means either planes and trains are a lot faster than I was aware or something's got mixed up somewhere with these numbers!


> 15 km further away every second

Nope, not true. Not every second. Depends on the time of year. For a few months, each year, Voyagers actually get closer to Earth.


Check your math. It gets ~65 australias away each day.


Australia is wider than the moon so not a terrible unit


Hmm… moon circumference is 10Mm, Australia width is 4Mm, so you can lay 2½ Australias end-to-end when wrapping them around. Figuring out any 2D tessellation is left as an exercise for the reader. But the process of wrapping them… well, the biggest earthquakes on record only damaged half a dozen buildings and structures, to a few million dollars’ damage; this process might just cause rather a lot more. Like a zillion Australias divided by a Tahiti or so, that many times as much. Yeah. It’ll surely also depend on what depth you peel the Australias at.


It’s 128,000,000,000,000 bald eagles


constant 15 km/s and 32 billion km gives something like 67 years. IF a 120 yard football field was equivalent to this distance and a very slow fly is moving through it, it means it's advancing 1.8 yards per year.

or something, dunno.


> Is it possible to launch a little cheap rocket with a transmitter just to correct Voyager's position?

Possible, maybe. Little or cheap, definitely not. Both Voyager probes relied on a unique alignment of the planets in the outer solar system that allowed them to get a series of speed boosts using gravity assists from the gas giants. If we wanted to launch a rocket anytime in the near future that would be able to catch up with Voyager 2 we'd probably have to rely on good old fashioned brute force (rocket power). But then if you want the rocket to catch up in the next thousand years it's going to need REALLY big ass rockets to catch up with Voyager... and if you want it to rendezvous with Voyager instead of just zipping past, it will need to haul more rockets all the way out to Voyager so it can slow down and match speeds (which means even bigger rockets to launch from earth, etc.).

tl;dr - space is big and the rocket equation is brutal.


I gets 2 poronkusema further away every second


This will make for the ultimate "that time I brought down production" story for the engineers involved in this oversight.


Does NASA have any sort of emulator to test commands against before they run them on live?

I mean, we're all human, I've made some really shitty fatal errors hacking untested code onto production servers.


It's hard to find anything about older programs, but they currently put a lot of work into simulators.

https://www.nasa.gov/sites/default/files/ivv_grubb_nasa_ivv_...

On the other hand, at one time there was a physical "proof test model" of the Voyagers.

https://www.jpl.nasa.gov/images/pia21734-voyager-test-model-...


Official blog post from a few days ago:

https://news.ycombinator.com/item?id=36909736


That official statement seems incredibly light on detail, almost as if written for children, or worse, members of congress.

I wonder, is there a technical publication elsewhere that has more substantial coverage for interested people?


I expect that a technical publication will be available soon at one of these sources: https://voyager.jpl.nasa.gov/ https://voyager.jpl.nasa.gov/mission/science/data-access/


What more is there to say? It seems like a pretty clear explanation to me.


Well, congressman, I might be curious what the actual commands were, why they were issued, how it led to the unfavourable outcome, how they detected and measured the degree of misalignment, what a corrected command sequence might’ve been, and then cross-referenced to a hopefully existing article on how the spacecraft will eventually re-align itself, and perhaps some further reading on other commands that are routinely or not-so-routinely issued and how they are received, decoded, and executed on board the spacecraft. Basic stuff, y’know; after all, this isn’t rocket science.

If there is such an archive, or some approximation thereof, it would surely be fascinating to pore over it.


Why would they owe you such detailed explanations? You're asking for a full-on incident report. These take days to write and there's no reason for the public at large to need it.


> Why would they owe

> there's no reason for the public at large to need it

As a member of said public, I would be curious to know. There's no need for taxpayer-funded agencies to operate in a cloak of darkness.

Most everything done by government should by default be open to the public, with an exceedingly high bar that must be met to be otherwise. Otherwise, you run into nonsensical things like how some details around the assassination of a president 60 years ago are still classified on "national security" grounds.


which of these is 'operating in a cloak of darkness':

- NASA informs the public immediately, and then makes the details available later after they've had time to compile the news and information into a format useful for the public

- NASA waits to inform the public until said report is finished

or perhaps you're after option c:

- NASA's network drives are open to the www in read-only mode, because, you know, 'open by default' entails realtime information (even though he doesn't actually care 99.9999% of the time. yet, someone should deliver this functionality, without it costing the taxpayer extra).

NASA routinely makes a LOT of data open to the public. Like, you can get very detailed JWST data directly from NASA. Probably far more detailed than you'd ever care to, because NASA does care about exactly your concern.

Actually, many agencies publish very detailed data if you care to look.


"This is light on info but they're making a report later." would be a non-darkness answer.

But do you have reason to believe they're working on a detailed public report?

Because if they're not, then you missed option "NASA informs the public immediately, but never makes the details available" which would be unfortunate.

Also they probably already answered a lot of these questions internally during the last week, so it wouldn't hurt to put some of that information out.


> As a member of said public, I would be curious to know. There's no need for taxpayer-funded agencies to operate in a cloak of darkness.

This is what the Freedom of Information Act is for:

https://www.nasa.gov/centers/armstrong/FOIA/request.html

The report may not exist yet, so you may need to wait.


I'm not here demanding an immediate report, but it is a publicly-funded agency with a goal of furthering the world's scientific understanding... and a detailed public writeup is not exactly a huge lift compared to all the other things they accomplish.

I'm also the sort of person who thinks that all code written with public money should be open source.


It's always a good thing for technical information about incidents like this to be made accessible to the public. NASA is a publicly funded organization and as such they do have a responsibility towards us.

Of course there are operational details that we don't need to be made aware of, but for an incident as big as this there's no reason to at least know how it happened and what could be changed to prevent it from happening again.


The is if they want the public at large to pay for it.


I am more interested in them working efficiently than wasting time writing a 50 page report every time someone drops a pen.


Ignoring the “pen dropping” strawman, how far does that trust go within the government?

Do you want the military to “work efficiently” if that means little to no oversight? How about Congress?

Oversight and accountability to the citizenry is a foundational principle in a functioning democracy.


> Ignoring the “pen dropping” strawman

Saying that and then countering with your own.

Yeah, the pen dropping is a bit over the top, but as of now the claim is that this situation is planed for and will resolve itself. A report now wont tell us anything of significance. It will get interesting if the realignment fails.

> Oversight and accountability to the citizenry is a foundational principle in a functioning democracy.

I don't see micromanagement in that list.


>I don't see micromanagement in that list.

Is micromanaging what you're claiming is a strawman in my position? I'm not claiming you are saying the military doesn't need oversight, I'm probing with a concrete example where you draw the line on what constitutes a reasonable threshold of accountability. Note my statements were framed as questions to get clarification; that's not a strawman.

Your micromanaging claim is however another strawman statement. I guess I could use clarification on your point. Your equating to micromanaging is misapplied IMO. "Micromanaging" would be a direct democratic vote on most or all issues, IMO. That's not what's being asked for here here. What seems to be asked for is transparency. Access to information is not the same as having authority to make all decisions. But it is paramount in a government when people elect representatives who make decisions (or appoint those who do). The big issue I'm asking is: where is the reasonable 'trust, no need to verify' stance when it comes to public/govt work? Can we just trust tens of millions of dollars on construction projects, but not when it gets to hundreds of millions? What about aerospace? Do we say it's fine to go ahead with limited accountability when it comes to billion-dollar robotic missions, but not when there's a safety-critical application?

>A report now wont tell us anything of significance.

What makes you so confident? A report can tell us if processes were followed appropriately and, if not, if anyone was held accountable for not following them. I'd say that is pretty significant if you care about governmental fraud, waste, and abuse.


I guess you and I are being downvoted because people on HN can’t tolerate engineers being questioned. Hey guys, everyone makes mistakes and it’s an important part of scientific advancement to understand and share that knowledge.


It's a public agency. The information is available by FOIA. Why not publish it upfront and save people the trouble?


It takes time and effort to prepare such a document for public release. Government agencies produce all kinds of reports which are of minimal interest to the public. Making the documents available on demand via FOIA is a reasonable way to ensure that time and money isn't wasted.


Because I pay for NASA and I can ask for NASA to do a post-mortem.


Normally this makes sense, because you're asking why money was wasted. But, in this case if it's permanently bricked you will actually save money, because if Voyager 2 is bricked the team working on it is now redundant. It's not like they had an incentive to be incompetent and waste money - very much the opposite.


You calculation only makes sense if you put zero value on operating a probe that far out in the galaxy - in which case you should be asking why there was a team working on it in the first place.

But that value is not zero, and replacing it costs quite a bit - both money and time. Asking how and why this happened is a valid inquiry.


Under the assumption that it is bricked, the value is indeed now zero. I think where we differ is that you are assuming it will be replaced, but I don't think it will be. It's way past its design life so it was going to expire at some point.

For science, I would want to do an enquiry anyway - I'm just commenting on the financial/accountability aspect.


I pay for NASA and I don't want them to spend needless resources releasing a public post-mortem. Talk about waste of resources.


As a spacecraft navigation engineer, I guarantee you said post-mortem is already being written, and is probably going to be posted "publicly" anyway on some deep corner of the NASA website


This is the right call, let the people of the NASA focus on what is really important, and not waste time on PR.

It's pretty obvious that the people who managed to extend the lifetime of Voyager are very smart, based on all the tricks they had to do.

They are remotely configuring an old-tech device that is billions of kilometers away, with insane lag, and uncertainty that the underlying hardware is even responding properly.

Absolutely anything could have gone wrong at this stage.

They'll anyway investigate internally what happened, in order to hopefully, find a solution.

There is no need to spend resources to make the material public, if the goal is mostly to satisfy curiosity (though it's interesting).


Does this assume the information is made available, but just not as polished as PR?


Asking for a post mortem is not too much to ask. This is the bare minimum for operating serious craft like this.


There's a difference between a post-mortem and a public post-mortem. Nasa is pioneering technology that shouldn't all be public. If you really think the same post-mortem would be published in public and internally, you should not be commenting on HackerNews because it's forbidden below 13 years old.


It definitely got written up internally. Making it public is just a matter of taking that, sticking it into a pdf, and hitting the publish button. A few hours' worth of additional work at most.


Because I’m an annoyingly precocious child of thirteen and this is how you capture my interest and enable my future glittering career in deep space telemetry engineering.


[flagged]


The context is a discussion of what explanations NASA owes in a brief public statement. Saying he'd like to know does not clearly denote that he is changing the parameters of the conversation to talk about something else.


Please show me where this comment thread introduced the term "owed" before the one instance to which I replied.


That's not how language works. The conversation was over what information NASA was obligated to give, which is the definition of owed.


Please show me which parts of the thread implied anything more than curiosity about what is being provided, since you're dodging the point.


> That official statement seems incredibly light on detail

They are complaining about the official statement, specifically its lightness on detail.


Ah, so you're equating mild dissatisfaction (and truly, it is incredibly mild, that's some beige entitlement alright) with demand and a sense of entitlement. I see what went wrong now! Thank you.


As the ultimate progenitor of this tangent I hereby validate thefurdrake’s interpretation. My remarks were intentionally worded to form an inquiring statement of observations and preferences, not a demand for action on the basis of obligation, and the attempt to derive an unstated and unintended sentiment of vituperative entitlement is, indeed, gross.

The unsubtle misparaphrasing of Mark Twain was included as a comedic flourish to provide a light-hearted framing of the comments, but upon review of the subsequent debate, I concede it’s possible that for some, any allusion to statecraft stimulates the adversarial lobes.


What a weird comment.


What a weird response to having intellectual dishonesty being pointed out.


Jesus, I bet you're also one of those people that are fine with mass surveillance because it's ok because your have nothing to hide. It's people like you who set the bar so low that we can't have nice things. Sheesh


I would not describe what you are asking for as "basic stuff."


Don’t tell me this thing doesn’t have a user manual.


FOIA it.

EDIT: Reply here with a link to your Muckrock.com FOIA request and I'll send you the $5.


Except that it actually is rocket science.


“Mistakes were made.” Between that and “I have no recollection,” that’s probably as far as any congressional hearing would be able to drill into this.


I'm amazed there was as much detail as there was. How do they know how far off the antenna is?


Presumably because that was the orientation the spacecraft was asked to get to before comms was lost.

Also, it is possible that 2 degrees of misalignment still allows some fraction of the signal to be detected, but it not being strong enough to be decoded. The received signal strength and the beamwidth of the antenna could then be used to estimate how far off the mark the Voyager 2 dish is.


[flagged]


I honestly can’t tell if this is satire or serious


I lost that ability at some point during Covid.


> which is currently almost 24 billion clicks away from Earth

It makes no sense to use that term in this article not to mention it is usually spelled klick.


Just two degrees off? Can they not wiggle the antenna a bit around [1] just as in the old days when you had to hold the TV antenna a bit above the TV to see anything but noise?

[1] Joking aside, they obviously can not, Voyager is missing the Earth by 4.5 AE. How wide is the beam, how precisely do they have to aim the antenna to maintain communication?


The beam width is 0.65° at x-band. If it's off pointed by 0.5°, the signal will be 7 dB lower (which in this case, is a lot).


Wasn't there some conversation a few years back about how anyone with a big enough antenna could take the probe over because it doesn't really have any security measures and all the command/etc documentation is now public?

In a way this sounds like a way to keep that from happening.


Would be very interested in any writeups on how NASA anticipates all the thousands of scenarios that can go wrong up-front and prepares for them. Sounds like there might be some useful thoughts there on how to write more resilient software


I thought just that about the JWST; I remember an interview with one of the lead engineers saying he wasn't stressed about the launch because he knew they had done everything possible to ensure success and everything was in fate's hands now.

For Voyager 2, 45 years of uptime in the hazardous space environment, billions of miles away, is simply incredible.


I think it isn't about anticipating every possible scenario as much as designing a platform with enough redundancy and ability to measure, turn off/on, adjust, reprogram, etc. pretty much everything.

Part of this is just necessary for ability to learn for future missions. If something fails in space, you want to be able to figure out what happened so that you don't make the same mistake the next time. And you don't have a chance to send a second mission just to "replicate" the problem.

So you do things like build your test equipment into the probe so you can measure stuff while in operation. Or maybe make sure you have a switch for everything so that you can turn something on or off to see if the problem persists.


"Voyager 2's trajectory is expected to remain unchanged" - I should hope so! There can't be any fuel left on board that would budge it, even if they wanted to.


So, until Oct/15, some poor planet with intelligent life in our galaxy will be thinking that an alien civilization is trying to make contact.


Does anyone know how Voyager calibrates their antennas?


Not in any detail, but as a hand-waving explanation it keeps tracks of the Sun and the star Canopus, so by two fixed reference points you can have a known orientation.


I can't even begin to imagine how you would go about building an automated star tracker in the 60s.


There's a bunch of publicly available documentation about how the Canopus star tracker on the Voyager probes works out there, last I looked, and it's quite an interesting design by modern standards. It uses an image dissector tube, which is weird and long obsolete vacuum tube tech that can measure the light in an electronically-controlled section of an image, to scan a slice of the sky around the roll axis of the spacecraft looking for an area in the right intensity range (which is fairly easy for Canopus since it's generally the brightest thing in that part of the spacecraft's view so long as the roll axis is correctly aimed at the Sun), and there's a bunch of hardwired digital electronics to control it and use that to adjust the spacecraft orientation.


The Sun is the brightest star and Canopus is the third brightest star (Sirius presumably is not in an appropriate position to be detected), so you don't really need a proper star tracker, you just need a brightness sensor.


But the ICBMs did. They also had star trackers to help them navigate, and they needed more robust maps than just the Sun and Canopus, since they had to be able to fly 24hrs a day, 365 days a year. Different problem space.


Are communications with voyager encrypted? Is it possible for someone to setup a big antenna in their backyard and take over the probe?


If someone sets up an antenna in their backyard to accurately transmit and receive signals 32 billion km away, I'm willing to bet NASA would gladly trade old probes for that scientific breakthrough of the century.


Their HOA would be _really_ mad.


"Encrypted". That's really funny.

A favorite anecdote of Voyager.

Paraphrasing, "You carry around more computing power in your pocket than what is on Voyager. I'm not talking about your phone, I'm talking about your key fob".

The data Golay encoded, but not encrypted. That's exhausting enough for the 1/2 dozen NAND gates up there that make up its computer.


A 70 metre antenna with enough control to point in the right direction. As voyagers batteries are meant to die in a couple of years, there’s probably more interesting things to do with your money.

https://voyager.jpl.nasa.gov/news/details.php?article_id=118


These guys[1] hacked a NASA space probe and refired its motors. I read the entire blog once but I can't remember if there was any sort of encryption on the communication, although I know that was brought up. Modern probes do use cryptography, but I doubt Voyager does. I suspect if you fired commands at it you could control it. For the lulz or whatever.

[1] https://en.wikipedia.org/wiki/International_Cometary_Explore...


I don't think there's any encryption going on there, just because it's so old

But I also don't think most back yards can fit an antenna that big... search "NASA deep space network" on google images to get a scale of the antennas that are used to talk to voyager


If you want to decode the downlink of a more recent probe, here's the details (apparently NASA don't have the source code for the decoder, but a binary was found):

https://skyriddles.wordpress.com/2023/07/03/stereo-a-comes-h...


It doesn't matter how big your antenna is if Voyager's antenna is no longer facing earth, as seems to have been accidentally induced here.


Untrue, strictly speaking. So long as the combination of your transmitter's power and your antenna's directionality (aka 'gain') provide enough extra dB's of signal strength (to compensate for the dB's "lost" on Voyager's end, due to the off-axis antenna) it'll work fine.

OTOH, dB's are effectively a log scale, and NASA's "not good enough now" transmitter & antenna cost quite a few $million. What's your budget?

(Yeah - if the Arecibo radio telescope was still on operation, it might well have been capable of doing this.)


I suspect it is true, strictly speaking.

In optimal orientation, Voyager's signal peaks at -160dBm when received on the 70m dishes. Now it's shooting 2 degrees off which means the signal misses earth by hundreds of millions of kilometres. What kind of magical high gain antenna do you envision that could still receive it, assuming money isn't a problem?



> The probe is currently around 32 billion kilometers from Earth

No it isn't. It's 20 billion kilometers. It's closer than Voyager 1.

https://voyager.jpl.nasa.gov/mission/status/


Why aren't there more space ships like voyager 2, going outside the solar system but still providing some signal?

It's got to be possible to launch some in space now and have them go faster than voyager 2, so that the outside can be explored faster?


> Why aren't there more space ships like voyager 2, going outside the solar system but still providing some signal?

Because that part is a side benefit not worth launching for, and the main motivation (grand tour of the outer planets) for the Voyagers relied on a once-in-175-years alignment of the planets.

But maybe we’ll have nice probes ready to launch in the 2150s next time the alignment happens.


"severs" huh. A bit clickbaity maybe? I think I'd gone with

"OOPS, NASA gave V2 the wrong number to phone home. Engineers of old have last laugh and reassure 'it's OK, V2 will sort it out'"


I'm surprised at this point that they don't have a "voyager simulator" akin to Kerbal Space Program, that they can try changes like this on before sending it to the real spacecraft...


One thing that’s bothering me - the story text says it’s 32 bn kilometres from earth but looking it up it appears to be about 19 bn kilometres. Which is it?


Jon Bois is probably livid and/or excited


I too know what this reference is


I can't believe it doesn't attempt to auto-calibrate after x days of no signal in some kind of exponential ramp up


It does. It’s supposed to realign itself twice a year


This is why you always have a backup Voyager


I find it hilarious that they're refering to the speed of the spacecraft.


This sounds like a tragic incidence!


TL;DR:

1. Voyager 2 has been pointing 2 degrees off from Earth

2. Been that way for a while and nobody noticed because very old computers.

3. Meaning that the probe has gone dark (ingress and egress comms are not possible)

4. However, both Voyager probes have software that tells them to routinely calibrate themselves every few months

5. Meaning that it should point at Earth in the next few months (most likely).


I don’t think the article or the news release from NASA actually says #2. They could have known for a week but took a week to release the news.


Can we also regain contact through the yearly movement of Earth on its orbit? Like the planet just walking into the new beam position?


I think we can answer this exactly. To visualize this think about the plane shared by the satellite and Earth. We'll imagine this as a 2D unit circle. In this graph Earth is at (1,0) and the satellite is at (0,0). So we end up with a scale where the radius is the same as the distance from the Earth to the satellite. But instead of pointing at (1,0), the satellite is now pointing at (cos(2), sin(2)) or (0.9994, 0.0349).

The distance from Earth (1,0) to the new location (0.9994, 0.0349) is about 0.0349. We need to scale that back up to "real" units so multiplying it by 15 billion miles. And we get about 520 million miles. The earth is about 93 million miles from the Sun, so its max positional shift (under extremely improbable absolutely perfect conditions) would be ~180 million miles.

So there's no way we could regain contact with just yearly movement, even before we account for the fact that it's getting further and further away. 2 degrees intuitively sounds small, but on an astronomical scale it's huge and this sounds like a pretty major flub by NASA.


Great explanation. Thank you!


The Earth is only about 150 million km (1 AU) from the sun, so nope the Earth can't move far enough to make up for a 2 degree miss from 32 billion km away, even if it's in the right direction.


Wouldn't work if the 2 degrees is not in the orbital plane of Earth. Right?


No. 2 degrees error at such a large distance equals huge distance error.

Also, empty space is huge and matter-things like Earth and the Sun are tiny compared to it.


> 5. Meaning that it should point at Earth in the next few months (most likely).

Provided that V2 still has enough propellent to make this adjustment.


aliens


Stub for arguing about what "bricked" means. These comments were originally replies to https://news.ycombinator.com/item?id=36941191, but we moved them because the offtopic discussion was choking the thread.

Normally I'd have marked the entire subthread offtopic, but hutzlibu's comment deserves to be at the top, even if it does use the word "bricked" wrong.


> In short, it was remote bricked, by giving it commands to rotate a bit. > But luckily it automatically readjust itself to earth automatically every half year exactly for these events.

I remember when bricking something meant it was totally unrecoverable. Now it means "temporarily not working but will automatically heal".


A device that is acting as a brick cannot receive commands and is not useful at all. That is the current status of voyager 2.

"Unbricking" will hopefully work automatically, because there is no other option. But that can also fail and there is no way to know, or influence it.

I use bricking in the definition of mobile phone tinkerers .. there are many results for unbricking btw, but I just checked and with the first result it seems that Apple now uses unbricking for activating a new device. Because technically before, it is also just a brick - but here I would agree, that it is not a appropriate term, but rather should be for somehow broken devices.


What this is telling us is that attempting to condense to "it was bricked" has actually introduced ambiguity, and that "brick" doesn't really explain a technical situation.

The JPL doco (https://news.ycombinator.com/item?id=36941433 https://news.ycombinator.com/item?id=36942321) calls it "Command Loss".


But you did understood my original comment? I described exactly in what way it "bricked". I used the term in the first place, because this was my first assoziation, when I learned about the situation. That "uppps" feeling when you did something wrong and there is no going back.. (poor guy) "bricking" describes these vibes for me and "Command Loss" does not.


It’s not bricked, it’s operating autonomously for some time. They’re incredibly different modes of operation.


After trying to optimize my laptops energy settings under linux, I once also ended up with a device that was operating low level autonomously for some time. It just would not accept commands from me, nor the power button, nor anything else. The CPU also wasn't running, but something was.

In other words it was effectivly a brick to me.

But since it was not a surface pro (I considered buying instead of that one), I could open it and disconnect the battery.

And in effect, unbricking it. Quite trivial fix sure, but nearly impossible with many modern devices, where the battery is glued in.

My point is, not every mode of operation is desired, especially if you cannot change it. Then you might as well have a brick in terms of usefulness.


No. Your laptop was not operating autonomously, by definition.

It was not making its own decisions, to achieve some goal.


Parts of it were. To make some checks for some hardware (as part of an automatic comand line tool). They just got into an infinite loop. Down on the hardware level.


That's not autonomous that's automatic


I think it's reasonable to say that it is operating autonomously and is currently "bricked" as a colloquialism. There's a certain helplessness for NASA in this case, which is similar to bricking one's device. Instead of hoping that the repair shop can fix it, they have to hope that their engineering foresight was adequate.


Instead of saying "bricked" you could say Voyager 2 is "soft locked".


Nothing is ever truly unrecoverable. If a device was built, it can be built again.

What is bricked vs recoverable has always greatly depended on time and effort, individual skill level, available hardware/software tools, documentation, crypto keys, physical access, willingness to replace individual parts etc.

Sometimes, even within an org, some teams e-waste expensive devices that aren't bricked deeper than what other teams recover from as part of everyday workflow.

Taking a typical network device as an example, where do you draw the line? Driving to a remote location to plug the cable into another port, pressing a reset button, booting from USB, flashing a new firmware with TFTP, plugging in an external or internal console cable, opening the case and soldering a header to get access to the console, doing the same with no documentation, or an unknown (but maybe Google-able or reverse engineerable) password, flashing firmware with JTAG, shipping the device back to the engineers (or shipping an engineer to the device)...? It's always been arbitrary.


If you can't fix it or find someone who can, it is bricked.

If you are able to fix it then it is not bricked.

One device may be bricked to one person but not to another. But that must still be the definition, right? Otherwise the word has no meaning.


Hi, I have drilled my hard drives but need to recover them, can you help?


My device is as worthless as a brick, but only for the 2 or 3 seconds it takes for the tip of my finger to travel to this reset button over here... <Tongue in="cheek"/>


Then it's not actually bricked.


I got into an argument with a fellow Tesla owner on a forum who was screaming their car was bricked after their 12V battery died. All they had to do was replace the battery. It wasn't bricked. I sure received a lot of vitriol for saying it wasn't bricked. If you can simply perform a maintenance task, it's not bricked.


> I remember when bricking something meant it was totally unrecoverable.

Precisely. 'Bricking' something means it is unrecoverable and is irreversible.

No idea at what point in time the definition was changed to mean 'temporarily not working'.


Bricking oftentimes can be reversed using JTAG connectors. IMO - bricking describes thr state that a device is not operable, not irrecoverably so - just that its difficult to reverse.

Also, it's not a technical term with a rigid definition, hence "soft-bricking"


I'd say "totally unrecoverable but physically intact". You wouldn't call a device bricked if it has the form of small pile of ashes.

Then "totally unrecoverable" is rare and the term bricked has always been relative. Your bricked device may be as good as new to someone who has a JTAG adaper and knows how to use it.


> I remember when bricking something meant it was totally unrecoverable

It may have seemed that way to you, but actually no. "Bricked" has generally referred to devices that are likely straightforwardly recoverable, but for a lack of documentation from the manufacturer.


No, that's not true, and it's never been true. The definition was always "turned into a device which is electronically indistinguishable from a brick and unrecoverable." Maybe an expert could do some deep diving to bring it back, but if it's beyond recovery to most folks, then we'd call it a brick. If you have to desoldier a flash chip and sldier on a new one with a filesystem that isn't trashed or with corrected software, then we've debricked it, but that's really a deep level repair.


Your division of "experts" vs "most folks" is doing a lot of work here, and speaks to my point.

Most folks don't really know how to use say Android fastboot or recovery modes either, yet we wouldn't call a device with a wiped system partition "bricked".

Most "bricks" are things like a bootloader getting erased. Reflashing that through the standard process of JTAG or another debug protocol is a straightforward action (after all, the manufacturer has to get the first bootloader on there to begin with). The port pinout and config info just hasn't been publicly documented by the manufacturer, which is what pushes it into the domain of "experts".


If doctors create the term "heart attack" and laypeople misuse it, that doesn't change the definition.


I don't really know how to connect your analogy. As far as I'm aware, the term "bricked" arose out of software/firmware modding communities (eg Android) to describe devices that were beyond their general abilities to straightforwardly fix.


No, bricked was a word IT techs were using at LEAST as early as the early 90s as that's when I learned it. I learned it when someone bricked a network switch in 93. Originally in the Android world (and before android with Symbian and others) bricked meant you flashed firmware that really killed it, maybe you can bring it back with a JTAG connection or something more extreme. Then as modding became more popular, they started being able to more easily recover these and UNbricking became a thing.


Bricked things can't be unbricked (unless it wasn't actually bricked to begin with and was misdiagnosed). That is why it is called bricked.


Bricked things absolutely can be unbricked, e.g. by opening them up and reflashing a component, or otherwise engaging a special-case recovery path.


Bricked things can only be unbricked because the word has gradually lost most of its meaning. At this rate some day you're going to hear someone say they bricked their phone and mean that it ran out of battery and needs to be recharged.


No, this is what the word has always meant.

An embedded device with a failed bootloader update is considered “bricked”, even if you can open it up and reflash it with a valid bootloader.

I don’t know why folks are so insistent on gatekeeping the word, as if doing so demonstrates some superior personal knowledge.


Some people don't just guard that word, they guard all words. We as a society even need to pass laws to protect the definition of words we use in commerce, like "ice cream" and "bread", otherwise people would abuse them to the point where they become meaningless.


It's not a term I've seen used recently, but a bricked thing that literally cannot be unbricked through a simple repair is toasted. At least according to mid-nineties computer repair jargon. I wonder if there are regional dialects...

Weird discussion since we're talking about a piece of hardware that is working fine and doing exactly what it was told to do. They just pointed it in the wrong direction and need to wait for it to recover, which it is set up to do.

We wouldn't say a server with the router IP address misconfigured was "bricked." (or maybe we would... I guess the jargon changes, but that would seem pretty crazy to me)


True, but they don't recover themselves automatically.


A brick can't fix itself in case of problems. Just grab a brick, put it in a corner of the room and you'll see. It stays there doing nothing, it's kind of amazing how little it can do.


There is also such a thing as subtlety and nuance. Words borrowed from physical objects do not need to have, and in general do not have exactly the same meaning when applied to software.

I would love to see a picture of your computer pulling itself up by the straps on its physical boots the next time you press reset. Bleeding when a process is "killed".

Even something as superficially similar to real-world behaviour a "queueing" is implmented in a very different way in software, for the most part.


You can also see nuance in humor, in particular when recognizing a joke.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: