Hacker News new | past | comments | ask | show | jobs | submit login
The James Webb Space Telescope – making 300 points of failure reliable (flyingbarron.medium.com)
150 points by kiyanwang on Jan 9, 2022 | hide | past | favorite | 123 comments



It's unfortunate that the article speaks so much of NASA engineers and managers, but fails to mention the actual engineers that designed, built, and tested it: TRW (now Northrup Grumman) and Ball Aerospace.

> The concepts used to create the JWST are built on the same concepts used on the first satellites and space probes in the 1950s.

AKA, the field of Systems Engineering [0]. Further reading if you really want to take a deep dive [1].

[0] https://en.wikipedia.org/wiki/Systems_engineering

[1] https://www.amazon.com/Mission-Analysis-Design-Technology-Li...


Fair enough, but I felt that detailing the contractors would lengthen the article and not add to my general points. I did mention Northrup Grumman engineers in the caption of one image.


Honestly, you writing it on medium is itself a failure point. Many people can't read it due to medium's shady paywall business.


+1 for systems engineering. Sadly it isn't more common in software development.


Systems engineering roles are typically considers lesser / lower skilled than software roles as well. Which is absolutely ridiculous


If programmers were held to standards routinely upheld by systems engineer, software would not be such crap.

There would also be less of it, which would be another boon.


It was when software delivery involved distribution of physical media. Now that it's all done over digital networks, people have moved to early deployment models where failure in the customer's hands is all but ubiquitous.


Indeed. I see this as our downfall: if the price of failure drops then failure becomes commonplace.


That's brilliant. It puts my general vibe of software basically for the past 10 years (maybe longer? for me mostly the advent of the app stores and near serial release of hundreds of thousands of apps) into a single sentance.


+1 for systems engineering indeed.

That's one of the reasons I write my blog - I think there are plenty of inspirational & aspirational lessons software developers (especially, but no only, in the DevOps and SRE domains) can learn from the space exploration.

If you're interested, my first article can be found at : https://medium.com/ibm-garage/out-of-this-world-lessons-from...


Northrup Grumman is the type of company that's happy to bill the government for 3 very senior engineers to spend 4 months to rewrite a small program in a different programming language.

It's no wonder the JWST is 10x+ over budget.


meh, the whole article is just a teaser for upcoming articles and not much more than a single line about what they did with webb

"The only solution left [...] was making sure that the JWST would be as reliable as possible.

[...]

I plan for the next few articles to be split between further details of the James Webb Space Telescope"


Well yes, this is an introductory article to a series of articles.


Yes, totally fine as that. I just expected more after the title and I'm a bit annoyed by pre-news and announcements on HN in general.

I'm really looking forward to some details and followed you on twitter (because I have no Idea how I would do that on medium without ail spam... I miss RSS...)

The only source for details I found so far was an old videolog that at least showed some close-ups of things in action too. https://www.youtube.com/watch?v=7DQe-OzrcJM&list=PLcy1hEnsej...



yea, but I'm not using my RSS reader anymore because it's not as common anymore :/


I just recently got into RSS and was pleasantly surprised to see that virtually all websites I regularly read provide it.


maybe I should give it another go :)


All fluff, no stuff


1) The JWST is not in the shadow of the Earth. It’s solar powered.

2) It didn’t cost 9,000,000 dollars to build. That’s off by 1000x.


To elaborate on point 1: Although L2 itself is in Earth’s penumbra, the orbit of JWST around L2 is not in the earth’s shadow at all. (In fact, JWST will orbit L2 at a distance greater than the distance from the earth to the moon.)


My guess is that the content you are referencing in your 2nd point is really the author's failed attempt at a variant of the "it's over 9000" meme. Nobody could reasonably think that it costs in the ballpark of $9 million.


No, it's simply a typo, missing the final ",000". It's been fixed now, thanks.


Thanks for fixing it. The internet has taught me to be wary of meme references everywhere.


Thanks. Changed that to "while staying behind the Earth"


If I see the "300 points of failure" mention one more time without them providing a list of these said 300 points of failure, I'm going to go insane


You might find this interesting:

https://space.stackexchange.com/questions/44093/details-on-t...

The discussion includes links to several relevant GAO reports. I agree that it would be interesting to see a single detailed list. That is not in these reports.

https://www.gao.gov/assets/gao-20-224.pdf

https://www.gao.gov/assets/gao-16-112.pdf

https://www.gao.gov/assets/gao-16-112.pdf


Do not get too excited for such a list though because I think around 180 of them were just bolts, locks or actuators securing the movable parts of the primary mirror.


Just? It's my understanding that each of those could have failed to deploy and if any of those would have, it were the end of the JWST. No way to repair or salvage. No 30% mission targets achieved or "underperforming" (as e.g. Hubble before the mirror correction, i.e. with approximated correction done in software on the ground). Nothing of value, just a flying, very expensive, brick.


Sometimes, the art world has articles about paintings... Without a picture of the painting.


If you see the painting, you risk making a judgement based on how 'pretty' it is.


some people are painful.


You might like this instead: https://youtu.be/uUAvXYW5bmI


Given the many years of launch delay, did they use the opportunity to upgrade some components, like CPU or camera sensor?


No, cameras' resolution is limited by diffraction that is a function of mirror size. JWST achieves that diffraction limit in range between 2 to 28.5 micron where most of it observations will be performed. As for sensitivity, installed instruments are state of the art, finding any improvements would necessitate multi-year research programs.

CPU it uses is RAD750, pretty standard radiation-hardened CPU for spacecrafts. https://en.wikipedia.org/wiki/RAD750

More powerful and modern CPU would not enable any more science to be gathered and only increase chance of failure. Maybe some power saving could be achieved, but that CPU requires only 5W from 2000W produced by solar panel anyway.


> As for sensitivity, installed instruments are state of the art, finding any improvements would necessitate multi-year research programs.

That was exactly my question. Did they use the 8 years from 2013 to 2021 to improve the instruments? Or did they remain at 2013 state of the art level?


Final optical instruments were integrated in 2014, proposed launch date at the time was 2018. There was not enough time to think about any upgrades and cost was ballooning anyway.

NIRCam has quantum efficiency of 70 to 90% depending on light frequency. https://jwst-docs.stsci.edu/files/97978241/97978242/1/159607...

MIRI has average efficiency of about 50%. https://jwst-docs.stsci.edu/files/97977640/97977649/1/159607...

Internal noise of electronics is not significant compared to thermal noise at operating temperature of MIRI (7 Kelvin).

Further small improvements might be possible in the future but no 'orders of magnitude' can be expected without going into even colder temperatures, that would require total redesign.

At the end of the day, efficiency can be offset by simply taking longer exposures so it is nice to have but does not have to be super optimised. Achieving diffraction limit is much more important and JWST's camera do that.

https://jwst-docs.stsci.edu/jwst-near-infrared-camera/nircam... https://jwst-docs.stsci.edu/jwst-mid-infrared-instrument/mir...


The raw efficiency numbers aren't that helpful without a reference. To put it another way, what would the key differences in the component performances look like if you used the state of the art components from this year vs whatever's is in there?


Quantum efficiency is a ratio of how many electrons are created from typical photon hitting a detector. For optical instruments max value is 100% and is the ultimate reference - every photon gets converted into electrical signal. This is why order of magnitude improvements are not expected to happen - there are physical limits on how much signal can be generated from certain amount of light. Solar cells for generating electricity may go over 100% because high energy photons can be converted into 2 electrons but this is undesirable for optical imaging and light will pass through narrowband filters before hitting detector to filter out such photons as they would skew results.

Nancy Grace Roman Telescope to be launched in 2027 is expected to have 75-95% QE for near infrared compared to 70-90% for JWST: https://roman.gsfc.nasa.gov/science/WFI_technical.html

Last upgrade to Hubble near-IR instrument in 2009 (WFC3) has about 75% QE: https://www.researchgate.net/profile/Bernard-Rauscher/public...

Is it simplified model, real life detectors have more parameters regarding noise and persistence but all produced in past 20 years are close to theoretical limits.


Did the state of the art change much in 8 years? I am not sure we can use consumer electronics heuristics to determine the rate of progress at the high end. Consumer electronics has only one constraint: cost. Sony could sell you a gigapixel camera and a 200 core CPU to process the data today. Consumers can't afford that, though, so it's not a product. But at the high end -- humanity's only new space telescope -- the cost constraint is not as big of a problem, so the tech is going to be be better than what comes in a $100 phone.

Hubble is 30 years old using 1970s and 1980s tech and it's still a useful scientific instrument. Something with 2013 tech is going to be spectacularly useful in 2022.


That would require an increase in the size of the mirror, which would mean a complete redesign of the satellite. It is already at the limit of what can be observed with a mirror that size so a better sensor won't automatically give you better images.


If you look at professional cameras (Sony Alpha for example), the sensors improved in the last 10 years from taking shitty pictures at night to being able to take pictures in almost total darkness. All this with little improvements in optics.

So is the telescope sensor able to count each and every photon hitting it, was it already at maximum possible performance?


Yes, it can detect single far red shifted photons. It's cooled down to 7(!!) Kelvin to achieve that incredible feat, while the telescope itself is at a relatively balmy 36 Kelvin. If you want to get more spatial resolution you will need a (much) larger mirror and likely an even more stable platform. This is already a machine of such ridiculous precision that new technology had to be invented to make it possible, those sensors are works of art on par with anything ever made.

You can read more about the sensors here:

https://jwst.nasa.gov/content/about/innovations/infrared.htm...

And about the cooler here:

https://webb.nasa.gov/content/about/innovations/cryocooler.h...


> If you want to get more spatial resolution you will need a (much) larger mirror

Or multiple mirrors spatially separated. Or a single mirror/camera at different positions at different times. Since JWST's orbit around L2 is fairly large, is that going to be used? You mentioned that stability of the platform would be a limit. How well is the position of the JWST known at any given time (I suppose it can be calibrated by viewing well known sources)?


That trick is hard enough when using radio telescopes connected to the mass of planet earth, with enormous wavelengths in comparison to what JWST is doing. It's position is known fairly accurately but even the telescope will undergo various vibrations from for instance the equipment on board and the course correction burns which will have a pretty significant effect on the satellite body.

The mirrors are dynamically deformed to correct for some errors, I don't think - but I also don't know for sure - if they are going to do long baseline tricks with the orbit, it would make good sense to do this for paralax shift measurements (which give you a good idea about the distance an object is at), but the orbit of the earth around the sun would be far more useful for that because it is so much larger.

Anyway, there are people on HN that are far more knowledgeable about this stuff than I am, I take it the designers and operators of the JWST are top in their fields and that anything interested laypeople can come up with has been debated, accepted or rejected a decade or more before this conversation, they're far from stupid, as evidenced by the incredible performance so far. Let's hope it stays that way and that the insertion burn goes well, that's the major scary thing that will happen next and the delta-v is nothing like the launch so I would assume that it will all go well but at the same time the telescope wasn't as fragile back then as it is now.


Yes, but Hubble also already did that. Astronomy has been at “we count every single photon” for at least the last two decades


They could have added a great deal more on-board data storage with negligible risk. As it is, they need to dump twice a day or lose observations.


The article talked about redundancy in the form of making duplicates. Another way (the way airliners are built) is onboard redundancy. Modern airliners have zero single points of failure.


> Modern airliners have zero single points of failure.

The have zero points of failures that are classified as single-point-of-failure.

Once in a while, the classification turns out to be incorrect, as seen in the 737 MAX crashes, especially the first one.


The redundancy in the MAX crashes was the pilots following the runaway trim procedure. Of the 3 MAX incidents, the one you never hear about is the one where the pilots followed that procedure and continued the flight normally.

Boeing still should never have had the MAX system rely on only one AOA sensor.


> pilots followed that procedure

Were these pilots trained with the new procedure and the other pilots who died not trained?


Just to add, it was the normal "runaway trim" procedure-- nothing new.

The thing is, this revealed a vulnerability: real pilots in runaway trim scenarios do not do nearly as well as they do in training. And 737MAX ended up testing crews with runaway trim scenarios at a much higher rate than they had been at the past. Some crews failed.


No. It was the LA flight immediately prior to the LA flight that crashed.

They trimmed it back to normal a couple times, then just turned it off.

Sounds too simple to be true, but it is true. There's a toggle switch on the console that cuts off power to the trim system. It's there in case of runaway trim.


> Modern airliners have zero single points of failure.

Safer to say that they're designed and certified attempting to have no single points of failure.

E.g. the MD-83 stab trim jackscrew system. Two nuts were used on the acme screw, on different portions of the screw, driven by redundant gearboxes and motors: no single point of failure! But there were common-mode failures stripping the screw and nuts.


If you pop a window you lose pressure right?

You blow a fuel pump and can’t fuel the engines.

Wings fall off.

I dunno. I can think of a bunch of failure points on a modern airliner.


Fuel pumps have backup pumps. Matter a fact, the whole fuel system is a lot more complex than this.

And as for “the wings fall off” part, makes me remember a quote of an engineer I once read that goes something like this: “The wings are the strongest part of the airplane. If you worry about them ripping apart or falling off, you have way bigger problems than that, like the rest of the plane missing”.


The flex test that the wings go through is pretty incredible:

https://www.wired.com/images_blogs/autopia/2010/03/index.jpg


Yes, I know! I don’t remember what video I saw but the wings were bent beyond imagination. Made me realise that what I saw while flying wasn’t even 30% of what they can withstand. Incredible engineering.


The Lockheed Elektra was infamous for its wings mysteriously coming off. Much research finally revealed that dynamic instability (flutter) was the cause. A great deal has been learned about how to prevent this, and a lot of effort goes into designing a wing that isn't susceptible to flutter.

I did the calculations for the 757 elevators to show it wouldn't flutter. And yes, it was also verified on the test stand.


There was an F-15 that had a mid-air collision, the entire wing ripped off, and it managed to safely land.

https://taskandpurpose.com/history/1983-negev-mid-air-collis...



The fuel systems at least are fully redundant. Each wing has its own system and then there’s redundant systems that allow fuel to be transferred or delivered from either wing.

I discovered this when trying to understand what the barking noise is on airbus planes.


The barking noise on airbus systems is the Power Transfer Unit which is about delivering hydraulic pressure across 2 independent systems, not fuel.


I thought the barking noise came from the Canine Transfer Unit in the hold. Thanks for clearing that up!


> If you pop a window you lose pressure right?

Right. But the plane drops oxygen masks and goes down to a lower altitude.

> You blow a fuel pump and can’t fuel the engines.

Multiple fuel pumps and multiple gas tanks.

> Wings fall off.

Wings have dual internal structure.

> I can think of a bunch of failure points on a modern airliner.

No single point of failure is an FAA requirement.


If you pop a window, everyone puts on their oxygen masks and the plane makes an emergency descent. The point isn't that nothing at all can go wrong, but rather that the plane will get its occupants onto the ground safely even if something does.


> I dunno. I can think of a bunch of failure points on a modern airliner.

not catastrophic failure, like you get with a rocket or a helicopter.


I'm not an expert but "wings falling off" seems like a pretty catastrophic failure for a typical aeroplane.


The wing root is the strongest part of any airframe. Wings falling off is way down there on the list of aviation risks, maybe next to getting struck by a meteor


Low probability or not, if an airplane can't stay in the air with one of the wings missing that definitely counts as a "single point of failure". Whether that SPOF needs to be mitigated is a tradeoff based on other constraints like the expected risk, economics, and safety regulations (and in the case of the space telescope, maximum launch mass of the rocket).


Wings don't just go missing. It would require multiple failures for a wing to "fall off" an airliner.


That's kind of ridiculous. A terrorist bomb in the cargo hold is a "single point of failure" that will crash the plane, as is quantum uncertainty causing one wing to break off spontaneously. This says nothing about the actual SPoF engineering requirement, which is a very real, very relevant thing.


https://www.aopa.org/news-and-media/all-news/2018/may/16/nts...

Not as small of a risk as you’d expect (though not anything to seriously worry about). This metal fatigue grounded a large chunk of the US training fleet and similar cracks were frequently found in other Piper Arrows (including ours).


That is actually kind of scary.

How many cycles does the airframe have to go through in order for this to become a problem? I tend to think that if the FAA gave a shit about general aviation maybe we wouldn't be flying planes from the 1960s


For airliners, the airplanes are retired after a certain number of cycles. The 757 was designed for 62,000 flying hours, if I recall correctly. It's scrap after that.

My dad's B-17 in WW2 was scrapped after 30 missions, it was considered worn out.


that's not redundancy. can a plane fly w/ only 1 wing and for how long?


Israeli F15 landed missing one wing but this was a fighter jet with super high thrust capability to help offset missing lift from the wings:

https://theaviationist.com/2014/09/15/f-15-lands-with-one-wi...


a plane without a wing is no longer a plane, realistically


The wings have dual spars.


One. Pilot error.


There are at least two of them.



I only skimmed the article and found a few errors...


Sorry about that! The price was obviously a typo :(

What other errors did you find?


JWST doesn't actually stay right in L2 (which is in Earth's shadow). Instead it orbits around L2 about twice a year in a halo orbit, with a radius large enough that it can have nearly constant access to solar power (I do not know if it fully avoids the moon's shadow). You can see a good animation near the top of https://www.jwst.nasa.gov/content/about/orbit.html


Awesome link. Thanks for posting. I'm wondering, does anyone know why we didn't send JWST to L4 or L5, where (as mentioned in the link above) it could remain orbiting indefinitely without the need for fuel? Is it because those points are a lot harder to get a spacecraft to?


JWST is making observations in orange to mid-infrared wavelengths. The sun, earth, and moon are all quite hot and emit a lot of light in that range of wavelengths. Near L2, JWST can use one sunshield to block out light from all three bodies, with less impact on what portion of the sky it can observe


Thanks. Changed that to "while staying behind the Earth"


Minor pedantic point—kelvin is the unit of measure of the Kelvin scale. There is no degree K, just K.


Does anyone know who the author is (beyond the name) and what their background is? Do they work at NASA, for example?


https://www.linkedin.com/in/flyingbarron/

"I'm a solution engineer specializing in AIOps, SRE, & ChatOps. I enjoy helping others solve problems even more than I enjoy solving them myself.

Besides my IBM work, I also write about SRE lessons from the early era of space exploration - the race to the Moon and the Space Shutte - at https://flyingbarron.medium.com

I'm a published author and I speak for myself and IBM at global conferences."


How far back in time do they need to look for x-rays emitted to arrive in the red-to-infrared range the telescope is equipped for? E.g., will we have to rely on x-ray emission from the earliest stars to be able to see them here/now in far infrared?


It's complicated for sure, but I still don't understand why it cost $9 BILLION. That's a lot of zeroes!


Because there is only one (no economies of scale), because it is large, because it is insanely precise, because it has to work the first time, because it is traveling very far, because it has to withstand the stress of launch, be 'foldable' (far easier said than done), because they encountered a number of problems while testing the various subsystems requiring re-design/re-manufacture, because it runs a delta-t of 290 some degrees celsius across a two meter distance and so on.

This is probably mankind's most amazing technological achievements bar none.


When you put it like this, 9 billion sounds like a good deal :)

Also, consider how many (presumably) well paid expert engineers are scientists have been working on this since 1996.


That is their most productive life for many. And if that fail nothing fail … some compensation is a must. I hate these “well paid” words. I respect the risk and just hope the people working on it has proper life.


A combination of space engineering costs being habitually lied about in proposals and the conventional space industry being about half science, half jobs program.

Frankly I wish someone would do an article on the "long tail" possibilities for Webb. Can it do a warm mission after its refrigerant runs out like Spitzer? Does it have low-bandwidth long range communications for when it runs out of fuel and falls out of L2? I feel like we're just supposed to be ok with a ten billion dollar telescope only lasting a decade.


Only one instrument the MIRI needs active cooling. Rest is all passive cooling. It has a closed loop cooling system so it will not run out of refrigerant. Once it runs out of fuel it will eventually drift out the L2 point and might lose orientation. They feel they can stretch the mission up to 20 years with the saved fuel. There is a port for refueling the craft in the future but the technology needs to built.


Why would it lose orientation? It has reaction wheels.


The simple answer is that you eventually have to do thruster burns to dump momentum from reaction wheels; they can't keep absorbing momentum forever. It also needs the fuel to adjust it's orbit every so often and keep it on station.


The question is "is it worth it?".

New scientific instruments usually come with a bunch of discoveries, because we are now able to see what we couldn't see before, but there are diminishing returns, because we run out of interesting things to look at.

Some platforms can be upgraded, like the ISS, ground based telescopes, and Hubble in 2009, so that new science can be done, but it is not the case for JWST. So I suspect that there will not be much left to do with it after 10 years, especially if it is not fully operational. It will not be completely useless, but I expect money to be better spent making a new instrument, with better technology and guided by the latest discoveries, than trying to extend the life of an aging instrument that has given most of what it could give.


My understanding is that most off the money went into designing and extensively testing one off components that work reliable in extreme environments.

Engineers are expensive


Hmmh, I'd think since much of JWST is i) one-off and ii) state-of-the-art, a lot of it was actually designed by PhD students, which are cheap.


> But the JWST was not designed or tested in a vacuum

this is false. NASA has a house-size vacuum chamber that was used for testing. Also the author got the price tag wrong.

I wonder what else about the article is incorrect.


In the NASA live stream yesterday, during one of the interview a engineer mentioned, that one of the harder aspects about the primary mirrors (if I remember correctly), was figuring out how to test the equipment in the cryogenic vacuum chamber. What they mentioned they couldn't fully test, was deployment in zero gravity.

So far I think NASA has done a great job, trough blog posts, videos, websites explaining and visualizing different aspect of JWST in a way that is tangible for laypeople.


> What they mentioned they couldn't fully test, was deployment in zero gravity.

Just test it once right side up and once upside-down ;-)


Yes, the price error was a typo which has been fixed.

"not designed in a vacuum" was a figure of speech, meaning that testing it was done in the context of previous missions.


peeps here can be very literal, it's I suspect no accident


It's a figure of speech. Looks like TFA clarified:

> But the JWST was not designed or tested in a vacuum (pun only partially intended)

It's not the clearest writing, even though they are going for a joke.

JWST was not "designed or tested in a vacuum" [in isolation without knowledge of prior art]". But it was designed for vacuum and its parts were tested in a vacuum, including I believe bringing the whole assembly down to cold vacuum.

Designing in a literal vacuum sounds painful for the designers. :)


" the JWST will orbit the Sun while in the shadow of the Earth. This will allow it to keep the temperature of its delicate instruments at just the right... "

Well that's just totally wrong.


Thanks. Changed that to "while staying behind the Earth"


Beyond.

Or farther away from the sun than the Earth and Moon.

Behind implies in the shadows of Earth.


More like "300 points of success" :)


I find it difficult to comprehend how much a risk they were taking. Documentation of some tests is publicly available; it's even mentioned that they found problems and had to partially redesign (and supposedly re-test).

If a bolt (or rather one quite like it) has been tested successfully 100 times in the expected conditions, chances are it'll work when needed, but I get a sense not all of those parts in the 300 single points could be tested that often (material cost, time, availability of sufficiently large climate chamber capable of simulating outer space condition). Were there actually parts too expensive to test even once? Is JWST sailing on prayers?


Why parts of the instruments need to be cooled below 50K? Do they have quantum computers on the JWST?


The detectors themselves, to reduce thermal noise. It is not uncommon to cool down sensitive detectors, for example, in electron microscopes, the x-ray detectors are typically cooled down below -40 Celsius.


Since JWST is trying to pick up weak infrared signal from the furthest reaches of the cosmos, any amount of noise would distort our measurements.

The telescope itself will naturally radiate infrared heat (much like our own bodies do), which is exactly what the instrument is trying to measure. This is why they're cooling the instrumentation side to such a cold temperature that it won't emit infrared energy and ruin the science.


Allright, so is that why it takes ~6 months of waiting time now? Would it not be possible to just measure the heat radiated from JWST and subtract it from readings?


Measure thermal noise? The amount of information contained in the black body radiation of a warm (well, really cold, but not cold enough) object must be absolutely immense, at least when you care about a signal at the level of single photons. Trying to capture it with sensors is futile. Sure you can try to do some noise suppression with a simplified model/probability distribution, but that won't work well if your signal to noise ratio is 0.00001.


That would be like throwing a beam of a light from a powerful flashlight 10 cm away from his face when he watches night scenes from last season of Game of Thrones, and saying ”just ignore these 1000 lumens, tell me what is happening?”


Sensors that can measure the heat will themselves be putting out their own heat. The scientists who built this thing are pros.


Things in and around the infrared sensors, since warm objects would blind the sensors with black-body radiation


There are a couple of instruments that are passively cooled but one instrument the MIRI needs active cooling down to 6.7K.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: