NOAA’s newest GOES 17 weather satellite may not be fixable, and a loss

From NOAA and NextGov reports.

The second satellite in NOAA’s $11 billion GOES program continues to experience issues with its most important instrument and officials still aren’t sure what’s wrong.

Artist rendering, GOES17 in orbit. Image: NOAA

The National Oceanic and Atmospheric Administration’s newest weather satellite—part of the $11 billion Geostationary Operational Environmental Satellite constellation set to operate for the next 20 years—is broken and officials still aren’t sure what’s wrong or how to fix it.

NOAA officials identified cooling issues with the satellite’s primary instrument in May. Due to the cooling failure, infrared and near-infrared imaging was only possible 12 hours per day. The issue affects 13 of the infrared and near-infrared channels on the instrument. Depending on sun’s position, as many as six channels will only be able to work part-time. Other instruments aboard have not been affected.

Officials told reporters Tuesday that two review teams consisting of NOAA, NASA and industry personnel continue to investigate what’s wrong with the Advanced Baseline Imager instrument and how to mitigate the loss to ensure weather forecasters on the ground continue to get high-quality satellite data.

In the meantime, GOES-17 remains in a holding pattern orbit 22,000 miles above the Earth’s surface, and the launch dates of two future GOES satellites, the next of which is scheduled for 2020, could be pushed back if experts and engineers aren’t able to figure out what went wrong with the older sibling satellite.

“There’s no doubt that the problems we are experiencing with the cooling system are disappointing and not what we expected of GOES-17 when we launched,” said Steve Volz, assistant administrator for NOAA’s satellite and information service. “But we are committed to getting this right, and we will figure out what happened on [GOES-17] so it doesn’t happen with our other GOES satellites.”

Pam Sullivan, the GOES-R program director, said the loss in functionality has been traced to a loop heat pipe, an integral part of the Advanced Baseline Imager’s cooling system. The pipe carries a coolant called propylene and helps keep the instrument at its optimal temperature of -350 degrees Fahrenheit.

The pipe was made by Northrop Grumman, Sullivan said, and the company is working with officials on the ground to further investigate. The investigation suggests the most likely causes are mechanical damage to the pipe, an issue with gas inside the pipe or potentially foreign object debris, but no conclusion has been made. Tests could take up to three months, Sullivan said, and will eventually involve the manufacturer of the Advanced Baseline Imager, Harris Corp.


This fact sheet (PDF) explains more about the loop heat pipe issue.

Click here to listen to the audio from the July 24, 2018 GOES-17 ABI media call.

Advertisements

103 thoughts on “NOAA’s newest GOES 17 weather satellite may not be fixable, and a loss

      • And even then, only the infrared and near-infrared imagers are affected. So no, not a total loss.

    • Drop in the bucket compared to money wasted on fake Climate Change remedies. And like the Climate Hustle, that money helped pay a lot salaries in California where most of Northrup-Gruman space hardware facilities reside.

    • “$11 billion up in smoke”

      Not really. It’s one of four “identical” satellites. One — GOES 16 — has been on station for a couple of years with no significant problems. Two more — 18 and 19 — are yet to be launched, and NOAA presumably has considerable interest in making sure that they don’t have sensor cooling problems. Even Goes 17 in reduced capability mode is presumably better than nothing.

      • LHPs are usually redundant, i.e. there is always a second identical LHP that is isolated from the 1st. If one fails the other is fully capable of assuming the load. Given the importance of this imager, I’d be surprised if a single point failure was tolerated in the LHP design.
        However, given the comments in the article I am led to believe that in fact there was just one LHP, I wonder who signed off on that risk?

        It would be interesting to know the qualification heritage of the LHP and the imager thermal control. Anyone?

        • re: “qualification heritage”

          Himawari-8 uses the same ABI imager, with the same cooling system: it’s been operational since October 2014 / still working great on all channels:
          http://rammb.cira.colostate.edu/ramsdis/online/himawari-8.asp

          GOES-16 has been operational for approximately 18 months / still working great on all a channels:
          http://rammb-slider.cira.colostate.edu/

          Himawari-9 was launched around the same time as GOES-16, tested A.O.K on all channels (currently in standby mode to replace Himawari-8)

          No failures on 3 of the 4 ABIs deployed to date.
          Approaching 4 years of fault free operation for the 1st unit.

    • Loss of function is only partial and only part time. (i.e. everything works properly most of the time right now). As the satellite enters the worst thermal conditions, additional loss of function on the IR sensors is expected but again that will not be all the time. Satellite operators have developed some pointing options that help minimize the impact of the poor cooling performance.

      This is a serious problem that needs to be investigated and understood in order to plan future operations and to prevent repeats in future missions; but… we are not helped by people going overboard in their criticism of NOAA (they deserve a lot).

  1. elone musk will be able to replace it….

    (just give him a 14 billion loan (with an 8 year forgiveness clause) and a 2 billion purchase agreement once it is operational and in orbit, and lifetime no income tax agreement)

    • Now if he can put a car almost in orbit that can travel almost to Mars he very well might be almost able to do it

      • no hate here, just a little joking based on the recent musk claim that he could submarine into the flooded cave (HE IS STARTING TO BELIEVE THAT HE CAN DO ANYTHING) and rescue the kids.

        and a tie-in that, given appropriate subsidies/loans, he can do a proportionately bigger amount of anything.

        (with appropriate respect to your illogic, why are you so in love with musk?)

      • Is this a new form of argument from authority? Unless you have accomplishments that exceed those of the great one, you aren’t allowed to criticize the great one?

    • Umm, he just launched a huge satellite to GTO and a constellation of 10 Iridium satellites in the past 3 days. And her recovered both first stage boosters. What a failure!

  2. ..is broken and officials still aren’t sure what’s wrong or how to fix it.

    1. It’s a virus called Windows
    2. Format and upgrade to DOS

    • if only they could only suck all the co2 out of the satellite to reduce the excess heat 😉

  3. You can bet that the pre-installation flight qualification and testing of that ABI particular cooling system were are all signed off by individuals on engineering teams at the various manufacturers. Lots of documentation is involved in the testing and check-outs at each step of assembly and functional testing of space flight hardware. Sort of like the “this garment inspected by XXX” tag in many high end clothes and suits with a lot more riding on it. No doubt that paperwork is being examined quite closely.

    I remember when the Hubble bad mirror issue was finally identified in the early 90’s after it was launched and “first light” images came back fuzzy. Boiled down to an engineer in the mirror grinding/polishing team installing the sticks in backwards that held the device that measuring device that guided the mirror polishing heads. And that error changed ever so slightly how the final surface of the mirror was altered from specification, it wasn’t quite the perfect spherical shape needed. No one caught the error of course until the fuzzy images came down from space. A lot of investigation went into how such a perfectly polished mirror had a uniform spherical aberration. I’m sure that engineering team felt pretty crappy for a long time.

    • “I’m sure that engineering team felt pretty crappy for a long time.”
      =========
      I’m sure they still do.

      • Building and installing the corrective optics created a lot of jobs, and Hubble was brought up to its original design specification.

        The repair work was hailed at the time as shining example of engineering problems being solved and overcome and having a Space Shuttle system to send astronauts up to install the corrective optics as a first ever repair mission on a broke satellite.

        And Hubble has performed so remarkably well in decades that being a part of that is justifiably a reason to be proud of the effort that went into creating the mirror.

        Sadly, there’s no one going to Geosynch orbit to fix a GOES problem. Just think if all the money wasted on the non-problem of Climate Change was spent on creating and building robotic satellite repair technology. For just a fraction of Climate money, we could’ve had a fleet of AAA-like on-call robotic repair robots ready to go up to Geo orbit to fix it, change a tire, replace a battery or an IR cooler.

      • Thanks DJ!
        That’s good link to the technical description of what went wrong on the Hubble mirror. It had been decades so all I remembered was something about a rod (or stick) installed wrong in the measurement device on the polisher. I hadn’t realized that it was made 10 years before its 1990 launch date, basically around 1980.

      • It certainly blows away the excuse most documentaries use: “They didn’t expect the mirror to change shape so much in zero gravity”.

    • Of course we fix Hubble with a special Space Shuttle mission, something we can no longer do for GOES.

        • Yeah, big difference between 300 miles up and 22,000 miles up. On the other hand, I believe the shuttle main engines are restart-able, so the question would be, “Can we boost enough fuel to get to GOES and back”. Emphasis on the “getting back” part of the trip. You might have to stockpile the fuel over several lifts.

          • I was wondering how much fuel an exo-suit would require to get up there and back.
            As well as other consumables.

          • Even if a shuttle had been provisioned with fuel to get there and back, a huge radiation problem exists between 600 miles and 6000 miles.
            To get to equatorial Geosynch orbit where GOES is positioned, a vehicle would transit the Van Allan radiation belts going up and then back, that is twice. The Shuttle wasn’t rated/qualified for flight through that environment and it would have been hazardous for the humans onboard. Not worth saving a weather satellite for that risk. Robotics are the answer, controlled by an engineering team safely on Earth or in LEO.

            The Apollo missions managed the VA Belts exposure by avoidance and speed.
            Except for Apollo moon missions, no humans have ventured beyond LEO.

          • Send a robot craft to retrieve the GOES satellite and return it to the space station for repairs then return it to its higher orbit.

            That sounds like a good development project for NASA. We are going to need an orbital transfer vehicle if we are going to do anything in space other than go to low-Earth orbit.

          • I like that idea! Assuming there a suitable hard points on the satellite, and in its deployed condition nothing will snap off or break loose during orbital transfer.

    • I recall that there was a paint chip between two surfaces that caused the mirror to have a slightly incorrect radius. Not so? If the part was installed backwards, that could explain the paint chip. Different overlap at each end.

      Have you seen how the error was corrected? It was extraordinarily complex. They were not even sure it would work, before it was installed.

  4. The Hubble story is worse than that, NASA paid for two “primary mirrors” (the big shiny bit) to be built by two different vendors. Each was a “mirror copy” (sorry) of the other. This was to reduce the risk if something happened to this critical part.

    Eastman Kodak (the vendor that made the perfect mirror now on display at the Smithsonian) included in their bid the costs to ship each mirror to the other vendor (they are only ~500 miles apart) for a “cross check”. Test each mirror with each vendors test equipment. This would validate the test measurements, allow NASA to pick the “best” mirror before assembly of the telescope and make sure the mirror met the specs. This would cost about $10 million (1980’s dollars).

    NASA decided that was not necessary and saved $10 million……

    The Hubble primary mirror is nearly perfect, the “radius” of the curve is perfectly precise (within nanometers) but it is the wrong “radius” for the rest of the telescope optics. This is a very simplified explanation.

    Eastman Kodak also bid on assembling and testing the telescope, suggesting a “full up”optical end to end test after all assembly steps are complete. That would have cost about $75 million, once again NASA decided that was not necessary and saved lots of money….. They decided that measuring all the parts before assembly was good enough, there was no need to test them again after assembly…

    Kodak; “Measure again and again and again, then launch”
    NASA; “Measure once, assemble and launch”

    The corrective optics did put the performance back to the original spec, but it meant the loss of one instrument that had to be removed to “jam in” the fix.

    Cheers, KevinK

    • Also, the follow on to the Hubble is the James Webb Space Telescope, it recently went through a “full up, end to end” optical test of the completely assembled telescope. This was done by the “descendants” of the Kodak folks (now a different corporate entity) that recommended a “full up, end to end” test for the Hubble.

      The group that manufactured the perfect Hubble Primary Mirror has a 50 year track record of no failures on orbit (for the sub-systems they designed/manufactured/tested). They also manufactured the Digital Globe Earth imaging satellite cameras in the same factory where the Perfect Backup Hubble mirror was “born”.

      Cheers, KevinK

  5. I guess I should apologize for being a bit intolerant. I am highly impressed with the quality of knowledge and intellect on this web site, and so I’m less tolerant of “fake” and bad English. As written in the article above, “loss in functionality”, “functionality” is not a real word, the phrase is supposed to be “loss in function”. Adding “ality” to “function” does not help the meaning of the word, make it “more real”, or show superior intellect. What I find at this web site are many people who are excellent at finding that information A plus information B proves to be always result C, and if there are faults in the testing/reasoning, can find them, point them out and be able to explain things so even I can understand . This kind of web site I enjoy reading. Thank you. John Minich

    • I don’t have precise numbers, but let’s assume that 5 instruments out of 10 work only part-time if ever (I have a great trouble with the 12-hours-per-day number). Is that a loss of function? To me that would indicate 10 dead instruments.

      Merriam-Webster: Examples of functionality in a Sentence
      – a design that is admired both for its beauty and for its functionality
      – The cameras are comparable in price and functionality.
      – a device with new functionalities

    • JM,
      To me, a “loss of function” would imply a complete failure of some critical component of a system. Whereas, a “loss of functionality” implies a softer failure such as degraded accuracy or precision. It is a nuanced difference; however, I don’t have a problem with “loss in functionality,” unless it was used to describe a complete failure. However, in this case what was described is a duty cycle of 50% of the design spec’s.

    • Ah well. Twas ever this.

      Here in the UK you get burgled by burglars.

      I was amazed to hear Cagney (or possibly Lacey) use the term ‘burglarized’

      But my amazement knew no bounds when I learned that this was not some scripting of a dumb-ass cop’s poor use of their native language. No, this was the ‘correct term to use in the USA.

      The country that turned ‘new clear’ into ‘nookoola’ and gave it to the rest of the world.

      • add shined a light- not shone
        trodded water..i kid you not!
        I’ve found, in more than one book, both of these abuses of the language.
        my all time favorite is Solder being pronounced as Sodder..I used to work in board assembly so it drove me bonkers.

    • A hammer has a function: pounding nails. It is also a functional window breaker;
      but, it’s functionality as a screwdriver leaves something to be desired.

      • When all you have is a hammer, everything else is a nail. What is the plural ? Funtionalities?

    • I severed a tendon in my hand. It didnt lead to a total loss of function, but the functionality of my hand was impaired. I can no longer flip off idiots

  6. Would have paid for half of Trumps wall,….

    11 billion, its a sick joke isn’t it really, the un-believable waste.

  7. One might feel a pang also for the cast of characters who had their names read out as the Sputnik ascended on its launch day. It was sooooo tempting fate.
    So now, Sid Scientist, Ed Engineer and Tina Tea-lady must be hiding under rocks right now.

    Even worse, all the unfortunates this thing was going to rescue, from the own idiocy inside meteorological or maritime misadventures. Hopefully now that Sputnik is on the scrap heap (bonfire) they may actually think twice and take a little more care of themselves.
    Newspapers and The Interweb might have to publish Good News, for a change.
    I ain’t holding any breath..

    One *really* has to feel for the proponents of GHG Theory.
    (For the record let it be noted that my first attempt to open that ‘heat pipe’ pdf crashed my Windows PC)

    One must ask why the infra-red sensor has to cooled to (precisely) minus 350 F in order to be sensitive to infra red (IR) radiations coming from the atmosphere & surface.
    GHG Theory says that atmospheric gases, dirt and water are sensitive to this IR at more ordinary temperatures. They seemingly absorb the IR energy and give a visible/recordable response at temperatures of circa 15 degC. (or whatever Earth’s average temperature is these days)

    Isn’t it odd that minus 350F is considerably less than the minus 110F that CO2 radiates at?
    Thermodynamics says its not odd at all and is an actual prerequisite for the IR energy to be absorbed, and hence be measurable – whether it creates a voltage in a circuit or raises the temperature of the object it impinges upon.
    Same thing really. We can see and record either.

    GHG theory also says that vanishingly low power levels (way down in the noise – has anyone properly measured them yet) cause temperature rises of whole numbers of degrees centigrade – *very* easy to measure even without a Sputnik.

    Their pdf file describing heat pipes, their purpose and function makes a TOTAL MOCKERY of the theory of the GHGE.

    Who’s going to tell them?

    • Infra-red sensors have to be cooled down so that IR radiation from the instrument itself doesn’t swamp the signal that you are looking for.

    • The instrument has to be colder than the uppermost layer of the atmosphere that is radiating. It is very cold up there. Obviously there is a cutoff energy level below which it makes no difference. It’s good enough for government work.

  8. The structure of the propylene coolant is CH2=CH-CH3.

    It melts at -301.4 F and boils at -53.3 F. It is physically incapable of cooling something to -350 F.

    But here’s an interesting suggestion for the source of the problem, that doesn’t appear in the list of possible causes. That “=” in the structure is a double bond.

    Propylene (but not propane) is susceptible to radical polymerization. Near-Earth space is a high radiation zone.

    If propylene experiences a flux of radiation, it could very well spontaneously polymerize into — wait for it — polypropylene.

    Here’s how that works: CH2=CH-CH3 + (high-energy particle) –> (•)CH2-CH(+)-CH3 + electron (which zooms away).

    A positive charge is left behind on the middle carbon, with loss of a negative electron. The “CH2(•)” indicates a methylene radical. The central positively charged electron deficient carbon site attacks another propylene molecule.

    (CH3)(CH2•)>CH(+)- + CH2=CH-CH3 –> (CH3)(CH2•)>CH-CH2-CH(+)-CH3

    The dimerized product at the first step remains electron deficient and goes on to attack another propylene. … and again and again and on to larger things.

    Something similar — radical polymerization — can happen starting at the terminal radical site as well.

    The result will be a solidified coolant, or at least a coolant that becomes more and more viscous.

    One wonders why they chose propylene as their cooling fluid when propane (CH3-CH2-CH3) is unreactive and has about the same thermal range (melts -309.5F; boils -43.8 F).

    • I guess the answer to the polymerization question is to find out if propylene is/has been used on other similar satellite systems and if so, when, and for how long, and are the systems still in use and performing properly. If the answer is “we’ve been using this stuff for a LONG time and have had NO similar issues” then radiation caused polymerization is NOT a probable cause.

      • NW Sage,
        “…then radiation caused polymerization is NOT a probable cause.” Unless this particular satellite has experienced a cosmic ray impact that others have not.

      • GOES16 is “identical” to GOES 17 (whatever “identical” means in NOAA-speak) and has been in a similar geostationary orbit since November 2016. I would assume the sensor cooling technology is the same as GOES-17 — except that GOES16 works.properly.

        • Its the exact same imaging system. It uses the same heat pipes.
          Sort of rules out design flaw.

          • “Sort of rules out design flaw”

            Probably. My limited and tangential experience has been that multi-million dollar platforms are rarely truly part for part “identical” over time. But I assume that NOAA has a change control process so any differences should be known It’s a dead certainty that any “improvements” in the heat pipes and imaging system between GOES16 and GOES17have already been gone over in excrutiating detail.looking for any possible cause of the problems with 17.

          • Haha.

            Take two identical pencils, balance them on their points, and tell me that the second will fall in the same direction as the first.

            Maybe the design flaw was a failure to accommodate unpredictability…

          • Not really. Early craft is working, later one has a failure. It could be that the early craft is on the verge of failing and will will not know until it does. With such low production volumes it is very difficult to predict real life failure rate, even with extensive and extreme testing/proving.

      • I had the same thought, NW Sage, and looked for coolant information for GOES 16 and also in the 2005 BAMS paper that described the GOES R satellite systems.

        No dice. I didn’t find any info on coolant liquids.

        If anyone else knows, I’d be interested to find out, too.

      • If Pat had said “LEO”, I’d agree with you.
        But “Near Earth” is pretty undefined. The VanAllan Radiation belts between 600 miles and 6000 miles are intense zones of “Near Earth” radiation because of the way the Geomagnetic shield interacts with the solar wind’s protons and electrons. More radiation there than in Geosynch orbit.

    • It probably has a mechanically driven Stirling engine cooler concentrating the heat, thus it can in fact cool to -351.

  9. “The investigation suggests the most likely causes are mechanical damage to the pipe, an issue with gas inside the pipe or potentially foreign object debris, but no conclusion has been made”

    So it could be anything, basically? 😀

    Are they sure it’s not working? Could it just be it isn’t showing AGW so they are convinced it must be broken? 😀

    • The issue is that this is a tried and proven design. This specific heat pipe design is already working on other spacecraft. So, we can rule out a design flaw. The next train of potential errors come from installation. Ostensibly the assembly was meticulously inspected and the proper procedures signed-off. Then there leaves potential damage from launch and orbital insertion.
      These things are very complicated and thoroughly tested, however space flight is risky.
      It is rocket science after all. 🙂

      • rocketscientist,
        Or an improbable, but possible, micro-meteor impact after insertion into orbit? Without the ability to inspect it in situ it is difficult to rule out such low-probability events.

  10. so its GOES has gone t1tsup?
    oops
    not another 20c grommet to save a few bucks one hopes

  11. The James Webb IR Telescope is delayed yet again – the huge sunshield has problems (folding?). It seems GOES does not use this kind of shield.
    Pity, can’t wait for the high redshift data.

  12. Cooling issues……. Prb’ly had to substitute some inferior cooling fluid because of environmental concerns. Don’t want to destroy the Van Allen Belts…..

  13. What’s wrong with it? Is it providing accurate temperature information rather than properly “adjusted” data?

  14. The truth is, scientists are infinitely less skilled today than in the past, take out linear tech advances and compare talent.

    Computers do everything today, and scarily it is more and more computers that are doing the science and not the scientists

Comments are closed.