June 22, 2021 – Testing Underway to Identify Issue and Restore Payload Computer on NASA’s Hubble Space Telescope
NASA continues to work to resolve a problem with the Hubble Space Telescope payload computer that halted on June 13. After performing tests on several of the computer’s memory modules, the results indicate that a different piece of computer hardware may have caused the problem, with the memory errors being only a symptom. The operations team is investigating whether the Standard Interface (STINT) hardware, which bridges communications between the computer’s Central Processing Module (CPM) and other components, or the CPM itself is responsible for the issue. The team is currently designing tests that will be run in the next few days to attempt to further isolate the problem and identify a potential solution.
This step is important for determining what hardware is still working properly for future reference. If the problem with the payload computer can’t be fixed, the operations team will be prepared to switch to the STINT and CPM hardware onboard the backup payload computer. The team has conducted ground tests and operations procedure reviews to verify all the commanding required to perform that switch on the spacecraft.
If the backup payload computer’s CPM and STINT hardware is turned on, several days will be required to assess the computer performance and restore normal science operations. The backup computer has not been powered on since its installation in 2009; however, it was thoroughly tested on the ground prior to installation on the spacecraft.
The payload computer is a NASA Standard Spacecraft Computer-1 (NSSC-1) system built in the 1980s that is located on the Science Instrument Command and Data Handling (SI C&DH) unit. After 18 years on orbit, the original SI C&DH experienced a failure in 2008 that delayed the final servicing mission to Hubble while a replacement was prepared for flight. In May 2009, STS-125 was launched and the astronauts installed the existing unit. The replacement contains original hardware from the 1980s with four independent 64K memory modules of Complementary Metal-Oxide Semiconductor (CMOS) memory. Only one memory module is used operationally, with the other three serving as backups. All four modules can be used and accessed from either of the redundant payload computers.
Launched in 1990, with more than 30 years of operations, Hubble has made observations that have captured imaginations worldwide and deepened our knowledge of the cosmos.
For more information about the Science Instrument Command and Data Handling unit view the following PDF:
Science Instrument Command and Data Handling unit excerpt
June 18, 2021 – Operations Continue to Restore Payload Computer on NASA’s Hubble Space Telescope
NASA continues to work on resolving an issue with the payload computer on the Hubble Space Telescope. The operations team will be running tests and collecting more information on the system to further isolate the problem. The science instruments will remain in a safe mode state until the issue is resolved. The telescope itself and science instruments remain in good health.
The computer halted on Sunday, June 13. An attempt to restart the computer failed on Monday, June 14. Initial indications pointed to a degrading computer memory module as the source of the computer halt. When the operations team attempted to switch to a back-up memory module, however, the command to initiate the backup module failed to complete. Another attempt was conducted on both modules Thursday evening to obtain more diagnostic information while again trying to bring those memory modules online. However, those attempts were not successful.
The payload computer is a NASA Standard Spacecraft Computer-1 (NSSC-1) system built in the 1980s that is located on the Science Instrument Command and Data Handling unit. The computer’s purpose is to control and coordinate the science instruments and monitor them for health and safety purposes. It is fully redundant in that a second computer, along with its associated hardware, exists on orbit that can be switched over to in the event of a problem. Both computers can access and use any of four independent memory modules, which each contain 64K of Complementary Metal-Oxide Semiconductor (CMOS) memory. The payload computer uses only one memory module operationally at a time, with the other three serving as backups.
Launched in 1990, Hubble has contributed greatly to our understanding of the universe over the past 30 years.
For more information about Hubble, visit: www.nasa.gov/hubble
June 16, 2021 – NASA is working to resolve an issue with the payload computer on the Hubble Space Telescope
NASA is working to resolve an issue with the payload computer on the Hubble Space Telescope. The computer halted on Sunday, June 13, shortly after 4 p.m. EDT. After analyzing the data, the Hubble operations team is investigating whether a degrading memory module led to the computer halt. The team is preparing to switch to one of several backup modules on Wednesday, June 16. The computer will then be allowed to run for approximately one day to verify that the problem has been solved. The team would then restart all science instruments and return the telescope to normal science operations.
The purpose of the payload computer is to control and coordinate the science instruments onboard the spacecraft. After the halt occurred on Sunday, the main computer stopped receiving a “keep-alive” signal, which is a standard handshake between the payload and main spacecraft computers to indicate all is well. The main computer then automatically placed all science instruments in a safe mode configuration. Control center personnel at NASA’s Goddard Space Flight Center in Greenbelt, Maryland restarted the payload computer on Monday, June 14, but it soon experienced the same problem.
The payload computer is a NASA Standard Spacecraft Computer-1 (NSSC-1) system built in the 1980s. It is part of the Science Instrument Command and Data Handling module, which was replaced during the last astronaut servicing mission in 2009. The module has various levels of redundancy which can be switched on to serve as the primary system when necessary.
For more information about Hubble, visit: www.nasa.gov/hubble
Hubble is one of the most astonishing pieces of technology ever devised, and its continued service after 27 years of full operational capability (31 years after launch) has delivered a lot of useful information. I hope they can get it back into operation.
It’s good to see Nasa working on a real problem for a change.
It appears the last iteration of the NSSC-1 was in the form of 69 MSI (medium-scale integration) chips … originally it used 1700 SSI (NOR gate) packages.
Shades of the Apollo era computers …
NSSC-1 technical details: https://fermatslibrary.com/s/development-and-application-of-nasas-first-standard-spacecraft-computer
Meh. You have the old chips lying around, have to use them somewhere, and Microchip Baseball is generally frowned upon in most major organizations. So is just sticking them in a power strip and turning on the power to watch them pop…
In an environment where ionizing radiation is a significant issue, the bigger and more widely spaced your transistors are, the fewer failures you’ll have. It’s a tradeoff between lower mass (and higher speed) of your system, higher mass for redundancy, and much higher mass for adequate shielding.
A quick search of the paper popped up this gem:
Think that 1980s technology is old?
We used plated-wire memory in the Peacekeeper ICBM flight computer, specifically because it was designed to fly in a nuclear blast environment (the entire missile was well hardened). It is still used in spacecraft for high-radiation environments, despite its bulk.
I suppose if you wanted higher speed, you could “soup up” the computer using KNORR gates…
re: “Think that 1980s technology is old?”
Please, “whipper-snapper”, I did hands-on work on the TIPI (Tactical Interpretation of Photo Information) computers produced by TI’s Computer PCC (Project Cost Center) in the mid/late 1970’s. I’m no stranger to ‘tech’ of this era. I think I still have my TI TTL LOGIC DATA BOOK from that era (kept only for sentimental reasons at this point). That era was before tri-state logic gates as well; Open collector devices were used instead on data ‘buses’ with simple resistor pull-ups.
BTW, the TIPI computer had two ‘stacks’ of 16 bit wide 32K core memory; Do you know what the MCCR board did in that computer? The name says it all: Memory Control and Current Regulator … that’s what controlled the ‘drive’ and timing to the read and write pulses sent to the mag core memory …
“When I started programming, we didn’t have any of these sissy ‘icons’ and ‘windows.’ All we had were zeros and ones — and sometimes we didn’t even have ones. I wrote an entire database program using only zeros.”
Dilbert replies: “You had zeros? We had to use the letter ‘o'”.
https://dilbert.com/strip/1992-09-08
Maybe they could get help from Southwest.
Thanks for the update – had read about this. It’s actually amazing that Hubble is still working — cosmic or solar radiation can zap computer components. I’m sure they’ll come up w/something to get it back working at some reasonable level.
Could the failure be caused by the very weak magnetic field we have now ? Allowing more cosmic rays and radiation from the sun into low earth orbit. Can we expect more failures in the communication and weather observation craft in orbit. https://electroverse.net/earths-magnetic-field-again-fails-to-deal-with-a-weak-solar-wind-stream/
There is a new one in the wings coming out later this year.
James Webb Space Telescope – Webb/NASA
Webb is going up on a French rocket… may the force be with us.
Hubble is extraordinary. I really hope it can get back to normal operations for the rest of its extraordinary life.