This is the little known story of what recently happened to the SORCE spacecraft and how it was nursed back to a mostly operational status over a period of weeks, after nearly dying in the cold of space due to what appears to be a software glitch. First, some background.
The Solar Radiation and Climate Experiment (SORCE) is a NASA-sponsored satellite mission that is providing state-of-the-art measurements of incoming x-ray, ultraviolet, visible, near-infrared, and total solar radiation. The measurements provided by SORCE specifically address long-term climate change, natural variability and enhanced climate prediction, and atmospheric ozone and UV-B radiation. These measurements are critical to studies of the Sun; its effect on our Earth system; and its influence on humankind.
The SORCE spacecraft was launched on January 25, 2003 on a Pegasus XL launch vehicle to provide NASA’s Earth Science Enterprise (ESE) with precise measurements of solar radiation. It launched into a 645 km, 40 degree orbit and is operated by the Laboratory for Atmospheric and Space Physics (LASP) at the University of Colorado (CU) in Boulder, Colorado, USA. It will continue the precise measurements of total solar irradiance (TSI) that began with the ERB instrument in 1979 and has continued to the present with the ACRIM series of measurements. SORCE also provides the measurements of the solar spectral irradiance from 1nm to 2000nm, accounting for 95% of the spectral contribution to TSI.
SORCE carries four instruments including the Spectral Irradiance Monitor (SIM), Solar Stellar Irradiance Comparison Experiment (SOLSTICE), Total Irradiance Monitor (TIM), and the XUV Photometer System (XPS).
What happened: The spacecraft went into Safe Hold on Sunday, Sept. 26th. The failure appears to be due to a zero length data packet that scrambled the software control due to it being unable to handle the error condition. 2 of 3 reaction wheels (like gyros) failed, sun aligned attitude for solar cell charging was lost, batteries discharged, and the temperature of the spacecraft internal electronics plunged to as low as -30°C. Recovery of the spacecraft took three weeks of work and the support of 82 ground tracking stations along with data relays via NASA’s TDRS network.
From the SORCE weekly status reports from 9/23 – 10/13:
SORCE experienced an OBC reset at 2010/269-17:57:40 due to the MU sending a CCSDS packet with a length of zero. An OBC reset results in the satellite regressing to Safehold, and performing basic power and attitude maintenance on the APE processor.
The following activities were performed to recover the observatory.
- Spacecraft time was jammed and data dumped from after the anomaly
- The OBC was reset to resync with the 1553 bus
- OBC patches 6.8, 6.9 and 7.0 were loaded
- All the spacecraft tables and RTS’s were reloaded
- SORCE was then commanded out of safehold and back to OBC control on DOY 269.
At the next contact after exiting safehold, the spacecraft was found to be operating on one reaction wheel with lower than expected power margins. The flight operations team manually shed RWA 3, both star trackers and turned off the transmitter and commanded safehold.
Poor pointing performance lead to low battery charge state, and APE power charging tables changed to charge at a higher value on the APE. To recover from this configuration and a hybrid flight software configuration, the following actions were performed:
- RWA FSW process was reinitialized to clear the faults.
- The OBC AC control tables were poked for 3 wheel control.
- The OBC was reset to clear any lingering issues which might have existed
- OBC patches 6.8 and 6.9 were loaded. OBC 7.0 for two wheel control was not loaded at this time due to questions about it’s performance.
- All the spacecraft tables and RTS’s were reloaded
- RWA over speed was disabled in table 108. It was determined that RWA 4 had an over speed fault which lead to the transition to one wheel control in contingency mode.
- SORCE was then commanded out of safehold and back to OBC control on DOY 272.
A phased approach was taken to re-warm the satellite back to operational temperature. The degraded battery necessitated this approach. To duty cycle the battery heater around eclipse as is done in normal operations, a special ATS was loaded that included commands to power off the heater in eclipse. Over the next several orbits after exiting safehold the Star Tracker heaters and Instrument Bench heaters were enabled.
- Star Tracker 1 and Star Tracker 2 were powered on DOY 273.
- SORCE was also commanded to normal pointing mode on DOY 273.
The operational temperature of the battery with duty cycling and the instruments off was cooler than desired. The redundant battery heater set point was raised to cycle between 1 and 2 deg. C. that improved battery performance.
The instrument suite was recovered as follows:
- MU turned on and FSW patched and configured for normal operations on DOY 274
- TIM turn on and science operations began on DOY 275
- SOLSTICE B turn on and science operations began on DOY 275
- SIM B turn on and science operations began on DOY 276
- On DOY 277 SOLSTICE A was turned on. Due to a known “feature” where sending an instrument turn off command, followed by another instrument turn off command will power off the instrument after a turn on command, SOLSTICE A turn on was not successfully completed. Successful turn on and return to science was completed on DOY 278.
- XPS was turned on and began taking science on DOY 279.
Read the entire summary at the SORCE weekly status reports from 9/23 – 10/13
The most recent SORCE weekly status report on 11/18/2010 is a bit more encouraging, as they finally got the SIM B instrument back online and the temperature/heater/available battery power situation seems to be managed now.
Here’s TSI data from SORCE regularly plotted by Dr. Leif Svalgaard along with other solar data, click to enlarge.
h/t’s to Leif Svalgaard and Harold Ambler