I’m a bit of a zombie myself as I write this, as I was up until 4:30AM PST dealing with this problem which turned out to affect National Weather Service offices nationwide in addition to my business, which uses the same satellite data feed. It is a rather interesting story of technology gone “rogue” and it all started last spring with one telecommunications satellite that turned naughty last April:
It seems a solar storm was to blame:
(more on this later at the end of the article)
They can’t control it any more, but the transmitter remains on:
Which brings us to last Friday, while I was out of the office, my staff was dealing with intermittent signal problems with our satellite data feed. The NWS Wallops Island uplink boosted the signal strength on the transponder, and the problem was solved…or so they thought.
Yesterday, all hell broke loose and we got this ugly message:
*Topic: *Family of Service Data Outage* * *Date/Time**:*December 05, 2010 2110 UTC* * *Product(s) or Data Impacted:*Single Field of View East and West*,*GOES LCB Ash,Blended TPW for GPS _NOAAPort and GPS_NOAAPort_Dist1, GOES East and West ASOS Imager and DPI WEST, ** *Date/Time of Initial Impact:*December 05, 2010 1900 UTC** *Date/Time of Expected End:*Unknown** *Length of Outage:*TBD* * *Details/Specifics of Change:*The Family of Service at NSOF is down and has affected the above products to be unavailable until further notice. Tech. control in Silver Spring reports, the problem is due to the NOAAPORT Satellite at NCF AWIPS having poor signal quality . The problem is being investigated.* *
That AWIPS acronym Advanced Weather Interactive Processing System (AWIPS) is for the internal computer system that the National Weather Service uses.
And they were down too. This is significant, since the AWIPS system is used to issue severe weather bulletins (via WarnGen) at every NWS WSFO in the country now, and they all share this common datafeed from the satellite, which the private sector can also tap into via a program known as NOAAPORT.
Supposedly, the satellite data problem was solved last night at 6:59 UTC, but we were still down two hours later. I put in a call to the NWS Telecommunications people and asked for help. They seemed clueless, and wanted to know “What Air Force base are you with?” when I mentioned NOAAPORT, giving me my first clue that the problem was even more widespread. There are a number of Met support offices at AFB’s that use NOAAPORT also. I left a number, and about 30 minutes later got a call back. They said we should be getting data, but in fact we weren’t, they said “we’ll look into it further” which is usually code for “we don’t know why”.
About an hour later, about 4:00 AM PST I got a call from a tech who said that he had noticed my trouble ticket, and told me very matter-of-factly that “you probably haven’t been given all of the information you need”.
He then proceeded to tell me that to solve the problem of the data outage (and at that time, I had no idea what was causing the problem) they had to switch transponder frequencies, and that I had to set my receiver to a new frequency. Did that, and it came right up. Whew!
My complaint about such info not being made public on NOAA’s status boards was met with “Sorry, that’s not my division”. I thanked him and hung up.
Later this morning, the real reason became clear with this message:
NOAAPORT SATELLITE INTERFERENCE
Effective: 12/4/10 – The Noaaport satellite transmission is subject to interferrence from this date forward thru 12/18/10. The below notice is the NWS’ official notice to-date regarding this interferrence.
Complete loss of Novra receiver satellite lock is being experienced due to the interference from the Galaxy15 satellite.
The NWS is now simulcasting the noaaport broadcast on an alternate frequency of 1138.5Mhz. Use your S75 receiver console software to enter in the alternate frequency if your receiver is not staying locked on the original frequency of 1193.5 More info here
NOXX10 KWBC 301457
DATA MGT MESSAGE 11-10.05
TO NOAAPORT USERS
FROM RTH WASHINGTON DATA MANAGEMENT
SUBJECT SBN/NOAAPORT POTENTIAL SATELLITE INTERFERENCE – NOTICE 1
EFFECTIVE DATE 12/12/10 TO 12/18/10 /FOR A PERIOD OF 2 TO 10 HOUR
A RISK HAS BEEN IDENTIFIED WHEN THE GALAXY 15 ROGUE SATELLITE TRAVERSES THE SES-1 ORBITAL POSITION WHICH MAY RESULT IN A POSSIBLE LOSS OF DATA VIA THE NWS AWIPS SBN/NOAAPORT FEED. THIS POTENTIAL INTERFERENCE MAY TAKE PLACE BETWEEN 12/12/10 AND 12/18/10 INCLUSIVE.
ACCORDING TO OUR SATELLITE CONTRACTOR. THE DURATION OF THE INTERFERENCE EVENT COULD BE ANYWHERE FROM 2 HOURS TO 10 HOURS. WE ARE CONTINUING TO RESEARCH AND ANALYZE THE FULL IMPACT AND RISK TO THE AWIPS SBN / NOAAPORT PROGRAMS.
More here from NOAA’s Network Control Facility (NCF)
Yep, the “rogue” Galaxy 15 satellite was spewing C-band microwaves as it drifts in space, interfering with critical downlinks used by NOAA and the U. S. Air Force. It almost sounds like an idea for a rouge nation to use to interfere with satellite communications.
Here’s what a recent story from space.com had to say:
The 4,171-pound (1,892-kg) Galaxy 15 satellite went rogue on April 5, when it stopped responding to controllers on the ground.
Yet, while the satellite veered from its assigned orbital slot of 133 degrees west longitude, 36,000 kilometers over the equator, the “zombie satellite” maintained an active payload, with its C-band telecommunications still functioning.
Interference from Galaxy 15’s stuck-on signal is the main concern, since the chance of it actually crashing into other satellites is remote to non-existent, Intelsat officials have said.
Several attempts to shut down Galaxy 15 have been unsuccessful, leaving the defunct satellite drifting in the cosmos.
“Normally when an anomaly occurs, the satellite just stops working and we don’t have to worry about it,” Good said. “Galaxy 15 is still operational, so in this case, the satellite is still “functioning” in a deterministic state. But, we know exactly where it is, we know what it’s doing, and we know the settings of the satellite.”
The future of zombiesat
Eventually, Galaxy 15 is expected to lose its Earth-pointing capability. Once this lock on Earth is lost and its solar panels are no longer pointed at the sun, the satellite’s battery power will eventually die.
“When the battery power decreases past a certain threshold, the payload will shut off,” Good explained. “It will no longer receive and transmit, and its batteries will continue to deplete.”
The satellite could also reach a threshold that causes its onboard computers to reset, said Good, but the possibility of this happening is still unknown.
“There is a possibility that the onboard computer could reset, but we don’t know what that probability is,” he said. “Still, there is a chance. It would almost be like a “control-alt-delete” on your computer. It would begin sending telemetry again. It would wake up and realize ‘What am I doing here?'”
Murphy’s law turned roguesat took down pretty much the entire forecast and warning infrastructure of the National Weather Service last night, and it seems nobody in the media has noticed. I certainly wouldn’t had I not been in the middle of it.
On the plus side, this didn’t happen in the middle of a tornado outbreak, so we can all count ourselves fortunate. But, this incident does underscore just how sensitive our space technology can be to solar flares.
Here’s an image of the likely solar culprit that made Galaxy 15 into a “zombie”: