I’m a bit of a zombie myself as I write this, as I was up until 4:30AM PST dealing with this problem which turned out to affect National Weather Service offices nationwide in addition to my business, which uses the same satellite data feed. It is a rather interesting story of technology gone “rogue” and it all started last spring with one telecommunications satellite that turned naughty last April:
Intelsat Loses Contact with Galaxy 15 Satellite | SpaceNews.com
It seems a solar storm was to blame:
FoxNews.com – Intense Solar Storm Spins Satellite Out of Control
(more on this later at the end of the article)
They can’t control it any more, but the transmitter remains on:
SPACE.com — Attempt to Shut Down Zombie Satellite Galaxy 15 Fails
Which brings us to last Friday, while I was out of the office, my staff was dealing with intermittent signal problems with our satellite data feed. The NWS Wallops Island uplink boosted the signal strength on the transponder, and the problem was solved…or so they thought.
Yesterday, all hell broke loose and we got this ugly message:
*Topic: *Family of Service Data Outage* * *Date/Time**:*December 05, 2010 2110 UTC* * *Product(s) or Data Impacted:*Single Field of View East and West*,*GOES LCB Ash,Blended TPW for GPS _NOAAPort and GPS_NOAAPort_Dist1, GOES East and West ASOS Imager and DPI WEST, ** *Date/Time of Initial Impact:*December 05, 2010 1900 UTC** *Date/Time of Expected End:*Unknown** *Length of Outage:*TBD* * *Details/Specifics of Change:*The Family of Service at NSOF is down and has affected the above products to be unavailable until further notice. Tech. control in Silver Spring reports, the problem is due to the NOAAPORT Satellite at NCF AWIPS having poor signal quality . The problem is being investigated.* *
That AWIPS acronym Advanced Weather Interactive Processing System (AWIPS) is for the internal computer system that the National Weather Service uses.
And they were down too. This is significant, since the AWIPS system is used to issue severe weather bulletins (via WarnGen) at every NWS WSFO in the country now, and they all share this common datafeed from the satellite, which the private sector can also tap into via a program known as NOAAPORT.
Supposedly, the satellite data problem was solved last night at 6:59 UTC, but we were still down two hours later. I put in a call to the NWS Telecommunications people and asked for help. They seemed clueless, and wanted to know “What Air Force base are you with?” when I mentioned NOAAPORT, giving me my first clue that the problem was even more widespread. There are a number of Met support offices at AFB’s that use NOAAPORT also. I left a number, and about 30 minutes later got a call back. They said we should be getting data, but in fact we weren’t, they said “we’ll look into it further” which is usually code for “we don’t know why”.
About an hour later, about 4:00 AM PST I got a call from a tech who said that he had noticed my trouble ticket, and told me very matter-of-factly that “you probably haven’t been given all of the information you need”.
He then proceeded to tell me that to solve the problem of the data outage (and at that time, I had no idea what was causing the problem) they had to switch transponder frequencies, and that I had to set my receiver to a new frequency. Did that, and it came right up. Whew!
My complaint about such info not being made public on NOAA’s status boards was met with “Sorry, that’s not my division”. I thanked him and hung up.
Later this morning, the real reason became clear with this message:
NOAAPORT SATELLITE INTERFERENCE
Effective: 12/4/10 – The Noaaport satellite transmission is subject to interferrence from this date forward thru 12/18/10. The below notice is the NWS’ official notice to-date regarding this interferrence.
Complete loss of Novra receiver satellite lock is being experienced due to the interference from the Galaxy15 satellite.
The NWS is now simulcasting the noaaport broadcast on an alternate frequency of 1138.5Mhz. Use your S75 receiver console software to enter in the alternate frequency if your receiver is not staying locked on the original frequency of 1193.5 More info here
NOXX10 KWBC 301457
DATA MGT MESSAGE 11-10.05
TO NOAAPORT USERS
FROM RTH WASHINGTON DATA MANAGEMENT
SUBJECT SBN/NOAAPORT POTENTIAL SATELLITE INTERFERENCE – NOTICE 1
EFFECTIVE DATE 12/12/10 TO 12/18/10 /FOR A PERIOD OF 2 TO 10 HOUR
A RISK HAS BEEN IDENTIFIED WHEN THE GALAXY 15 ROGUE SATELLITE TRAVERSES THE SES-1 ORBITAL POSITION WHICH MAY RESULT IN A POSSIBLE LOSS OF DATA VIA THE NWS AWIPS SBN/NOAAPORT FEED. THIS POTENTIAL INTERFERENCE MAY TAKE PLACE BETWEEN 12/12/10 AND 12/18/10 INCLUSIVE.
ACCORDING TO OUR SATELLITE CONTRACTOR. THE DURATION OF THE INTERFERENCE EVENT COULD BE ANYWHERE FROM 2 HOURS TO 10 HOURS. WE ARE CONTINUING TO RESEARCH AND ANALYZE THE FULL IMPACT AND RISK TO THE AWIPS SBN / NOAAPORT PROGRAMS.
More here from NOAA’s Network Control Facility (NCF)
Yep, the “rogue” Galaxy 15 satellite was spewing C-band microwaves as it drifts in space, interfering with critical downlinks used by NOAA and the U. S. Air Force. It almost sounds like an idea for a rouge nation to use to interfere with satellite communications.
Here’s what a recent story from space.com had to say:
The 4,171-pound (1,892-kg) Galaxy 15 satellite went rogue on April 5, when it stopped responding to controllers on the ground.
Yet, while the satellite veered from its assigned orbital slot of 133 degrees west longitude, 36,000 kilometers over the equator, the “zombie satellite” maintained an active payload, with its C-band telecommunications still functioning.
Interference from Galaxy 15’s stuck-on signal is the main concern, since the chance of it actually crashing into other satellites is remote to non-existent, Intelsat officials have said.
…
Several attempts to shut down Galaxy 15 have been unsuccessful, leaving the defunct satellite drifting in the cosmos.
“Normally when an anomaly occurs, the satellite just stops working and we don’t have to worry about it,” Good said. “Galaxy 15 is still operational, so in this case, the satellite is still “functioning” in a deterministic state. But, we know exactly where it is, we know what it’s doing, and we know the settings of the satellite.”
Oh, goody.
The future of zombiesat
Eventually, Galaxy 15 is expected to lose its Earth-pointing capability. Once this lock on Earth is lost and its solar panels are no longer pointed at the sun, the satellite’s battery power will eventually die.
“When the battery power decreases past a certain threshold, the payload will shut off,” Good explained. “It will no longer receive and transmit, and its batteries will continue to deplete.”
The satellite could also reach a threshold that causes its onboard computers to reset, said Good, but the possibility of this happening is still unknown.
“There is a possibility that the onboard computer could reset, but we don’t know what that probability is,” he said. “Still, there is a chance. It would almost be like a “control-alt-delete” on your computer. It would begin sending telemetry again. It would wake up and realize ‘What am I doing here?'”
Murphy’s law turned roguesat took down pretty much the entire forecast and warning infrastructure of the National Weather Service last night, and it seems nobody in the media has noticed. I certainly wouldn’t had I not been in the middle of it.
On the plus side, this didn’t happen in the middle of a tornado outbreak, so we can all count ourselves fortunate. But, this incident does underscore just how sensitive our space technology can be to solar flares.
Here’s an image of the likely solar culprit that made Galaxy 15 into a “zombie”:
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
How’s this for a conspiracy theory…
“…it all started last spring with one telecommunications satellite that turned naughty last April:
and:
“The robotic space shuttle secretly left Earth seven months ago.”
http://pinoytutorial.com/techtorial/x-37b-space-shuttle-secret-mission-new-discoveries/
Douglas DC says:…..
Virgin Galactic has an active orbital transport, easy to attach a missile or 10 to it.
(just saying)
[d]
http://www.telegraph.co.uk/earth/environment/climatechange/8178162/Are-we-freezing-because-of-global-warming.html
I was going to put this in tips and notes, but it is not accepting comments? What happened?
‘Programming error’ caused Russian rocket failure
Has there been a noticeable increase in satellite failures, for various reasons, lately? Or is this to be expected? Given the costs and effort involved, I’d expect better quality control with more backup and damage-preventative measures.
In this case, the satellites were for Russia’s shiny new Glonass navigation system, a rival to the US GPS system. And GPS can be used for missile guidance as well as its normal military applications, and I would expect Glonass to be used the same way… Thus I will freely presume the Russian military, and likely their intelligence service(s), are involved with setting up and running this new system. Which still had this dramatic deployment failure.
BTW, are these the same Russian space experts we’ll be depending on to get to the International Space Station once the US shuttle fleet gets retired? ]:-)
Eggs? Baskets?
Expect it is hard to underestimate the breath & depth of impact anything like a repetition of the Great Solar Storm of 1859 would have on today’s wired-to-the-max ”just in time” world. Except for EMP-hardened MIL-Spec stuff, pretty much everything with even basic modern electronics in it let alone computers would likely go Tango Uniform. . . . Like cars, for instance: 1859 repeat, nobody (or at least very few people (not many Model-As around anymore) ) drive. It would be ugly; and not just for weeks or even months. People in big cities could start to seriously envy those in rural areas with cows and gardens.
And as far as doing something effective to get Galaxy 15 to shut up:
Tough problem. As already noted in prior comment: Easiest thing to do at Geo-sync altitude (note ”easiest” in this case is RELATIVELY speaking) is blow it up; but that’s also the LAST thing you want to do; i.e.: Replacing 1 piece of space junk with 1000s if not more fragments is to be avoided.
Apparently the Sun now has an owner who intends to make money out it. See
http://www.heraldsun.com.au/lifestyle/the-other-side/spanish-woman-claims-ownership-of-sun-wants-to-tax-it/story-e6frfhk6-1225962735887
The property rights registration appears to be purely to make money out it by charging for solar use. With that comes liability: for the damage done by the property. You, NASA, the US Airforce and everyone else affected by the satellite’s down time could file a claim for the CME’s damage.
I was thinking of lodging a claim for all my skin cancers, but they pre-date the registration of ownership by several years, so I think I might be out of luck.
Anthony, thanks for the brilliantly informative bulletin and I hope you get a long, restful night’s sleep tonight.
Makes me regret selling my ’34 Dodge Sedan fifty years ago – almost bulletproof, fast enough for all practical purposes (9 to 90 mph in top gear), big hydraulic brakes (a Chrysler 1st), handled nicely on slippery gravel roads and would plod through deep mud, reasonably light and so wonderfully simple not much ever went wrong with it. My current car, a very good Japanese two litre thingy, is nice, civilised, warm and comfortable among all the ice and snow, but if anything goes awry I would need a mainframe computer to sort it. And it has nil character.
Some very heavy government scientific dudes have done some intense study on just this type of occurrence.
At least one ex military physicist – (a retired Rear Admiral) has released a paper on this subject – it’s out on the net.
Very sobering reading.
Apparently the problem was worse than just changing frequencies without telling anybody. Globecomm apparently tried to also boost the power on the SES World Skies sat to override the interference, causing receivers to tank out due to signal swamping.
Reminds me of the SCTV “CCCP1” episodes where a Soviet satellite kept breaking into the SCTV feed, leading to much hilarity. Though “hilarity” was likely far from your mind at 4 am.
What does one use when the satellite goes down? Divining rods? A weather oracle? A deranged cat? The Farmers Almanac?
😉
DL says:
December 6, 2010 at 7:01 pm
“One would think that their would be some sort of heart beat protocol with the satellite that would be used to maintain control.”
We called that a “dead man timer” and yes it is (or at least was) common practice in computer architecture.
http://en.wikipedia.org/wiki/Dead_man%27s_switch
I suppose we called it a timer instead of a switch in my field (IBM PC compatibles) because it was usually implemented by way of a timer output which was periodically reset by an interrupt service routine. If the ISR failed to reset the timer then when it timed out it raised a signal on an output pin that caused a hard reset to occur.
Not to scare any pantz off of anyone but if it weren’t for reality I’d say it was the Hippie Mega Mind at work calling all stations for ready take over.
But as it has it, reality probably trumps my fiction in this matter, I mean how can a mega mind stack up to the fact that Mannschtick and his minion panzer pantz Romm probably just made a slight schtickup in their trajectory calculation when they tried to go ballistic a ways back, there by hitting the poor hard working (for no pay) satellite square in the barren space like it wasn’t having a hard time already the little one.
There is discussion elsewhere (use Google News/Blog Search for “noaaport galaxy”) that the 12/5-12/6 outage was *not* Galaxy 15 related, but next week there could be. Any updates on this story, Anthony?
The Galaxy 15 thing is a known and relatively predictable issue, it has stepped on other satellites on its way around. Hopefully it turns turtle and dies soon.
Intelsat had gotten together with all the Satellite operators and made very extensive plans to avoid/minimise problems due to Zombie Sat. A great effort of cooperation by Intelsat, admitting it’s their problem and providing all the orbit information to other operators. The orbit is well defined and I cannot believe the NOAA people were clueless this was going to happen!
pRadio
#
#
jimgineer says:
December 6, 2010 at 9:25 pm
Too bad the satellite transmitters are not spread spectrum. Same channels different spreading codes. Probably need a good computer and phase control… It will take years to develop, lots of money. If satellites end up with web browsers and prepaid minutes plans Im going to scream conspiracy.
###
Spread Spectrum is used where appropriate, e.g. the command and control channel. For a number of reasons spread spectrum has not been used on data downlink. BTW, Spread Spectrum was being used in space hardware a decade before most people even heard of it.
Hmmm … malfunctioning ‘transponder’, not transmitter … how does this work?
Input (uplink) on about 4 GHz, heterodyne to 6 GHz, amplify (a lot, via TWT tubes at least a decade back) and that amplified output is fed to a downlink horn (maybe even the same dish used as the uplink).
Does no one else question what the Zombiesat is seeing on the uplink to cause the downlink to actually have ‘signal’ (after all, this is a transponder, not a self-generating signal transmitter in the usual sense)?
This aspect has not been adequately explained to date …
.
Seriously overblown I think.
Seems most of this ‘conjecture’ is based on a few series-wired street-light strings popping fuses in Hawaii …
.
@ur momisugly jimgineer: Why would you need phase control for spread spectrum? Besides, even spread spectrum would not have solved this problem, only ultra wide band would have helped. For some strange reason the satellite community seems skeptical of UWB.
Having made three (OK they were tiny) satellites that are now space-junk, my sympathies with the Intelsat and Oribital folks. But I still dont understand why they did not implement a “dead man switch”! It is not even a “real” switch just software code and pretty damn simple to implement:
START
IF (No input telemetry data for 3 days)
RESTART (main())
END
sheesh! You’d have thought they’d have learned a thing or two by now!
@DesertYote: List several reasons.
@Amit: By phase control I was referring to low phase noise oscillators and ability to compensate for phase jitter in the receiver. I was referring to technologies used in CDMA and TDMA cellular phones.
In adition too watchdog timers, a simple uplink RF carrier detect relay could be interlocked with downlink transmit enable at a hardware level. No computers!!!!
Broadcasters often tie their studio transmitter links to their transmitters in this way to satisfy FCC requirements to be in control of the transmitter in case computerized telemetry fails.
_Jim (~jim):
The first two links say Galaxy 15 has been emptied of all its customers and is no longer broadcasting on those 24 transponder channels, EXCEPT that there was a discrete payload to serve FAA on L-Band and that remains on, listening and rebroadcasting. The prognosis for the sat to lose earth orientation was August 15, obviously wrong, so apparently it’s still listening to earth in the key of L and transponding in kind.
Update Dec. 30: Good news. The zombie satellite is back from the walking dead.
“A zombie satellite that spent months sending out signals while it was adrift in orbit has sprung back to life, resetting itself after its unexplained breakdown in space earlier this year. ”
http://www.msnbc.msn.com/id/40846679/ns/technology_and_science-space/
I have been led to believe that AlGore is inside it trying to change the codes