I’m a bit of a zombie myself as I write this, as I was up until 4:30AM PST dealing with this problem which turned out to affect National Weather Service offices nationwide in addition to my business, which uses the same satellite data feed. It is a rather interesting story of technology gone “rogue” and it all started last spring with one telecommunications satellite that turned naughty last April:
Intelsat Loses Contact with Galaxy 15 Satellite | SpaceNews.com
It seems a solar storm was to blame:
FoxNews.com – Intense Solar Storm Spins Satellite Out of Control
(more on this later at the end of the article)
They can’t control it any more, but the transmitter remains on:
SPACE.com — Attempt to Shut Down Zombie Satellite Galaxy 15 Fails
Which brings us to last Friday, while I was out of the office, my staff was dealing with intermittent signal problems with our satellite data feed. The NWS Wallops Island uplink boosted the signal strength on the transponder, and the problem was solved…or so they thought.
Yesterday, all hell broke loose and we got this ugly message:
*Topic: *Family of Service Data Outage* * *Date/Time**:*December 05, 2010 2110 UTC* * *Product(s) or Data Impacted:*Single Field of View East and West*,*GOES LCB Ash,Blended TPW for GPS _NOAAPort and GPS_NOAAPort_Dist1, GOES East and West ASOS Imager and DPI WEST, ** *Date/Time of Initial Impact:*December 05, 2010 1900 UTC** *Date/Time of Expected End:*Unknown** *Length of Outage:*TBD* * *Details/Specifics of Change:*The Family of Service at NSOF is down and has affected the above products to be unavailable until further notice. Tech. control in Silver Spring reports, the problem is due to the NOAAPORT Satellite at NCF AWIPS having poor signal quality . The problem is being investigated.* *
That AWIPS acronym Advanced Weather Interactive Processing System (AWIPS) is for the internal computer system that the National Weather Service uses.
And they were down too. This is significant, since the AWIPS system is used to issue severe weather bulletins (via WarnGen) at every NWS WSFO in the country now, and they all share this common datafeed from the satellite, which the private sector can also tap into via a program known as NOAAPORT.
Supposedly, the satellite data problem was solved last night at 6:59 UTC, but we were still down two hours later. I put in a call to the NWS Telecommunications people and asked for help. They seemed clueless, and wanted to know “What Air Force base are you with?” when I mentioned NOAAPORT, giving me my first clue that the problem was even more widespread. There are a number of Met support offices at AFB’s that use NOAAPORT also. I left a number, and about 30 minutes later got a call back. They said we should be getting data, but in fact we weren’t, they said “we’ll look into it further” which is usually code for “we don’t know why”.
About an hour later, about 4:00 AM PST I got a call from a tech who said that he had noticed my trouble ticket, and told me very matter-of-factly that “you probably haven’t been given all of the information you need”.
He then proceeded to tell me that to solve the problem of the data outage (and at that time, I had no idea what was causing the problem) they had to switch transponder frequencies, and that I had to set my receiver to a new frequency. Did that, and it came right up. Whew!
My complaint about such info not being made public on NOAA’s status boards was met with “Sorry, that’s not my division”. I thanked him and hung up.
Later this morning, the real reason became clear with this message:
NOAAPORT SATELLITE INTERFERENCE
Effective: 12/4/10 – The Noaaport satellite transmission is subject to interferrence from this date forward thru 12/18/10. The below notice is the NWS’ official notice to-date regarding this interferrence.
Complete loss of Novra receiver satellite lock is being experienced due to the interference from the Galaxy15 satellite.
The NWS is now simulcasting the noaaport broadcast on an alternate frequency of 1138.5Mhz. Use your S75 receiver console software to enter in the alternate frequency if your receiver is not staying locked on the original frequency of 1193.5 More info here
NOXX10 KWBC 301457
DATA MGT MESSAGE 11-10.05
TO NOAAPORT USERS
FROM RTH WASHINGTON DATA MANAGEMENT
SUBJECT SBN/NOAAPORT POTENTIAL SATELLITE INTERFERENCE – NOTICE 1
EFFECTIVE DATE 12/12/10 TO 12/18/10 /FOR A PERIOD OF 2 TO 10 HOUR
A RISK HAS BEEN IDENTIFIED WHEN THE GALAXY 15 ROGUE SATELLITE TRAVERSES THE SES-1 ORBITAL POSITION WHICH MAY RESULT IN A POSSIBLE LOSS OF DATA VIA THE NWS AWIPS SBN/NOAAPORT FEED. THIS POTENTIAL INTERFERENCE MAY TAKE PLACE BETWEEN 12/12/10 AND 12/18/10 INCLUSIVE.
ACCORDING TO OUR SATELLITE CONTRACTOR. THE DURATION OF THE INTERFERENCE EVENT COULD BE ANYWHERE FROM 2 HOURS TO 10 HOURS. WE ARE CONTINUING TO RESEARCH AND ANALYZE THE FULL IMPACT AND RISK TO THE AWIPS SBN / NOAAPORT PROGRAMS.
More here from NOAA’s Network Control Facility (NCF)
Yep, the “rogue” Galaxy 15 satellite was spewing C-band microwaves as it drifts in space, interfering with critical downlinks used by NOAA and the U. S. Air Force. It almost sounds like an idea for a rouge nation to use to interfere with satellite communications.
Here’s what a recent story from space.com had to say:
The 4,171-pound (1,892-kg) Galaxy 15 satellite went rogue on April 5, when it stopped responding to controllers on the ground.
Yet, while the satellite veered from its assigned orbital slot of 133 degrees west longitude, 36,000 kilometers over the equator, the “zombie satellite” maintained an active payload, with its C-band telecommunications still functioning.
Interference from Galaxy 15’s stuck-on signal is the main concern, since the chance of it actually crashing into other satellites is remote to non-existent, Intelsat officials have said.
…
Several attempts to shut down Galaxy 15 have been unsuccessful, leaving the defunct satellite drifting in the cosmos.
“Normally when an anomaly occurs, the satellite just stops working and we don’t have to worry about it,” Good said. “Galaxy 15 is still operational, so in this case, the satellite is still “functioning” in a deterministic state. But, we know exactly where it is, we know what it’s doing, and we know the settings of the satellite.”
Oh, goody.
The future of zombiesat
Eventually, Galaxy 15 is expected to lose its Earth-pointing capability. Once this lock on Earth is lost and its solar panels are no longer pointed at the sun, the satellite’s battery power will eventually die.
“When the battery power decreases past a certain threshold, the payload will shut off,” Good explained. “It will no longer receive and transmit, and its batteries will continue to deplete.”
The satellite could also reach a threshold that causes its onboard computers to reset, said Good, but the possibility of this happening is still unknown.
“There is a possibility that the onboard computer could reset, but we don’t know what that probability is,” he said. “Still, there is a chance. It would almost be like a “control-alt-delete” on your computer. It would begin sending telemetry again. It would wake up and realize ‘What am I doing here?'”
Murphy’s law turned roguesat took down pretty much the entire forecast and warning infrastructure of the National Weather Service last night, and it seems nobody in the media has noticed. I certainly wouldn’t had I not been in the middle of it.
On the plus side, this didn’t happen in the middle of a tornado outbreak, so we can all count ourselves fortunate. But, this incident does underscore just how sensitive our space technology can be to solar flares.
Here’s an image of the likely solar culprit that made Galaxy 15 into a “zombie”:
Sounds like you need an early night tonight.
You have my understanding and condolences… You’d think the USAF would have taken a laser to the thing by now, given that it messed up THEIR service as well 😉
getting data, but in fact we weren’t, they said “we’ll look into it further” which is usually code for “we don’t know why”.
We’ll look into iT Further” is how I always took that line, especially when speaking it 😉
About an hour later, about 4:00 AM PST I got a call from a tech who said that he had noticed my trouble ticket, and told me very matter-of-factly that “you probably haven’t been given all of the information you need”.
Which usually translated to “OMG, we were so crazy chicken head lost we forgot to call you back with The Fix”
At least you have an alternate freq. now…
Thanks for the interesting coverage of the event.
So what we need is a restart button on the exterior of the sat so we can send the X-37B
Robotic Spy Plane along for a reboot? (LOL) http://www.wired.com/dangerroom/2010/12/gallery-inside-the-secret-space-plane-landing/
oops I hit the spam filter, again
(sorry mods)
[ I looked in SPAM and found nothing. -MOD ]
“But, this incident does underscore just how sensitive our space technology can be to solar flares.”
Not just our space technology.
http://www.space.com/scienceastronomy/mystery_monday_031027.html
The Great Storm: Solar Tempest of 1859 Revealed
I say we get the USAF to use a Laser to nuke the entire sat from space. It’s the only way to be sure…/aliens off
not to be paranoid but are they sure it wasn’t the Chinese and some killer satellite? i hear rumors that the rocket or whatever it was on the west coast several weeks ago may have actually been launched by the Chinese.
REPLY: All that is absolute garbage – Anthony
Where is Anthony (aka Buck) Rogers when he is needed?
Interesting post. Thanks.
This is why I love this site. Not only is there excellent banter between edgacated folks slamming the “climate disruption” community, real news. Like it really happened. Not with some editor playing “spin Dr.”.
OT: Britain Is Freezing to Death
http://www.express.co.uk/posts/view/215510/Britain-is-freezing-to-death
To stay warm, the elderly are using their free travel passes to ride buses all day long.
Meanwhile, thousands of UN delegates are whooping it up in toasty Cancun.
The juxtaposition is priceless.
try to get some sleep, anthony.
hope the news on the medical front is looking good. best wishes.
I’m amazed this wasn’t forecast. Given all the passive space junk that’s tracked by radar, how could people forget about this light bulb that won’t go off?
Someone’s gotta be embarrassed about the outage.
R. de Haan says:
December 6, 2010 at 5:59 pm
> So what we need is a restart button on the exterior of the sat so we can send the X-37B Robotic Spy Plane along for a reboot? (LOL)
Would be nice, but I’d be very surprised if the X-37B could climb that far out of the gravity well (and come back down).
Didn’t the U.S. and China just recently play a game of “look what we can do” by shooting down “rogue” satellites and claiming so publicly? Seems like we should put them to good use.
One would think that their would be some sort of heart beat protocol with the satellite that would be used to maintain control. For example, if the satellite computer does not reset a hardwire function in response to a message from the ground periodically then the hardware will trigger a reset of the computer. This is a standard, or was anyway, method of maintaining control of a computer over which the controller has only remote access.
I wonder if a similar system is installed in this satellite and if not why not?
Ric Werme says:
December 6, 2010 at 6:39 pm
Would be nice, but I’d be very surprised if the X-37B could climb that far out of the gravity well (and come back down).
Rick, you’re right of course:
The first X-37B, OTV-1, was launched on 22 April, 2010 into an orbit probably around 430-450 km in altitude and 28 and 40 degrees in inclination.3 The Center upper stage which placed it into orbit performed a fuel depletion burn which placed it in orbit around the Sun
http://www.secureworldfoundation.org/images/X-37BOTVFactsheet.pdf
Altitude Galaxy 15 Sat: about 22369miles (36000 kilometers)
Back to the drawing board me thinks.
Ah, the joys of modern technology and customer service (or lack thereof!).
I’m glad the problem is resolved tho. Also, VERY much hate to think of what could happen if we had a major flair such as the one in the mid or late 1800’s. That could get really nasty, really quickly.
Hope you get a good night’s sleep tonight, Anthony!
Ah, yep, Dave Springer says: December 6, 2010 at 6:11 pm pegged the coronal mass ejection event I was thinking of (didn’t see his post before I’d submitted mine). I don’t recall ever seeing that there’s a magnetic pole direction issue with these that had an effect, however (it was mentioned in the Dave’s linked article).
If I recall correctly, ejections/storms of that magnitude are expected roughly once every 100 years…. which means we’re more likely to see one sooner rather than later. Perhaps our ‘quiet sun’ and it’s Maunder Minimum like activity will spare us for awhile?
I gather that a 1859 sort of a solar storm today would take us months if not years to recover from. And, because our supplies such as groceries, gas, etc., are turned over in a matter of a few days, things could get rather nasty for a bit if something like this occurred.
Of course, we’d also be in a sticky wicket if anyone decided to lob a single high altitude nuke burst above the nation – the EMP from that would fry electronics, wires, distribution cables, transformers, etc. and we’d be in a nasty position. I gather that for a few hundred million we could harden much of our electrical power transmission grid and other key infrastructure – but so far the politics aren’t such that it gets any real attention. I’d sure vote for not only a vast reduction in “green” and “AGW” related spending, putting those funds to reducing the deficit, but also using the small part of it necessary to harden infrastructure against EMP/solar storms.
“…It almost sounds like an idea for a rouge nation to use to interfere with satellite communications.”
Rouge nation? That would be France. Or San Francisco.
Didn’t the U.S. and China just recently play a game of “look what we can do” by shooting down “rogue” satellites and claiming so publicly? Seems like we should put them to good use.
Popping off satellites willy-nilly is not a “good use”. The debris up there is bad enough already. That’s why the Chinese destruction of one of their old satellites was considered to be extremely poor form.
What is needed is a rocket to gently push the offending satellite out of orbit, or to get close to it and fry it. But that would be very expensive.
re post by R. de Haan says: December 6, 2010 at 5:59 pm
R. de Haan, they’ll have to create a super secret tiny robot – I envision the red ‘Rock’em Sock’em’ Robot – to send up and hit that reset button for problem satellites, or destroy them with their new and improved modern day super powers!! {VBG}
(anyone not familiar with the old rock’em sock’em robots, just google ’em)
[ I looked in SPAM and found nothing. -MOD ]
I called solar flare and blamed a man who tried to marry the planet.
Sol got jealous.
(thanks for looking, though)
[d]
OT, bit Tips and Notes appears closed
Al Gore in major downsizing of operations, but not disclosing details about the money….the 300 million
http://www.politico.com/news/stories/1210/45985.html
To get out there you’d need a goodly booster, like a Delta IV or better. and not too heavy. No ASATS go that high, at least right now. Let alone the X-37 (though I think the
X-37 is some program!) .
December 6, 2010 at 6:26 pm…Still off topic…Dickens…not that “toasty” yesterday if you’re in US… Sunday max +24C…min +11C….Heading for an unprecedented morning
snowfall…Still of course only weather…No wonder the delegates, martini marxists or not
stay in the water still at 25-27C [Weatheronline=temp source] On topic, 3 screens, only
2 mouses, touchscreen?? Off topic again: Sweden’s biggest and most effective ice-breaker, Oden is in the Antarctic…to make some bucks 1.5 millions or so meanwhile the Gulf of Bothnia is freezing over…The unrealists have completely taken
over…??!! [N.B. “Oden” is named after the old Nordic god “Oden”, and she that is the
ice-breaker, since Oden was a male god, has an unusual bow, flat, and with many pumps that spray seawater over the broken ice…{for ice-breakers, slippery roads are better…!}]…Even last winter Oden was in the Antarctic, and then some Sweden-Finland big ferries were stuck in the ice, but no worries at “Sjöfartsverket”…Yesterday
a spokesperson for that ministry “reassured” us that “Oden” can be called “home” when needed…From McMurdo Sound to the Baltic in a jiffy…March speed 13 knots,
9600 NM roughly…no cyclones in the Indian ocean hopefully, no ferocious giant sharks outside Sharm-el-Sheik, no new little war in the Mid East, etc, no hurricane strength winds over the Bay of Biscaya…but wait, for 100000 quid we can help some
guys in the port of London… By then the ice wall has rolled in and Hell frozen over…
Writing these last lines, my internet connection was broken for the first time in many months… -SUN, please take it easy… I write ABOUT Oden aka Odin, I do NOT worship
him…
Too bad the satellite transmitters are not spread spectrum. Same channels different spreading codes. Probably need a good computer and phase control… It will take years to develop, lots of money. If satellites end up with web browsers and prepaid minutes plans Im going to scream conspiracy.
How’s this for a conspiracy theory…
“…it all started last spring with one telecommunications satellite that turned naughty last April:
and:
“The robotic space shuttle secretly left Earth seven months ago.”
http://pinoytutorial.com/techtorial/x-37b-space-shuttle-secret-mission-new-discoveries/
Douglas DC says:…..
Virgin Galactic has an active orbital transport, easy to attach a missile or 10 to it.
(just saying)
[d]
http://www.telegraph.co.uk/earth/environment/climatechange/8178162/Are-we-freezing-because-of-global-warming.html
I was going to put this in tips and notes, but it is not accepting comments? What happened?
‘Programming error’ caused Russian rocket failure
Has there been a noticeable increase in satellite failures, for various reasons, lately? Or is this to be expected? Given the costs and effort involved, I’d expect better quality control with more backup and damage-preventative measures.
In this case, the satellites were for Russia’s shiny new Glonass navigation system, a rival to the US GPS system. And GPS can be used for missile guidance as well as its normal military applications, and I would expect Glonass to be used the same way… Thus I will freely presume the Russian military, and likely their intelligence service(s), are involved with setting up and running this new system. Which still had this dramatic deployment failure.
BTW, are these the same Russian space experts we’ll be depending on to get to the International Space Station once the US shuttle fleet gets retired? ]:-)
Eggs? Baskets?
Expect it is hard to underestimate the breath & depth of impact anything like a repetition of the Great Solar Storm of 1859 would have on today’s wired-to-the-max ”just in time” world. Except for EMP-hardened MIL-Spec stuff, pretty much everything with even basic modern electronics in it let alone computers would likely go Tango Uniform. . . . Like cars, for instance: 1859 repeat, nobody (or at least very few people (not many Model-As around anymore) ) drive. It would be ugly; and not just for weeks or even months. People in big cities could start to seriously envy those in rural areas with cows and gardens.
And as far as doing something effective to get Galaxy 15 to shut up:
Tough problem. As already noted in prior comment: Easiest thing to do at Geo-sync altitude (note ”easiest” in this case is RELATIVELY speaking) is blow it up; but that’s also the LAST thing you want to do; i.e.: Replacing 1 piece of space junk with 1000s if not more fragments is to be avoided.
Apparently the Sun now has an owner who intends to make money out it. See
http://www.heraldsun.com.au/lifestyle/the-other-side/spanish-woman-claims-ownership-of-sun-wants-to-tax-it/story-e6frfhk6-1225962735887
The property rights registration appears to be purely to make money out it by charging for solar use. With that comes liability: for the damage done by the property. You, NASA, the US Airforce and everyone else affected by the satellite’s down time could file a claim for the CME’s damage.
I was thinking of lodging a claim for all my skin cancers, but they pre-date the registration of ownership by several years, so I think I might be out of luck.
Anthony, thanks for the brilliantly informative bulletin and I hope you get a long, restful night’s sleep tonight.
Makes me regret selling my ’34 Dodge Sedan fifty years ago – almost bulletproof, fast enough for all practical purposes (9 to 90 mph in top gear), big hydraulic brakes (a Chrysler 1st), handled nicely on slippery gravel roads and would plod through deep mud, reasonably light and so wonderfully simple not much ever went wrong with it. My current car, a very good Japanese two litre thingy, is nice, civilised, warm and comfortable among all the ice and snow, but if anything goes awry I would need a mainframe computer to sort it. And it has nil character.
Some very heavy government scientific dudes have done some intense study on just this type of occurrence.
At least one ex military physicist – (a retired Rear Admiral) has released a paper on this subject – it’s out on the net.
Very sobering reading.
Apparently the problem was worse than just changing frequencies without telling anybody. Globecomm apparently tried to also boost the power on the SES World Skies sat to override the interference, causing receivers to tank out due to signal swamping.
Reminds me of the SCTV “CCCP1” episodes where a Soviet satellite kept breaking into the SCTV feed, leading to much hilarity. Though “hilarity” was likely far from your mind at 4 am.
What does one use when the satellite goes down? Divining rods? A weather oracle? A deranged cat? The Farmers Almanac?
😉
DL says:
December 6, 2010 at 7:01 pm
“One would think that their would be some sort of heart beat protocol with the satellite that would be used to maintain control.”
We called that a “dead man timer” and yes it is (or at least was) common practice in computer architecture.
http://en.wikipedia.org/wiki/Dead_man%27s_switch
I suppose we called it a timer instead of a switch in my field (IBM PC compatibles) because it was usually implemented by way of a timer output which was periodically reset by an interrupt service routine. If the ISR failed to reset the timer then when it timed out it raised a signal on an output pin that caused a hard reset to occur.
Not to scare any pantz off of anyone but if it weren’t for reality I’d say it was the Hippie Mega Mind at work calling all stations for ready take over.
But as it has it, reality probably trumps my fiction in this matter, I mean how can a mega mind stack up to the fact that Mannschtick and his minion panzer pantz Romm probably just made a slight schtickup in their trajectory calculation when they tried to go ballistic a ways back, there by hitting the poor hard working (for no pay) satellite square in the barren space like it wasn’t having a hard time already the little one.
There is discussion elsewhere (use Google News/Blog Search for “noaaport galaxy”) that the 12/5-12/6 outage was *not* Galaxy 15 related, but next week there could be. Any updates on this story, Anthony?
The Galaxy 15 thing is a known and relatively predictable issue, it has stepped on other satellites on its way around. Hopefully it turns turtle and dies soon.
Intelsat had gotten together with all the Satellite operators and made very extensive plans to avoid/minimise problems due to Zombie Sat. A great effort of cooperation by Intelsat, admitting it’s their problem and providing all the orbit information to other operators. The orbit is well defined and I cannot believe the NOAA people were clueless this was going to happen!
pRadio
#
#
jimgineer says:
December 6, 2010 at 9:25 pm
Too bad the satellite transmitters are not spread spectrum. Same channels different spreading codes. Probably need a good computer and phase control… It will take years to develop, lots of money. If satellites end up with web browsers and prepaid minutes plans Im going to scream conspiracy.
###
Spread Spectrum is used where appropriate, e.g. the command and control channel. For a number of reasons spread spectrum has not been used on data downlink. BTW, Spread Spectrum was being used in space hardware a decade before most people even heard of it.
Hmmm … malfunctioning ‘transponder’, not transmitter … how does this work?
Input (uplink) on about 4 GHz, heterodyne to 6 GHz, amplify (a lot, via TWT tubes at least a decade back) and that amplified output is fed to a downlink horn (maybe even the same dish used as the uplink).
Does no one else question what the Zombiesat is seeing on the uplink to cause the downlink to actually have ‘signal’ (after all, this is a transponder, not a self-generating signal transmitter in the usual sense)?
This aspect has not been adequately explained to date …
.
Seriously overblown I think.
Seems most of this ‘conjecture’ is based on a few series-wired street-light strings popping fuses in Hawaii …
.
@ jimgineer: Why would you need phase control for spread spectrum? Besides, even spread spectrum would not have solved this problem, only ultra wide band would have helped. For some strange reason the satellite community seems skeptical of UWB.
Having made three (OK they were tiny) satellites that are now space-junk, my sympathies with the Intelsat and Oribital folks. But I still dont understand why they did not implement a “dead man switch”! It is not even a “real” switch just software code and pretty damn simple to implement:
START
IF (No input telemetry data for 3 days)
RESTART (main())
END
sheesh! You’d have thought they’d have learned a thing or two by now!
@DesertYote: List several reasons.
@Amit: By phase control I was referring to low phase noise oscillators and ability to compensate for phase jitter in the receiver. I was referring to technologies used in CDMA and TDMA cellular phones.
In adition too watchdog timers, a simple uplink RF carrier detect relay could be interlocked with downlink transmit enable at a hardware level. No computers!!!!
Broadcasters often tie their studio transmitter links to their transmitters in this way to satisfy FCC requirements to be in control of the transmitter in case computerized telemetry fails.
_Jim (~jim):
The first two links say Galaxy 15 has been emptied of all its customers and is no longer broadcasting on those 24 transponder channels, EXCEPT that there was a discrete payload to serve FAA on L-Band and that remains on, listening and rebroadcasting. The prognosis for the sat to lose earth orientation was August 15, obviously wrong, so apparently it’s still listening to earth in the key of L and transponding in kind.
Update Dec. 30: Good news. The zombie satellite is back from the walking dead.
“A zombie satellite that spent months sending out signals while it was adrift in orbit has sprung back to life, resetting itself after its unexplained breakdown in space earlier this year. ”
http://www.msnbc.msn.com/id/40846679/ns/technology_and_science-space/
I have been led to believe that AlGore is inside it trying to change the codes