I’m a bit of a zombie myself as I write this, as I was up until 4:30AM PST dealing with this problem which turned out to affect National Weather Service offices nationwide in addition to my business, which uses the same satellite data feed. It is a rather interesting story of technology gone “rogue” and it all started last spring with one telecommunications satellite that turned naughty last April:
Intelsat Loses Contact with Galaxy 15 Satellite | SpaceNews.com
It seems a solar storm was to blame:
FoxNews.com – Intense Solar Storm Spins Satellite Out of Control
(more on this later at the end of the article)
They can’t control it any more, but the transmitter remains on:
SPACE.com — Attempt to Shut Down Zombie Satellite Galaxy 15 Fails
Which brings us to last Friday, while I was out of the office, my staff was dealing with intermittent signal problems with our satellite data feed. The NWS Wallops Island uplink boosted the signal strength on the transponder, and the problem was solved…or so they thought.
Yesterday, all hell broke loose and we got this ugly message:
*Topic: *Family of Service Data Outage* * *Date/Time**:*December 05, 2010 2110 UTC* * *Product(s) or Data Impacted:*Single Field of View East and West*,*GOES LCB Ash,Blended TPW for GPS _NOAAPort and GPS_NOAAPort_Dist1, GOES East and West ASOS Imager and DPI WEST, ** *Date/Time of Initial Impact:*December 05, 2010 1900 UTC** *Date/Time of Expected End:*Unknown** *Length of Outage:*TBD* * *Details/Specifics of Change:*The Family of Service at NSOF is down and has affected the above products to be unavailable until further notice. Tech. control in Silver Spring reports, the problem is due to the NOAAPORT Satellite at NCF AWIPS having poor signal quality . The problem is being investigated.* *
That AWIPS acronym Advanced Weather Interactive Processing System (AWIPS) is for the internal computer system that the National Weather Service uses.
And they were down too. This is significant, since the AWIPS system is used to issue severe weather bulletins (via WarnGen) at every NWS WSFO in the country now, and they all share this common datafeed from the satellite, which the private sector can also tap into via a program known as NOAAPORT.
Supposedly, the satellite data problem was solved last night at 6:59 UTC, but we were still down two hours later. I put in a call to the NWS Telecommunications people and asked for help. They seemed clueless, and wanted to know “What Air Force base are you with?” when I mentioned NOAAPORT, giving me my first clue that the problem was even more widespread. There are a number of Met support offices at AFB’s that use NOAAPORT also. I left a number, and about 30 minutes later got a call back. They said we should be getting data, but in fact we weren’t, they said “we’ll look into it further” which is usually code for “we don’t know why”.
About an hour later, about 4:00 AM PST I got a call from a tech who said that he had noticed my trouble ticket, and told me very matter-of-factly that “you probably haven’t been given all of the information you need”.
He then proceeded to tell me that to solve the problem of the data outage (and at that time, I had no idea what was causing the problem) they had to switch transponder frequencies, and that I had to set my receiver to a new frequency. Did that, and it came right up. Whew!
My complaint about such info not being made public on NOAA’s status boards was met with “Sorry, that’s not my division”. I thanked him and hung up.
Later this morning, the real reason became clear with this message:
NOAAPORT SATELLITE INTERFERENCE
Effective: 12/4/10 – The Noaaport satellite transmission is subject to interferrence from this date forward thru 12/18/10. The below notice is the NWS’ official notice to-date regarding this interferrence.
Complete loss of Novra receiver satellite lock is being experienced due to the interference from the Galaxy15 satellite.
The NWS is now simulcasting the noaaport broadcast on an alternate frequency of 1138.5Mhz. Use your S75 receiver console software to enter in the alternate frequency if your receiver is not staying locked on the original frequency of 1193.5 More info here
NOXX10 KWBC 301457
DATA MGT MESSAGE 11-10.05
TO NOAAPORT USERS
FROM RTH WASHINGTON DATA MANAGEMENT
SUBJECT SBN/NOAAPORT POTENTIAL SATELLITE INTERFERENCE – NOTICE 1
EFFECTIVE DATE 12/12/10 TO 12/18/10 /FOR A PERIOD OF 2 TO 10 HOUR
A RISK HAS BEEN IDENTIFIED WHEN THE GALAXY 15 ROGUE SATELLITE TRAVERSES THE SES-1 ORBITAL POSITION WHICH MAY RESULT IN A POSSIBLE LOSS OF DATA VIA THE NWS AWIPS SBN/NOAAPORT FEED. THIS POTENTIAL INTERFERENCE MAY TAKE PLACE BETWEEN 12/12/10 AND 12/18/10 INCLUSIVE.
ACCORDING TO OUR SATELLITE CONTRACTOR. THE DURATION OF THE INTERFERENCE EVENT COULD BE ANYWHERE FROM 2 HOURS TO 10 HOURS. WE ARE CONTINUING TO RESEARCH AND ANALYZE THE FULL IMPACT AND RISK TO THE AWIPS SBN / NOAAPORT PROGRAMS.
More here from NOAA’s Network Control Facility (NCF)
Yep, the “rogue” Galaxy 15 satellite was spewing C-band microwaves as it drifts in space, interfering with critical downlinks used by NOAA and the U. S. Air Force. It almost sounds like an idea for a rouge nation to use to interfere with satellite communications.
Here’s what a recent story from space.com had to say:
The 4,171-pound (1,892-kg) Galaxy 15 satellite went rogue on April 5, when it stopped responding to controllers on the ground.
Yet, while the satellite veered from its assigned orbital slot of 133 degrees west longitude, 36,000 kilometers over the equator, the “zombie satellite” maintained an active payload, with its C-band telecommunications still functioning.
Interference from Galaxy 15’s stuck-on signal is the main concern, since the chance of it actually crashing into other satellites is remote to non-existent, Intelsat officials have said.
…
Several attempts to shut down Galaxy 15 have been unsuccessful, leaving the defunct satellite drifting in the cosmos.
“Normally when an anomaly occurs, the satellite just stops working and we don’t have to worry about it,” Good said. “Galaxy 15 is still operational, so in this case, the satellite is still “functioning” in a deterministic state. But, we know exactly where it is, we know what it’s doing, and we know the settings of the satellite.”
Oh, goody.
The future of zombiesat
Eventually, Galaxy 15 is expected to lose its Earth-pointing capability. Once this lock on Earth is lost and its solar panels are no longer pointed at the sun, the satellite’s battery power will eventually die.
“When the battery power decreases past a certain threshold, the payload will shut off,” Good explained. “It will no longer receive and transmit, and its batteries will continue to deplete.”
The satellite could also reach a threshold that causes its onboard computers to reset, said Good, but the possibility of this happening is still unknown.
“There is a possibility that the onboard computer could reset, but we don’t know what that probability is,” he said. “Still, there is a chance. It would almost be like a “control-alt-delete” on your computer. It would begin sending telemetry again. It would wake up and realize ‘What am I doing here?'”
Murphy’s law turned roguesat took down pretty much the entire forecast and warning infrastructure of the National Weather Service last night, and it seems nobody in the media has noticed. I certainly wouldn’t had I not been in the middle of it.
On the plus side, this didn’t happen in the middle of a tornado outbreak, so we can all count ourselves fortunate. But, this incident does underscore just how sensitive our space technology can be to solar flares.
Here’s an image of the likely solar culprit that made Galaxy 15 into a “zombie”:
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
Sounds like you need an early night tonight.
You have my understanding and condolences… You’d think the USAF would have taken a laser to the thing by now, given that it messed up THEIR service as well 😉
getting data, but in fact we weren’t, they said “we’ll look into it further” which is usually code for “we don’t know why”.
We’ll look into iT Further” is how I always took that line, especially when speaking it 😉
About an hour later, about 4:00 AM PST I got a call from a tech who said that he had noticed my trouble ticket, and told me very matter-of-factly that “you probably haven’t been given all of the information you need”.
Which usually translated to “OMG, we were so crazy chicken head lost we forgot to call you back with The Fix”
At least you have an alternate freq. now…
Thanks for the interesting coverage of the event.
So what we need is a restart button on the exterior of the sat so we can send the X-37B
Robotic Spy Plane along for a reboot? (LOL) http://www.wired.com/dangerroom/2010/12/gallery-inside-the-secret-space-plane-landing/
oops I hit the spam filter, again
(sorry mods)
[ I looked in SPAM and found nothing. -MOD ]
“But, this incident does underscore just how sensitive our space technology can be to solar flares.”
Not just our space technology.
http://www.space.com/scienceastronomy/mystery_monday_031027.html
The Great Storm: Solar Tempest of 1859 Revealed
I say we get the USAF to use a Laser to nuke the entire sat from space. It’s the only way to be sure…/aliens off
not to be paranoid but are they sure it wasn’t the Chinese and some killer satellite? i hear rumors that the rocket or whatever it was on the west coast several weeks ago may have actually been launched by the Chinese.
REPLY: All that is absolute garbage – Anthony
Where is Anthony (aka Buck) Rogers when he is needed?
Interesting post. Thanks.
This is why I love this site. Not only is there excellent banter between edgacated folks slamming the “climate disruption” community, real news. Like it really happened. Not with some editor playing “spin Dr.”.
OT: Britain Is Freezing to Death
http://www.express.co.uk/posts/view/215510/Britain-is-freezing-to-death
To stay warm, the elderly are using their free travel passes to ride buses all day long.
Meanwhile, thousands of UN delegates are whooping it up in toasty Cancun.
The juxtaposition is priceless.
try to get some sleep, anthony.
hope the news on the medical front is looking good. best wishes.
I’m amazed this wasn’t forecast. Given all the passive space junk that’s tracked by radar, how could people forget about this light bulb that won’t go off?
Someone’s gotta be embarrassed about the outage.
R. de Haan says:
December 6, 2010 at 5:59 pm
> So what we need is a restart button on the exterior of the sat so we can send the X-37B Robotic Spy Plane along for a reboot? (LOL)
Would be nice, but I’d be very surprised if the X-37B could climb that far out of the gravity well (and come back down).
Didn’t the U.S. and China just recently play a game of “look what we can do” by shooting down “rogue” satellites and claiming so publicly? Seems like we should put them to good use.
One would think that their would be some sort of heart beat protocol with the satellite that would be used to maintain control. For example, if the satellite computer does not reset a hardwire function in response to a message from the ground periodically then the hardware will trigger a reset of the computer. This is a standard, or was anyway, method of maintaining control of a computer over which the controller has only remote access.
I wonder if a similar system is installed in this satellite and if not why not?
Ric Werme says:
December 6, 2010 at 6:39 pm
Would be nice, but I’d be very surprised if the X-37B could climb that far out of the gravity well (and come back down).
Rick, you’re right of course:
The first X-37B, OTV-1, was launched on 22 April, 2010 into an orbit probably around 430-450 km in altitude and 28 and 40 degrees in inclination.3 The Center upper stage which placed it into orbit performed a fuel depletion burn which placed it in orbit around the Sun
http://www.secureworldfoundation.org/images/X-37BOTVFactsheet.pdf
Altitude Galaxy 15 Sat: about 22369miles (36000 kilometers)
Back to the drawing board me thinks.
Ah, the joys of modern technology and customer service (or lack thereof!).
I’m glad the problem is resolved tho. Also, VERY much hate to think of what could happen if we had a major flair such as the one in the mid or late 1800’s. That could get really nasty, really quickly.
Hope you get a good night’s sleep tonight, Anthony!
Ah, yep, Dave Springer says: December 6, 2010 at 6:11 pm pegged the coronal mass ejection event I was thinking of (didn’t see his post before I’d submitted mine). I don’t recall ever seeing that there’s a magnetic pole direction issue with these that had an effect, however (it was mentioned in the Dave’s linked article).
If I recall correctly, ejections/storms of that magnitude are expected roughly once every 100 years…. which means we’re more likely to see one sooner rather than later. Perhaps our ‘quiet sun’ and it’s Maunder Minimum like activity will spare us for awhile?
I gather that a 1859 sort of a solar storm today would take us months if not years to recover from. And, because our supplies such as groceries, gas, etc., are turned over in a matter of a few days, things could get rather nasty for a bit if something like this occurred.
Of course, we’d also be in a sticky wicket if anyone decided to lob a single high altitude nuke burst above the nation – the EMP from that would fry electronics, wires, distribution cables, transformers, etc. and we’d be in a nasty position. I gather that for a few hundred million we could harden much of our electrical power transmission grid and other key infrastructure – but so far the politics aren’t such that it gets any real attention. I’d sure vote for not only a vast reduction in “green” and “AGW” related spending, putting those funds to reducing the deficit, but also using the small part of it necessary to harden infrastructure against EMP/solar storms.
“…It almost sounds like an idea for a rouge nation to use to interfere with satellite communications.”
Rouge nation? That would be France. Or San Francisco.
Didn’t the U.S. and China just recently play a game of “look what we can do” by shooting down “rogue” satellites and claiming so publicly? Seems like we should put them to good use.
Popping off satellites willy-nilly is not a “good use”. The debris up there is bad enough already. That’s why the Chinese destruction of one of their old satellites was considered to be extremely poor form.
What is needed is a rocket to gently push the offending satellite out of orbit, or to get close to it and fry it. But that would be very expensive.
re post by R. de Haan says: December 6, 2010 at 5:59 pm
R. de Haan, they’ll have to create a super secret tiny robot – I envision the red ‘Rock’em Sock’em’ Robot – to send up and hit that reset button for problem satellites, or destroy them with their new and improved modern day super powers!! {VBG}
(anyone not familiar with the old rock’em sock’em robots, just google ’em)
[ I looked in SPAM and found nothing. -MOD ]
I called solar flare and blamed a man who tried to marry the planet.
Sol got jealous.
(thanks for looking, though)
[d]
OT, bit Tips and Notes appears closed
Al Gore in major downsizing of operations, but not disclosing details about the money….the 300 million
http://www.politico.com/news/stories/1210/45985.html
To get out there you’d need a goodly booster, like a Delta IV or better. and not too heavy. No ASATS go that high, at least right now. Let alone the X-37 (though I think the
X-37 is some program!) .
December 6, 2010 at 6:26 pm…Still off topic…Dickens…not that “toasty” yesterday if you’re in US… Sunday max +24C…min +11C….Heading for an unprecedented morning
snowfall…Still of course only weather…No wonder the delegates, martini marxists or not
stay in the water still at 25-27C [Weatheronline=temp source] On topic, 3 screens, only
2 mouses, touchscreen?? Off topic again: Sweden’s biggest and most effective ice-breaker, Oden is in the Antarctic…to make some bucks 1.5 millions or so meanwhile the Gulf of Bothnia is freezing over…The unrealists have completely taken
over…??!! [N.B. “Oden” is named after the old Nordic god “Oden”, and she that is the
ice-breaker, since Oden was a male god, has an unusual bow, flat, and with many pumps that spray seawater over the broken ice…{for ice-breakers, slippery roads are better…!}]…Even last winter Oden was in the Antarctic, and then some Sweden-Finland big ferries were stuck in the ice, but no worries at “Sjöfartsverket”…Yesterday
a spokesperson for that ministry “reassured” us that “Oden” can be called “home” when needed…From McMurdo Sound to the Baltic in a jiffy…March speed 13 knots,
9600 NM roughly…no cyclones in the Indian ocean hopefully, no ferocious giant sharks outside Sharm-el-Sheik, no new little war in the Mid East, etc, no hurricane strength winds over the Bay of Biscaya…but wait, for 100000 quid we can help some
guys in the port of London… By then the ice wall has rolled in and Hell frozen over…
Writing these last lines, my internet connection was broken for the first time in many months… -SUN, please take it easy… I write ABOUT Oden aka Odin, I do NOT worship
him…
Too bad the satellite transmitters are not spread spectrum. Same channels different spreading codes. Probably need a good computer and phase control… It will take years to develop, lots of money. If satellites end up with web browsers and prepaid minutes plans Im going to scream conspiracy.