Subject: Administrative, GOES-15 Status Update, March 22, 17:45 UTC
—————————————–
Update: #6 GOES-15 (GOES-West) recovery is underway. The problem has been isolated; however, the estimated return to service will occur no earlier than 1600 UTC on Friday, March 23, 2012. In addition, a comprehensive list of products impacts is provided in this message. Lastly, we are currently experiencing some challenges that are impacting our group e-mail notification distribution. We will return to normal operations as soon as possible, and regret any inconvenience this outage may cause.
Update: #5 The GOES West data outage status has not changed. LRIT TEXT EMWIN DCS NWS are still red. GOES-15 recovery activities continue and all payloads and services remain off. The Engineering are estimating the recovery time to service for GOES-15 West on J/Day 083 at 1200Z. Engineering continues to run recovery procedures. GOES-13 East is operating in a Full Disk imaging mode while GOES-15 is out of service.
Update: #4 The GOES West data outage status has not changed. LRIT TEXT EMWIN DCS NWS are still red. GOES-15 recovery activities continue and all payloads and services remain off. The estimated return to service for GOES-15 West is undetermined at this time. Engineering continues to run recovery procedures. GOES-13 East is operating in a Full Disk imaging mode while GOES-15 is out of service
.
Update: #3 The GOES West data outage status has not changed. LRIT TEXT EMWIN DCS NWS are still red. GOES-15 recovery activities continue and all payloads and services remain off. The estimated return to service for GOES-15 West is undetermined at this time. Engineering continues to run recovery procedures. GOES-13 East is operating in a Full Disk imaging mode while GOES-15 is out of service.
Update: #2 The GOES West data outage status has not changed. LRIT TEXT EMWIN DCS NWS are still red. GOES-15 recovery activities continue and all payloads and services remain off. The estimated return to service for GOES-15 West is undetermined at this time. Engineering continues to run recovery procedures. GOES-13 East is operating in a Full Disk imaging mode while GOES-15 is out of service.
Update: #1 The GOES West data outage status has not changed. Engineers are currently
investigating this outage.
Topic: GOES-15 West data outage.
Date/Time Issued: March 22, 2012, 1705 UTC
Product(s) or Data Impacted: GOES-15 West Imager and Sounder data and associated products, ancillary communication services (DCS/LRIT/EMWIN/SAR)
Date/Time of Initial Impact: March 21, 2012, 2045 UTC
Date/Time of Expected End: March 23, 2012 1600 UTC (estimated)
Length of Outage: TBD
Update: #6 GOES-15 (GOES-West) recovery is underway. The problem has been isolated; however, the estimated return to service will occur no earlier than 1600 UTC on Friday, March 23, 2012.
Please note: GOES-13 East is operating in a Full Disk imaging mode while GOES-15 is out of service.
Here is a comprehensive list of products impacts.
GOES-15 Imager (Specific Images):
Super National Image (truncated in west)
NO West CONUS Sector Imagery
NO Alaska Regional and National Sector Imagery
NO Hawaii Regional and National Sector Imagery
GOES-15 Sounder Imagery: There is NO sounder data for any GOES-West regions (GOES-15 Sounder
Products including DPI, ASOS SCP, vertical profiles and GOES-15 Derived Products)
LRIT: Out on GOES-West; however, users can re-point their antennas to GOES-East (GOES-13) if they
can “see” the spacecraft. In addition, depending on vendor for LRIT station, some users can point
to MTSAT (Japanese satellite)
EMWIN: Out on GOES-West
DCS: Out on GOES-West; however, we have switched the feed for GOES West to GOES East. Those of users within the footprint of GOES East will most likely get the data. Those users outside the footprint of GOES East, or with an angle that doesn’t reach GOES East, will not.
SARSAT: With the INSAT satellite and GOES East we cover most of the area under GOES-W. There is a gap in coverage from 150W to 170E about 40 degrees where there is no GEOSAR detection at this time. This area will only be covered by LEO satellites which will likely increase latency of any alerts that might occur. New Zealand, Fiji, New Caledonia and several other smaller islands are in this region.
GINI Products: GOES-West was switched into a “WEST-LOOK-EAST” mode at 2105Z. This means that the West GINI started using GOES-East data for processing/creating products. There is a gap of data after 2030Z on 3/21/2012 until 2115Z. After that, every half hour using GOES-East data starting at 2115Z.
Super National Blended TPW Sector Imagery (truncated in west)
Low Cloud Base
GOES Sounder DPI
Cloud Top Pressure
Effective Cloud Amount
Lifted Index
Total Precipitable Water
Skin Temperature
Winds
ASOS SCP
Soundings
Contact Information for Further Information: ESPC Operations at ESPCOperations@noaa.gov at
301-817-3880
Web Site(s) for applicable information:
See http://www.ssd.noaa.gov/PS/SATS/messages.html for this and other satellite related messages.
See http://www.ospo.noaa.gov/Operations/GOES/schedules.html for full GOES scanning schedules and scan sectors.
Good thing it didn’t go down a couple days ago before the storm moved into the west coast.
Gah. What a jumble. A few minutes with cut-and-paste could order them for normal top-down reading.
Co-incidental that the largest late March storm system on record on the West coast isn’t documented?
Metaphorically.
But, then isn’t satellite itself metaphorical? Oh man, I think I just broke my brain.
Oh, I see. The area left uncovered is in the Pacific, on either side of the date line. “There is a gap in coverage from 150W to 170E about 40 degrees where there is no GEOSAR detection at this time.”
Fortunately, we look to have a spare parked in orbit: GOES 14, as shown here (did I post this already?):
.http://www.oso.noaa.gov/goesstatus/
and here:
http://en.wikipedia.org/wiki/Geostationary_Operational_Environmental_Satellite#History.2Fstatus_of_GOES_satellites
Thanks Anthony for the updates. Very useful info indeed!
???
GOES 13 in full disk mode provides ‘coverage’ of the West coast … no?
Here’s an image captured just a few minutes ago: http://i42.tinypic.com/x3fzmo.jpg
The extended coverage may be at the expense of increased coverage (imagery scans) of the areas normally assigned to GOES 13, however. Increased coverage such as in the case during Thunderstorm outbreaks; often more images (visual, IR, etc) of a specific area may be taken than is normally scheduled for inspection and evaluation by the NWS meteorologists …
.
Interesting Power Point on the GOES series of sats:
http://directreadout.noaa.gov/miami11/docs/3.3_Hampton_GOES_Overview_2011.pptx
Note this information gleaned from that ppt referencing the new manufacturer of this bird contrasted with previous GOES birds:
GOES I-M Series:
Built by Space Systems / Loral in Palo Alto, CA
Based on Loral 1300 bus series
Design Life: 5yrs
GOES N-P Series: [Note: P became GOES-15 when it became operational]
Built by Boeing Satellite Systems in El Segundo, CA
Based on Boeing 601 series
Design Life: 10yrs (bus)
GOES-13/14/15 have similar instruments to GOES-8-12, but on a different spacecraft bus.
– – – – – – – – – – –
Could be a problem specific to the Boeing “601 series” bus since all systems/imagers etc seem to have gone down, rather than just a specific function.
.
Jim said: “…Fortunately, we look to have a spare parked in orbit: GOES 14, as shown here…”
It’s not dead yet…it’s getting better…
ah-la Monty Python
The sort of imagery provided by GOES-15 (the current “GOES-West”) is at
http://www.goes.noaa.gov/
http://www.goes.noaa.gov/goesfull.html
It’s a bit odd that the pages don’t have a warning that the GOES West images are stale. The first one does note that “This server and data are not considered “operational”. We do not guarantee availability or timely delivery of data and this web site should not be used to support operational observation….” So, they are covered, but these are popular pages.
For those watching the live aurora data, the POES satellite is now back on stream after a software update.
http://tinyurl.com/2dg9u22/images/aurora.htm
11:50pm EDT update – anticipate normal on-orbit ops in about 4 hours.
Only for those interested, the ultimate GOES 15 (the satellite itself) resource (written by Boeing the contractor on this sat) –
– – – – – – – –
GOES N DataBook
26 April 2006 edition of this web page
The “GOES N Databook” is a 220-page paperback introduction to the technical aspects of the spacecraft and instruments. The Databook was assembled by the vendor, Boeing, which retains copyright to the material. First drafted in 2004, it was re-released as “Revison B” in February 2006.
To get a PDF digital copy of “Revison B”, click on the main link to the entire book (26 MB), or on the following links to files for each of the 20 sections of that book (100-1000 kB each).
– – – – – – – –
Individual sections or the whole enchilada can be found here:
http://goes.gsfc.nasa.gov/text/goes.databookn.html
Of note at this time (given difficulties):
8.0 Communication Subsystem
http://goes.gsfc.nasa.gov/text/GOES-N_Databook/section08.pdf
9.0 Telemetry and Command Subsystem
http://goes.gsfc.nasa.gov/text/GOES-N_Databook/section09.pdf
10.0 Electrical Power
http://goes.gsfc.nasa.gov/text/GOES-N_Databook/section10.pdf
.
According the CIMSS Satellite Blog at the U of Wisconsin-Madison:
After 20:30 UTC on 21 March 2012, GOES-15 (GOES West) experienced a bad momentum unload, and at this point the satellite went into a sun acquisition mode and stopped transmitting data.
http://cimss.ssec.wisc.edu/goes/blog/
Sounds like the operator made an error in loading some sort of station-keeping or positioning info and the satellite automatically attempted to establish a spatial reference and reorient itself. Commands and telemetry are handled through omnidirectional antennas allowing communication even when the spacecraft is incorrectly oriented. However, all of the sensors and high gain antennas are mispointed, thus making the payload unusable.
It is not uncommon after this type of thing (particularly if someone screwed up) to slowly bring the spacecraft back online. It’s too valuable an asset to risk a hasty return to service without understanding what caused the original malfunction. We built it at Boeing in El Segundo. It’s a good satellite, there’s no problem with the 601s.
Off topic but —- That EPA ad is crashing my computer and acting like a virus.
It keeps blanking my screen of all your information and just leaving the advert.
You should complain. It is F’n up my computer … more Google nonsense????
Following up, when I stated the payload was unusable I meant in a temporary state until the spacecraft is reoriented. I would guess things went something like this (speculation);
1) Received bad positioning command
2) Spacecraft exceeds some positioning threshold, looses spatial reference
3) Sensors and directional antennas mispointed, high speed data links lost
4) Payload shuts down automatically
5) Spacecraft automatically reestablishes spatial reference using the sun or stars
6) Spacecraft is reoriented to correct position via low speed data link
7) Payload is brought back on line
8) Normal service resumes
I suspect we’ve been stuck on #6 or #7 for a while until the engineers figured out what caused the problem in the first place.
Hmmm … the text actually reads “momentum unload” as opposed to “upload” …
From here; text message and brief context excerpted below:
That’s not to say that some uploaded command or command sequence didn’t precipitate that response though (or some other position/station keeping, or stabilization component burped).
Background: “momentum unload” in satellite parlance –
From 5th paragraph here.
.
Yes, I see Jeff C quoted ‘unload’ from his original cited source … too late and too difficult to keep all the uploads and unloads straight, on account of the late hour here (18 min past midnight in the central time zone) …
Thanks Jim, my mistake in reading it too fast. I’m a comm guy, not an attitude control expert, but what you describe sounds a bit more ominous, as if a normal maneuver went haywire.
I think my steps 2 to 8 are right, but it probably wasn’t an attitude command error.
It’s a pain in the arse when you are a TV met and trying to show what’s coming in off the Pacific… At least GOES East does give us some coverage (theoretically out to about 150 but that last 10-20 degrees are so warped as to be pretty useless.) Thank goodness we have those…otherwise we’d be stuck trying to show polar orbiters to the public. Now that would get ugly.
It’s not the first time this has happened… And and I doubt it will be the last. Sometimes you just gotta roll with the punches.
For those interested in this sort of thing and not already aware of it…..
Everything GOES:
http://www.oso.noaa.gov/goes/index.htm
Detailed GOES status:
http://www.oso.noaa.gov/goesstatus/
(Linked above already ?)
The problem I found yesterday, none of those links offers any technical details, the information seems to be days old, and at the first link you posted the ‘status’ link is 3 lines up from the bottom of the body of the text (status I would think should be a little more prominent) …
The nearest thing to be found to near real-time status seems to be this page, where new messages are queued to the top:
. . . http://www.ssd.noaa.gov/PS/SATS/messages.html
What’s worse, I went to the main NOAA page (last night) and there was nary a word nor a link to the the GOES-West (15) issue … everybody operationally concerned with weather and weather forecasting undoubtedly knew/knows that Goes-West (15) is out, but I would have thought that might be ‘front page’ news … although it being bad news; perhaps NOAA doesn’t want that prominently known?
.
This just in, from:
. . . . http://www.ssd.noaa.gov/PS/SATS/MESS/MSG0831428.02.txt
*Update: #11 *The GOES West services (LRIT, EMWIN,) have been restored.
We expect all instruments to be restored to normal operations no later then [sic] 1600z (4 PM EDT) with the exception of *SXI*, which will be restored as soon as possible thereafter.
Additional notifications will be sent when scheduling information is received.
– – – – – – – –
.
In prowling around publicly accessible datasets I came across this list of weekly scheduled activities that are performed at or on the BOEING 601 platform (GOES 13-15, not including GOES 12) satellites themselves.
From: http://noaasis.noaa.gov/cemscs/wklyscev.txt
– – – – – – – – – – – – – – – – – – – – – – – – – – –
WEEKLY SPACECRAFT EVENTS
20 MAR 2012 THRU 27 MAR 2012
…
GOES S/C DATE/TIME EVENT
GOES-9 – DECOMMISSIONED
GOES-10 – DECOMMISSIONED
GOES-11 – DECOMMISSIONED
GOES-12 – STANDBY SPACECRAFT.
G-12 DAILY//1215Z TRIM TAB ADJUSTMENT.
G-12 DAILY//1216Z SMC UPLOAD FOR BOOM SNAP.
G-12 DAILY//1235Z WHEEL UNLOAD IF NECESSARY.
G-12 DAILY//1835Z AOCE CLOCK RESET.
G-12 DAILY//1618Z SMC ENABLE FOR BOOM SNAP.
G-12 MON //1745Z SCC UPLOAD.
GOES-13
G-13 DAILY//0034Z IMC ACTIVATION.
G-13 DAILY//1331Z IMC UPLOAD.
G-13 DAILY//1534Z DAILY MOMENTUM UNLOAD.
G-13 DAILY//1540Z CLOCK ADJUST.
G-13 WED //1704Z SEM CALIBRATION.
G-13 DAILY//2031Z IMC UPLOAD.
GOES-14 – STORAGE MODE.
GOES-15
G-15 DAILY//0045Z IMC ACTIVATION.
G-15 THUR //1800Z SEM CALIBRATION.
G-15 DAILY//1931Z IMC UPLOAD.
G-15 DAILY//2045Z DAILY MOMENTUM UNLOAD.
G-15 DAILY//2151Z CLOCK ADJ.
G-15 DAILY//2231Z IMC UPLOAD.
G-15 20 MAR/2000Z YAW FLIP AT 2045Z-2115Z.
– – – – – – – – – – – –
Notes: GOES 13, 14 and 15 are the Boeing 601 series satellites. Bolding by _Jim
Given the time of the satellite outage, it indeed looks like the momentum unload or ‘desat’ (momentum desaturation) was the coincident event. Whether it was the causative event it is fairly well established that if there is anything which affects the satellite’s attitude it is the three-axis stabilization system using the electrically-powered reaction (mass) wheels (which are in constant rotation) also called ‘momentum’ wheels or the thrusters (Reaction Wheel Assemblies or “RWA”s on the Boeing 601 platform).
During ‘desat’ or ‘momentum unload’ excess momentum that builds up in the reaction wheels (due to external torques caused for example by solar wind or photon pressure or gravity gradients) is removed (daily in the case of the Boeing 601 system based satellites like GOES 13, 14 and 15) from the system by applying torque (via thrusters) to the spacecraft body/frame and allowing the wheels to acquire a desired increase or decrease inspeed, all (thrusters and reaction wheels) under computer control. This is done during maneuvers called momentum desaturation, (desat), or momentum unload maneuvers.
More on attitude control on the GOES 15 bird (Boeing 601 sat platform) can be found in:
11.0 Attitude Control
http://goes.gsfc.nasa.gov/text/GOES-N_Databook/section11.pdf
.
Only to get a clear answer for someone (me) , who doesn`t understand any thecnical question, a normal person without any knoledge of anything, Beyond cooking food for her family everyday: :
Fact one: On 21th march, GOES-15 autonomously went to safe mode. They doesn`t know why, ok.
2: Goes-15 it is the most important and significance, in analyze solar flares, The only one, we could say..
3: on 23th march, Goes 15 turn to be operative again? Ok, but 100 % operational? what about SXI and Xray… Operational Status ???
Please someone could answer me THIS question ? Anyone knows… ?
For a “normal” person it is impossible to get a clear answer about this. NOAA website does not make clear, noor easy. Thanks indeed,
Sofia.