Broken Water Pipe Takes Out U.S. Buoy Data

Reposted from Cliff Mass Weather Blog

March 14, 2021

Broken Water Pipe Takes Out U.S. Buoy Data

I have gotten a number of emails from folks asking about the lack of U.S. buoy data during the past several days.  

They are right…something has happened.  A broken pipe and flooding has closed down a key NOAA data facility in Silver Spring, Maryland, and there is no estimate of when this data source will become available.  And, in fact, other important data sources are also being affected, such as the coastal marine sites (CMAN) and even some airport locations.

The NOAA National Data Buoy Center page notes the situation:

Buoy data over the oceans is a very important weather data source, in support of both numerical weather prediction and marine activities.  There are, in fact, two main types of ocean buoys:  fixed or moored buoys, which possess a relatively wide range of weather/ocean observations, and drifting buoys that move around the ocean freely (see pictures below).
Fixed buoys:

Drifting buoy:

To give you an idea of the distribution and number of the buoy data, here is an example of the drifting buoy data locations immediately before the water leak (which occurred on March 9).  Quite a few! 

And the locations of fixed buoys (including coastal marine–i.e. lighthouses) are found here:

The loss of much of this data is a problem for numerical weather prediction, which uses oceanic data to provide information on surface conditions over the oceans.   The plot of the buoys coming into the U.S. GFS global weather forecasting model shows the seriousness of the problem, with 3/4 of the buoy data gone.

And the coastal marine data has gone to zero.

Now the lack of marine data is not only an issue for weather prediction, but it can be an even larger problem for those involved in marine operations, who use weather data for the management of ports and sea lanes.   Thus, the loss of surface marine observations put life and property at risk.

This outage is important because it shows the vulnerability of critical data infrastructure at NOAA, with dependencies on single servers and the lack of backup systems.  There are others.    
NOAA operations acutely need “hot” backups that can immediately take over when primary servers go down.  
This is not only true for the ingestion of observations, but for the critically important weather modeling as well.  Cloud-based backups are one way to do this at a reasonable cost.    And there is always the old fashion way of maintaining independent servers in different locations.
Whatever approach NOAA takes, creating a more robust data and modeling infrastructure should be a high priority.   
I am sure there are folks at Microsoft and Amazon who would be happy to help.😋

4.9 11 votes
Article Rating
62 Comments
Oldest
Newest Most Voted
Inline Feedbacks
View all comments
mario lento
March 14, 2021 6:06 pm

Oh oh… another broken pipe causes a change in the data… hmmmmm

Michael S. Kelly
Reply to  mario lento
March 14, 2021 6:21 pm

Buoy oh buoy! My first thought, exactly!

Bryan A
Reply to  Michael S. Kelly
March 14, 2021 10:07 pm

Nothing worse than broken pipe dreams

Mike Dubrasich
Reply to  mario lento
March 14, 2021 7:36 pm

They’re called bongs these days.

Scissor
March 14, 2021 6:09 pm

Broken water pipe, where have I heard that before?

philincalifornia
Reply to  Scissor
March 14, 2021 7:39 pm

Ruby Freeman got a promotion for being good at her job.

Vuk
Reply to  Scissor
March 15, 2021 11:03 am

SDO pages have disappeared from the web since early this morning (UK time)
https://sdo.gsfc.nasa.gov/

Len Werner
March 14, 2021 6:27 pm

I’m certainly no expert on this, but didn’t a broken water pipe take out a lot of election data somewhere too?

Janice Moore
Reply to  Len Werner
March 14, 2021 6:52 pm

Well, not exactly…. that “water leak” in a malfunctioning toilet FAR from any ballots (or, perhaps, not…. flushing stuff down the toilet gets rid of it…. the Flushing Algorithm:
IF vote for Trump THEN flush) or voting tabulators was only a tissue-thin EXCUSE to stop counting ballots and send all the observers away.

As the data quantity and trend showed Trump would almost certainly win late that night…… uh, oh. Activate Operation S. O. B. (Save Ol’ Biden).

“Go away! We have to shut down. ……. Water leak!….yeah….that’s the ticket.

Janice Moore
Reply to  Janice Moore
March 14, 2021 7:15 pm

Just really intrigued….why would someone minus 1 that? I mean, when other, similar comments were either plussed (a lot, sometimes) or left alone.

Shrug. Looks like I have a personal enemy.

Heh. Thanks for the attention. I’m a force to be reckoned with. Bwah, ha ha ha haaaaaaaa!

Janice Moore
Reply to  Janice Moore
March 14, 2021 7:20 pm

Challenge to Minus One: come out of the shadows like an adult and state why you disagree.

Janice Moore
Reply to  Janice Moore
March 14, 2021 7:23 pm

Hey! The minus 1 *poof* disappeared…… (shrug)

Janice Moore
Reply to  Janice Moore
March 14, 2021 8:45 pm

Minus 1 is back again. Lol. Can’t you say something? Is minus-oneing all you can do?

Reply to  Janice Moore
March 14, 2021 9:40 pm

Dear Janice, the minus/plus is cumulative on the majority side, and you probably earned a few minuses for being off topic, indiscreet and commenting on your own self. Those were mostly cancelled by those who can see your attempts at humour, even when you try explain your own jokes.
None of it is personal, relax… and look up “rhetorical question”, Len had one of those, and you answered him.

Janice Moore
Reply to  paranoid goy
March 14, 2021 9:54 pm

Thank you, P. Goy, for informing me about how the + – function works. It is a bit odd, nevertheless, ….. the + + and – – above always net to 1… odd….

philincalifornia
Reply to  Janice Moore
March 14, 2021 7:37 pm

I didn’t dislike it Janice, but maybe it was because the timing you mentioned was all wrong. The broken toilet was earlier in the day. The observers were sent home later that night.

…. and then they started “counting the votes” again, when no one was looking. The Perdue Ossoff Senate vote was the clincher for me. Banana Republic-style election.

Janice Moore
Reply to  philincalifornia
March 14, 2021 8:18 pm

Thank you for the correction, Phil. I should have said, w/regard to that particular gambit, that they stopped the vote counting.

And, because making mistakes is a little embarrassing, I am going to comfort myself by stating here that I did TRY to edit that comment, but, lol, I took so long to write (about 10 minutes (I had a lot of details : ) ), that when I tapped “Save” I was told “This comment can no longer be edited.”

To end on a bright note: wasn’t it SO COOL to look outside at 7:30PM this evening and it was STILL LIGHT!? 🙂 (well, around latitude 48N it was – was it still light then in the Bay Area? Well, light or dark, hope all is well with you.)

Janice Moore
Reply to  Janice Moore
March 14, 2021 8:21 pm

FYI for anyone interested, re: the “details” mentioned by me above, here’s an excellent source: https://navarroreport.com/#342e5f15-d44c-436a-bb58-9051a9397783

stablesort
Reply to  Janice Moore
March 14, 2021 10:05 pm

Move your clock ahead another hour and it will still be light at 8:30 PM. We’ve already lost High Noon; so there should be no further damage.

philincalifornia
Reply to  Janice Moore
March 15, 2021 8:23 am

Thanks Janice, yes I’m alive and well and vaccinated. I forgot to look out of the window yesterday evening but hey ho 48 North has some good places on it too:

comment image

MarkW
Reply to  Janice Moore
March 14, 2021 8:08 pm

Other sites allow one to view who has voted up or down, by hovering over the ‘+’ or ‘-‘ symbols.

Last edited 27 days ago by MarkW
J N
March 14, 2021 6:28 pm

And yet, papyrus inscriptions, made by Egyptians over 3000 years ago, can still be read today. This shows how important it is to be careful about relying solely on electronics. Imagine if from today to tomorrow, all bank records were corrupted. 

commieBob
March 14, 2021 6:35 pm

Back in the day, Novell networks were standard fare. There was standard training for Novell network engineers (not real PEs). The training specified that most of a Novell network engineer’s time would be spent administering back-ups.

Backups are dead easy. If you’re big enough, you can do it over your own WAN.

I am sure there are folks at Microsoft and Amazon who would be happy to help.

They will gladly help for a fee. NOAA should have the required manpower in house.

Most of NOAA’s servers run Linux so forget Microscoff. link

If the servers aren’t properly backed up (obviously local RAID doesn’t count) then someone needs to answer some questions. I hope the problem now is just to install some new infrastructure and boxen.

MarkW
Reply to  commieBob
March 14, 2021 8:28 pm

The company I work for is contractually obligated to maintain at least 98% uptime, with 100% guarantee of data integrity. We recently ran through an exercise where we switched from our primary server farm to a backup farm in just a few hours.

While maintaining backupsand the ability to switch over to it quickly isn’t quite rocket science, it is however quite doable and thousands of companies all over the planet manage to do it every day.

Meanwhile progressives continue to tell us that important things must be run by the government, because the private sector can’t be trusted.

davidmhoffer
Reply to  commieBob
March 14, 2021 9:13 pm

Backups are easy? I’ve been selling backup systems for IT to companies large and small for 30 years and I would admonish anyone who thinks it is “easy” to think again. Failure rates in backup audits are well over 50%, true 30 years ago when I was getting started in the IT industry and true today. And that’s just for the companies willing to spend the money to find out! Among the companies too cheap to have outside expertise double check their internal expertise (who have a vested interest in giving themselves a passing grade) the failure rate is likely much higher. They get outside auditors to check their financial record keeping, they should apply the same discipline to their backup systems.

Yes lots of companies get it right. But they spend a ton of time and effort getting it right and they check on a regular basis to see if it is STILL right. A small change in a production system, or a storage array, or even a network change can leave things looking like they are working properly when they aren’t.

commieBob
Reply to  davidmhoffer
March 15, 2021 12:13 am

For the last half of my career I usually had some kind of sub net going. As a result I worked closely with the school’s IT department. As far as I can tell, data loss was a non-issue because those folks were doing their jobs properly.

Ryan
Reply to  commieBob
March 15, 2021 2:11 am

Microsoft do Linux. With their Azure cloud services, probably more Linux than Windows.

Rainer Bensch
Reply to  commieBob
March 15, 2021 6:53 am

Couldn’t they ask the NSA for help?

Oldseadog
Reply to  commieBob
March 15, 2021 11:30 am

“……. forget Microscoff” Love it.
And Amazon would probably sell the data.

AlexBerlin
Reply to  Oldseadog
March 18, 2021 10:49 am

…which might be the most reliable way to NOT lose it! The more copies around, the better. The more PAID copies, even better still, since people who pay for something are more likely to keep it safe!

Sweet Old Bob
March 14, 2021 6:49 pm

Hide the decline …

😉

H.R.
March 14, 2021 7:39 pm

Are they sure it was a broken pipe and not rising sea levels? 😜
.
.
.
That’s too bad. Anyone who has faced a broken water pipe and the subsequent mayhem can relate.
.
.
.
But from the article:

This outage is important because it shows the vulnerability of critical data infrastructure at NOAA, with dependencies on single servers and the lack of backup systems. There are others.

.
.
So there are dufuses in charge wasting taxpayer money on anything but securing the data, for what it’s worth, that taxpayers paid for.
.
.
.

NOAA operations acutely need “hot” backups that can immediately take over when primary servers go down. 

.
.

Duh. Yah think, Vern? Who pays these clowns? Oh wait…
.
.
.

This is not only true for the ingestion of observations, but for the critically important weather modeling as well. Cloud-based backups are one way to do this at a reasonable cost.  And there is always the old fashion way of maintaining independent servers in different locations.

.
.
.
NOAA’s weather modeling isn’t half shabby. They do about as well as most. ‘Tis a shame, then.

Now if they had lost a fair number of climate models……. let’s just say I would have likely hidden the adjustable wrench if they came ’round looking for a way to shut off the water.

Doug Ferguson
March 14, 2021 7:50 pm

What about the Argo system of ocean robot buoys? Is that data processed at the same center, or is the Argo system data systematically ignored anyway by our politically motivated NOAA?

Just asking.

philincalifornia
Reply to  Doug Ferguson
March 14, 2021 8:07 pm

Well since you asked, it is well known in climate science that you just can’t beat ship’s buckets for accuracy.

Janice Moore
Reply to  philincalifornia
March 14, 2021 9:04 pm

Heh.

Robert of Ottawa
Reply to  philincalifornia
March 15, 2021 3:47 am

Plenty of room for adjustments

MarkW
Reply to  Doug Ferguson
March 14, 2021 8:33 pm

I’m hoping that there is some kind of active storage backup that is storing all of the data that is still being streamed to these centers. It would be criminal if this data was being lost.

rbabcock
March 14, 2021 7:54 pm

It’s not the backup that counts, it’s the restore.

MarkW
Reply to  rbabcock
March 14, 2021 8:32 pm

They are both important. Without backup, there is nothing to restore.

davidmhoffer
Reply to  MarkW
March 14, 2021 9:43 pm

rbabcock is quoting an oft repeated phrase that’s been true in IT for decades. It is a sort of sarcastic remark meant to underscore that backups are, indeed, easy. Backups that can be RESTORED… not so much. When disaster strikes is NOT the time to test to see if you can actually completely restore your data. But that’s when the majority of holes in backup systems are discovered, unfortunately.

MarkW
March 14, 2021 8:00 pm

What, buoy data can’t float?

MarkW
March 14, 2021 8:05 pm

I’m really surprised that there would be a single point of failure in this distribution network.

Richard (the cynical one)
March 14, 2021 8:18 pm

The broken pipe can only be the result of runaway AGW, (or is it now ACC) (or maybe the dire and catastrophic D&CACC) as no other excuse will fit the needed script.

BobM
March 14, 2021 8:29 pm

The increased risk of just this sort of unexpected level of flooding in our building is exactly the sort of thing we expect to see more of in the future unless we deal effectively with the climate crisis.

Janice Moore
Reply to  BobM
March 14, 2021 9:05 pm

🙂 Heh.

Reply to  BobM
March 14, 2021 9:41 pm

Nope! Dealing with the “climate crisis” would mean sacking all the dweebs pretending to be doing climate science, and from their track record, none of them would be any good at serious, high-level work such as plumbing…
Ergo, solving the climate crisis will not prevent flooding in buildings!

Joel O'Bryan
March 14, 2021 9:55 pm

The “dog ate my homework” excuse is too old-school for NOAA’s inconvenient data I’m guessing.

Last edited 27 days ago by joelobryan
saveenergy
March 14, 2021 11:51 pm

And when they do restore, are we about to see ‘the great reset’ of data ??

Rod Evans
March 15, 2021 12:15 am

Has the broken pipe been attributed to Climate Change?
Just wondering.

Robert of Ottawa
March 15, 2021 3:39 am

Didn’t a fictitious broken water pipe provide the excuse to steal an election somewhere recently? I’d watch those numbers carefully when they re-emerge, as it were.

Mumbles McGuirck
March 15, 2021 6:41 am

Part of the problem may be that most NOAA facilities are still at Phase 0 in COVID-19 response. There was only a skeleton crew at Silver Springs when the leak occurred. Damage was made worse by slow response and even cleanup and recovery are delayed while people are forced to work remotely.
NOAA and other Federal agencies are rather blase about returning to normal operations. You’d think we’re all a bunch of unionized teachers!

H. D. Hoese
Reply to  Mumbles McGuirck
March 15, 2021 8:33 am

It’s not just NOAA, universities, including libraries, are denying various amounts of access including students, especially public. Don’t know how many, but a few had already been closed, more turned into computer labs presumably with the rise of online access. This event just helps prove the fallacy. Try these.

https://www.americanthinker.com/articles/2021/02/with_libraries_like_these_who_needs_to_burn_books.html    

 https://www.library.wisc.edu/gwslibrarian/bibliographies/disrupting-whiteness-in-libraries/

https://www.charlotteobserver.com/news/nation-world/national/article249798083.html
They apparently are too busy to worry about academics. Texas Eyes are now racist, according to some students. ‘“The report does not have a vindication or a smoking gun,” said panel chairman Richard Reddick, associate dean for equity, community engagement and outreach in the college of education.’ 

ResourceGuy
March 15, 2021 7:07 am

And it may not get fixed until after COP26.

DMacKenzie
March 15, 2021 7:08 am

The military must be pulling their hair out, having “lost” 3/4 of their floating submarine listening posts.

MarkW
Reply to  DMacKenzie
March 15, 2021 7:58 am

I’m pretty sure the military has a private feed of this data.

Neo
March 15, 2021 8:37 am

I blame the Earth shaking Biden Administration

Editor
March 15, 2021 3:00 pm

When I was a young man, I installed undefloor water detectors for computer rooms that detected these problems and prevented this type of destruction.

They don’t use water detectors anymore?

D. J. Hawkins
Reply to  Kip Hansen
March 18, 2021 3:05 pm

If you have a leaky hot water line, that’s one thing. If it’s a broken 4″ sprinkler main, the alarm is as useful as the idiot lights on your car dashboard that just tell you which system is the source of that very bad sound.

niceguy
March 15, 2021 4:18 pm

At least the urinal didn’t stop the counting.
I’m going to call the 2020 elections the urinal counts.

rhb2
March 16, 2021 10:18 am

Excuse one of my major pet peeves, but… Why do designers use pastel colors for graphs and web page alphanumerics? If your objective is to have people actually be able to read what you are presenting you should present it in a high contrast format. The titles and labeling on the graphs presented in this article are almost unreadable unless you squint and get a few inches form the screen! There! I feel better.

D. J. Hawkins
Reply to  rhb2
March 18, 2021 3:07 pm

You’re better off complaining to the source. Folks here are generally just re-posting, not generating original material.

George Ellis
March 19, 2021 4:59 am

Where is the headline for “Raising water levels effect climate data”? The NYT missed it?

%d bloggers like this: