Reposted from Cliff Mass Weather Blog
March 14, 2021
Broken Water Pipe Takes Out U.S. Buoy Data
I have gotten a number of emails from folks asking about the lack of U.S. buoy data during the past several days.
They are right…something has happened. A broken pipe and flooding has closed down a key NOAA data facility in Silver Spring, Maryland, and there is no estimate of when this data source will become available. And, in fact, other important data sources are also being affected, such as the coastal marine sites (CMAN) and even some airport locations.
The NOAA National Data Buoy Center page notes the situation:
Buoy data over the oceans is a very important weather data source, in support of both numerical weather prediction and marine activities. There are, in fact, two main types of ocean buoys: fixed or moored buoys, which possess a relatively wide range of weather/ocean observations, and drifting buoys that move around the ocean freely (see pictures below).
Fixed buoys:
Drifting buoy:
To give you an idea of the distribution and number of the buoy data, here is an example of the drifting buoy data locations immediately before the water leak (which occurred on March 9). Quite a few!
And the locations of fixed buoys (including coastal marine–i.e. lighthouses) are found here:
The loss of much of this data is a problem for numerical weather prediction, which uses oceanic data to provide information on surface conditions over the oceans. The plot of the buoys coming into the U.S. GFS global weather forecasting model shows the seriousness of the problem, with 3/4 of the buoy data gone.
And the coastal marine data has gone to zero.
Now the lack of marine data is not only an issue for weather prediction, but it can be an even larger problem for those involved in marine operations, who use weather data for the management of ports and sea lanes. Thus, the loss of surface marine observations put life and property at risk.
This outage is important because it shows the vulnerability of critical data infrastructure at NOAA, with dependencies on single servers and the lack of backup systems. There are others.
NOAA operations acutely need “hot” backups that can immediately take over when primary servers go down.
This is not only true for the ingestion of observations, but for the critically important weather modeling as well. Cloud-based backups are one way to do this at a reasonable cost. And there is always the old fashion way of maintaining independent servers in different locations.
Whatever approach NOAA takes, creating a more robust data and modeling infrastructure should be a high priority.
I am sure there are folks at Microsoft and Amazon who would be happy to help.😋
Oh oh… another broken pipe causes a change in the data… hmmmmm
Buoy oh buoy! My first thought, exactly!
Nothing worse than broken pipe dreams
They’re called bongs these days.
Broken water pipe, where have I heard that before?
Ruby Freeman got a promotion for being good at her job.
SDO pages have disappeared from the web since early this morning (UK time)
https://sdo.gsfc.nasa.gov/
I’m certainly no expert on this, but didn’t a broken water pipe take out a lot of election data somewhere too?
Well, not exactly…. that “water leak” in a malfunctioning toilet FAR from any ballots (or, perhaps, not…. flushing stuff down the toilet gets rid of it…. the Flushing Algorithm:
IF vote for Trump THEN flush) or voting tabulators was only a tissue-thin EXCUSE to stop counting ballots and send all the observers away.
As the data quantity and trend showed Trump would almost certainly win late that night…… uh, oh. Activate Operation S. O. B. (Save Ol’ Biden).
“Go away! We have to shut down. ……. Water leak!….yeah….that’s the ticket.
Just really intrigued….why would someone minus 1 that? I mean, when other, similar comments were either plussed (a lot, sometimes) or left alone.
Shrug. Looks like I have a personal enemy.
Heh. Thanks for the attention. I’m a force to be reckoned with. Bwah, ha ha ha haaaaaaaa!
Challenge to Minus One: come out of the shadows like an adult and state why you disagree.
Hey! The minus 1 *poof* disappeared…… (shrug)
Minus 1 is back again. Lol. Can’t you say something? Is minus-oneing all you can do?
Dear Janice, the minus/plus is cumulative on the majority side, and you probably earned a few minuses for being off topic, indiscreet and commenting on your own self. Those were mostly cancelled by those who can see your attempts at humour, even when you try explain your own jokes.
None of it is personal, relax… and look up “rhetorical question”, Len had one of those, and you answered him.
Thank you, P. Goy, for informing me about how the + – function works. It is a bit odd, nevertheless, ….. the + + and – – above always net to 1… odd….
I didn’t dislike it Janice, but maybe it was because the timing you mentioned was all wrong. The broken toilet was earlier in the day. The observers were sent home later that night.
…. and then they started “counting the votes” again, when no one was looking. The Perdue Ossoff Senate vote was the clincher for me. Banana Republic-style election.
Thank you for the correction, Phil. I should have said, w/regard to that particular gambit, that they stopped the vote counting.
And, because making mistakes is a little embarrassing, I am going to comfort myself by stating here that I did TRY to edit that comment, but, lol, I took so long to write (about 10 minutes (I had a lot of details : ) ), that when I tapped “Save” I was told “This comment can no longer be edited.”
To end on a bright note: wasn’t it SO COOL to look outside at 7:30PM this evening and it was STILL LIGHT!? 🙂 (well, around latitude 48N it was – was it still light then in the Bay Area? Well, light or dark, hope all is well with you.)
FYI for anyone interested, re: the “details” mentioned by me above, here’s an excellent source: https://navarroreport.com/#342e5f15-d44c-436a-bb58-9051a9397783
Move your clock ahead another hour and it will still be light at 8:30 PM. We’ve already lost High Noon; so there should be no further damage.
Thanks Janice, yes I’m alive and well and vaccinated. I forgot to look out of the window yesterday evening but hey ho 48 North has some good places on it too:
Other sites allow one to view who has voted up or down, by hovering over the ‘+’ or ‘-‘ symbols.
And yet, papyrus inscriptions, made by Egyptians over 3000 years ago, can still be read today. This shows how important it is to be careful about relying solely on electronics. Imagine if from today to tomorrow, all bank records were corrupted.
Back in the day, Novell networks were standard fare. There was standard training for Novell network engineers (not real PEs). The training specified that most of a Novell network engineer’s time would be spent administering back-ups.
Backups are dead easy. If you’re big enough, you can do it over your own WAN.
They will gladly help for a fee. NOAA should have the required manpower in house.
Most of NOAA’s servers run Linux so forget Microscoff. link
If the servers aren’t properly backed up (obviously local RAID doesn’t count) then someone needs to answer some questions. I hope the problem now is just to install some new infrastructure and boxen.
The company I work for is contractually obligated to maintain at least 98% uptime, with 100% guarantee of data integrity. We recently ran through an exercise where we switched from our primary server farm to a backup farm in just a few hours.
While maintaining backupsand the ability to switch over to it quickly isn’t quite rocket science, it is however quite doable and thousands of companies all over the planet manage to do it every day.
Meanwhile progressives continue to tell us that important things must be run by the government, because the private sector can’t be trusted.
Backups are easy? I’ve been selling backup systems for IT to companies large and small for 30 years and I would admonish anyone who thinks it is “easy” to think again. Failure rates in backup audits are well over 50%, true 30 years ago when I was getting started in the IT industry and true today. And that’s just for the companies willing to spend the money to find out! Among the companies too cheap to have outside expertise double check their internal expertise (who have a vested interest in giving themselves a passing grade) the failure rate is likely much higher. They get outside auditors to check their financial record keeping, they should apply the same discipline to their backup systems.
Yes lots of companies get it right. But they spend a ton of time and effort getting it right and they check on a regular basis to see if it is STILL right. A small change in a production system, or a storage array, or even a network change can leave things looking like they are working properly when they aren’t.
For the last half of my career I usually had some kind of sub net going. As a result I worked closely with the school’s IT department. As far as I can tell, data loss was a non-issue because those folks were doing their jobs properly.
Microsoft do Linux. With their Azure cloud services, probably more Linux than Windows.
Couldn’t they ask the NSA for help?
“……. forget Microscoff” Love it.
And Amazon would probably sell the data.
…which might be the most reliable way to NOT lose it! The more copies around, the better. The more PAID copies, even better still, since people who pay for something are more likely to keep it safe!
Hide the decline …
😉
Are they sure it was a broken pipe and not rising sea levels? 😜
.
.
.
That’s too bad. Anyone who has faced a broken water pipe and the subsequent mayhem can relate.
.
.
.
But from the article:
.
.
So there are dufuses in charge wasting taxpayer money on anything but securing the data, for what it’s worth, that taxpayers paid for.
.
.
.
.
.
Duh. Yah think, Vern? Who pays these clowns? Oh wait…
.
.
.
.
.
.
NOAA’s weather modeling isn’t half shabby. They do about as well as most. ‘Tis a shame, then.
Now if they had lost a fair number of climate models……. let’s just say I would have likely hidden the adjustable wrench if they came ’round looking for a way to shut off the water.
What about the Argo system of ocean robot buoys? Is that data processed at the same center, or is the Argo system data systematically ignored anyway by our politically motivated NOAA?
Just asking.
Well since you asked, it is well known in climate science that you just can’t beat ship’s buckets for accuracy.
Heh.
Plenty of room for adjustments
I’m hoping that there is some kind of active storage backup that is storing all of the data that is still being streamed to these centers. It would be criminal if this data was being lost.
It’s not the backup that counts, it’s the restore.
They are both important. Without backup, there is nothing to restore.
rbabcock is quoting an oft repeated phrase that’s been true in IT for decades. It is a sort of sarcastic remark meant to underscore that backups are, indeed, easy. Backups that can be RESTORED… not so much. When disaster strikes is NOT the time to test to see if you can actually completely restore your data. But that’s when the majority of holes in backup systems are discovered, unfortunately.
What, buoy data can’t float?
I’m really surprised that there would be a single point of failure in this distribution network.
The broken pipe can only be the result of runaway AGW, (or is it now ACC) (or maybe the dire and catastrophic D&CACC) as no other excuse will fit the needed script.
The increased risk of just this sort of unexpected level of flooding in our building is exactly the sort of thing we expect to see more of in the future unless we deal effectively with the climate crisis.
🙂 Heh.
Nope! Dealing with the “climate crisis” would mean sacking all the dweebs pretending to be doing climate science, and from their track record, none of them would be any good at serious, high-level work such as plumbing…
Ergo, solving the climate crisis will not prevent flooding in buildings!
The “dog ate my homework” excuse is too old-school for NOAA’s inconvenient data I’m guessing.
And when they do restore, are we about to see ‘the great reset’ of data ??
Has the broken pipe been attributed to Climate Change?
Just wondering.
Didn’t a fictitious broken water pipe provide the excuse to steal an election somewhere recently? I’d watch those numbers carefully when they re-emerge, as it were.
Part of the problem may be that most NOAA facilities are still at Phase 0 in COVID-19 response. There was only a skeleton crew at Silver Springs when the leak occurred. Damage was made worse by slow response and even cleanup and recovery are delayed while people are forced to work remotely.
NOAA and other Federal agencies are rather blase about returning to normal operations. You’d think we’re all a bunch of unionized teachers!
It’s not just NOAA, universities, including libraries, are denying various amounts of access including students, especially public. Don’t know how many, but a few had already been closed, more turned into computer labs presumably with the rise of online access. This event just helps prove the fallacy. Try these.
https://www.americanthinker.com/articles/2021/02/with_libraries_like_these_who_needs_to_burn_books.html
https://www.library.wisc.edu/gwslibrarian/bibliographies/disrupting-whiteness-in-libraries/
https://www.charlotteobserver.com/news/nation-world/national/article249798083.html
They apparently are too busy to worry about academics. Texas Eyes are now racist, according to some students. ‘“The report does not have a vindication or a smoking gun,” said panel chairman Richard Reddick, associate dean for equity, community engagement and outreach in the college of education.’
And it may not get fixed until after COP26.
The military must be pulling their hair out, having “lost” 3/4 of their floating submarine listening posts.
I’m pretty sure the military has a private feed of this data.
I blame the Earth shaking Biden Administration
When I was a young man, I installed undefloor water detectors for computer rooms that detected these problems and prevented this type of destruction.
They don’t use water detectors anymore?
If you have a leaky hot water line, that’s one thing. If it’s a broken 4″ sprinkler main, the alarm is as useful as the idiot lights on your car dashboard that just tell you which system is the source of that very bad sound.
At least the urinal didn’t stop the counting.
I’m going to call the 2020 elections the urinal counts.
Excuse one of my major pet peeves, but… Why do designers use pastel colors for graphs and web page alphanumerics? If your objective is to have people actually be able to read what you are presenting you should present it in a high contrast format. The titles and labeling on the graphs presented in this article are almost unreadable unless you squint and get a few inches form the screen! There! I feel better.
You’re better off complaining to the source. Folks here are generally just re-posting, not generating original material.
Where is the headline for “Raising water levels effect climate data”? The NYT missed it?