Wow. The vulnerability here is stunning. Yesterday there were all sorts of delays and timeouts on the NWS main page. The reason turned out to be a single overzealous app that was querying the page for data, and the CGI processes to serve up forecast data on a city by city basis got overwhelmed. Apparently the devloper didn’t know there are more efficient ways of getting the forecast data. The National Weather Service’s Telecommunication Operation Center posted this message on its status page yesterday afternoon:
NOXX01
KWBC 251835 TO – ALL CUSTOMERS SUBJECT – POINT FORECAST ISSUES WE ARE PROVIDING NOTICE TO ALL THAT NIDS HAS IDENTIFIED AN ABUSING ANDROID APP THAT IS IMPACTING FORECAST.WEATHER.GOV. WE HAVE FORCED ALL SITES TO ZONES WHILE WE WORK WITH THE DEVELOPER. AKAMAI IS BEING ENGAGED TO BLOCK THE APPLICATION. WE CONTINUE TO WORK ON THIS ISSUE AND APPRECIATE YOUR PATIENCE AS WE WORK TO RESOLVE THIS ISSUE.
NIDS – KM
Today, we have this new message:
Last Update: Tue Aug 26 14:50:01 2014 GMT
NWS TOC Operational Status Message
Tue Aug 26 02:42:58 2014 GMT
NOXX01 KWBC 260240 TO – ALL CUSTOMERS SUBJECT – POINT FORECAST ISSUES . AKAMAI HAS INSTALLED FILTERS WHICH BLOCK THE OFFENDING TRAFFIC. NIDS HAS VERIFIED THAT THE TRAFFIC IS BEING BLOCKED. ALL SYSTEM ARE NORMALIZED. WE APPRECIATE YOUR PATIENCE. PLEASE NOTIFY THE TOC AT TOC.NWSTG-AT-NOAA.GOV IF ANY FURTHER ISSUES ARE IDENTIFIED. THANKS FOR YOUR PATIENCE. NIDS – KM
I suppose a little bit of server hardening will be in order. This sort of thing wouldn’t look much different than a DDoS attack.
h/t to WUWT reader Paul H
The NWS web site system (and especially weather.gov) is a complete disgrace. It is slower than dirt, full of bugs, and if you scrape more than two layers deep you end up in a c. 2000 web site design.
Sounds to me like a perfect opportunity for any wackadoodle alarmists in the NWS to further “adjust” their data. Hope someone has a hard copy somewhere.
In fairness, until the system is attacked, why bother spending money upgrading it?
The US NWS may well have other things to spend money on.
Was the downtime that occurred so critical that preventing this would have been worthwhile?
which app is it?
M Courtney: why bother putting locks on your doors until after you’re burglerised?
I appreciate the site only place I can get even a close forecast to were I live. Radio and TV are worthless they only care about the big city nearby. In fact I was at the website directly before reading this article. No ads is a plus.
I wrote a Windows game years ago that messaged my central server, but didn’t bother waiting for the reply. After a few days the server would crash. Once I made it multi-threaded, and the thread waited politely for the server’s reply, while the game carried on, all was good.
I was pretty surprised you could crash the server that way (it was just some standard Apache webserver) so I wouldn’t be too hard on these NWS people!
“CGI processes” – seriously?
I won’t bore others with a treatise on “why”, but puhleeeeeez
Gerry Shuller “I won’t bore others with a treatise on “why”, but puhleeeeeez”
Yep, that’s what I say when I hope no one asks me for details and I’d rather just be snarky.
One reason Apache is popular is that it forks itself for each connection. After some cycles it dies and launches a fresh copy entirely. Its developers expect bugs and memory leaks and this is an excellent way to deal with that near-certainty. In the case of IIS, eventually you have to restart it by hand, or in severe cases, reboot the actual server (host).
As such it (Linux/apache) is highly reliable but not really designed for 10,000 simultaneous connections.
As to your pretended offense at CGI, I use it and love it. Fast, efficient, robust. But that’s because my program is written in “C” as a single executable. No Java, no session variables, no cookies, no dynamically linked libraries.
So, please bore me with a treatise on “why”.
My first job out of college was at NIST in Gaithersburg, MD. Having had to deal with federal IT services on a regular basis, this does not surprise me in the least.
I’m sure that’s exactly what the managers at NWS thought.
But how much time did they spend scrambling to fix the issue ASAP, instead of analyzing their site beforehand for vulnerabilities and systematically creating, testing and deploying a less vulnerable system? It shows these people feel secure in their jobs as managers in many business would have worked to prevent this type of problem rather than risk being fired for it.
Government (Federal) quality control.
Thanks, Anthony. This explains the difficulties using my favorite weather links during part of the day yesterday.
NOAA-NWS is the place to go for localized weather nowcasting and short-range forecasting. It is most important for my Florida weather pages.
I feel I have to defend your NWS here. They simply didn’t anticipate the vulnerability and they have been very open to any request.
Here in Europe we pay every NWS through high taxes and then we have to pay twice if we want to look at the data. I rather send my taxes to your NWS and ask them to do it worldwide, at least I could look at the results freely.
I don’t know what the NWS’s budget is these days, but I’m pretty sure that it’s dwarfed by its parent agency NOAA, and I’m not even sure what NOAA does. The NWS is certainly its most visible component. The rather small NWS budget installs and maintains a network of doppler radars, employs meteorologists and technicians in all 50 states, maintains a temperature (ASOS) network, maintains a COOP (temperature) network, buys and flies 2+ balloons every day for upper air data and maintains the equipment that tracks them, monitors lakes and rivers, provides flood forecasts, and during their spare time, the meteorologists take the time to develop various web pages that provide information for their local customers (those pages below the first level). It’s amazing that it works as well as it does given their budget.
Amen brother. I use NOAA weather exclusively and skip local newsbroadcast weather.
I have a website, funded by me, and one say it just siezed up solid. I managed eventually to get some RAM run a console and shut down the web server.
Then I went looking at the logs. Ok a couple of millions hits a day., That’s normal BUT what’s this. Fourfold increase in web bandwidth? That’s not…then I looked to see if it was DDOS. Nope. It was hundreds of sites that had never contacted my site before requesting data that they (obviously) didn’t have in their cache. They were all redirects from one site.
Reddit.com.
Some A***hole had mentioned my site as ‘kewl’ and loads of people had simply rushed in to see the ‘fashionable’ site and the traffic overloaded its very slender memory.
Yes, you can service a million hits a day with 384Mbyte of RAM as long as most of the hits are ‘yep, you have the latest graph’
You can’t service 30,000 unique sites an hour joining in to see the fun.
Since its vaguely climate related I’ve included a link to the site in the profile for this message.
Enjoy, but not all at once, please.
CGI? Really? That’s so 1970s — no wonder the site doesn’t scale worth a damn.
Et tu John? So what’s wrong with CGI? I use it and it works great. Code-reuse means that if a thousand people are connected, although it looks like you have a thousand Apache (httpd’s) really you have just one with a thousand stacks; which is of course pretty much the same as any webserver that tries to manage all connections in one instance of the executable in memory. You are still burning the same amount of memory either way, but keeping your server in memory runs the risk of memory leak (IIS 6 anyone?) and eventually you have to manually restart it (iisreset).
So far as I know this is true of CGI programs. Once in memory the code base is simply re-used, with a single instance of the code base (text) and a thousand heap/stacks with the linux host managing those heaps.
It is not clear to me where your scalability complaints kicks in, or why now suddenly you have an issue with it when for the past few years you probably didn’t.
I suppose with two people complaining about CGI (neither revealing the secret of what the letters mean — Common Gateway Interface) maybe I’ll ask the “oracle” why people are dissing CGI.
1970s? That was ARPAnet era. 1990s perhaps.
Two people complained about CGI. By itself that is nothing to complain about. What matters more is the nature of your application that is launched via the Common Gateway Interface. In my case it is a small, statically bound “C” program that is going to be about 50 times faster than an interpreted language such as PHP, Python, Perl.
Another issue that makes sense however is database connections. Mine creates a new database connection for each launch, but it is also taking advantage of database server security on top of application security, so the database connection knows who just connected and can differentiate permissions thereby.
The Wiki talks about “fastcgi” which deals with repeatedly made database connections. You can make it just once and then your application talks to the fastcgi server rather than the database server. I suppose that would work as a “shim” in the case that all users connect to the database with the same parameters anyway. Of course, what the database sees is a single connection coming from the fastcgi server so you lose rather a lot of auditing and security options.