What's going on? CRU takes down Briffa Tree Ring Data and more

My take on this would just be that they finally figured out it was THEIR web server that was the source of the leak (and having had data leak from there a couple of times in the past…).
In the case of a ‘break in’ you shut down for a bit, grab tape of every server you can, do a basic “was it hosed or buggered” inspection and put the ‘good servers’ that pass inspection back up. Then you start digging. (I think we saw all that happen when the leak first happened and there was a short outage.) If you are good, you put a bunch of added monitoring and tripwires in place to see if the perp returns and catch ’em. I doubt they did that part.
THEN:
When you figure out where the break-in or leak came from, you scrub that box to an incredible degree. There are all sorts of “clever” things that can be hidden in places you don’t normally see, even as root / sysadmin. You can put ‘tools’ in what are marked as ‘bad blocks’ on the disk. You can hide things in plain sight by swapping binaries out. Etc.
So, depending on how much you care (and at this point, I’d guess they care a lot…) you might go so far as to start from “bare metal” and re-install every single thing. NOT a ‘restore from backups’ that might carry ‘tools’ back in. But a real, vetted all the way, new install. With new OS copy with newest patches, virus software, etc. Otherwise it is a long time to find where some odd bit of binary software had it’s permissions set just a bit different and lets someone hack in.
(On systems I ran, we did an “ls -l” long format listing of systems files and kept it in a “special place”. Periodically it was re-run and compared. ANY changes were cause for investigation. ALMOST always it was a slightly less then perfect systems admin – sometimes me. Sometimes it was an ‘inside engineer customer’ being cute. We were ‘tight enough’ that it never was an outsider… though they tried every single day… BTW, we also had a ‘bare metal instal’ golden master tape. If a machine were compromised, we didn’t have to re-create the whole install process, we already had a Secure Golden Master of it and could, in fact, just do a restore from tape. These things are much easier if you thought them out in advance… )
So my guess is that they are just recreating every single thing on that server from ‘first sources’. It’s what I would do (at least, without a Secure Golden Master to work from.)
Wait, come to think of it, they have yet to do things the way “I would do”… so maybe not… 😎
FWIW, we actually had a paper printer attached to the main systems console of a couple of our key machines and printed the console log. Why? One of the ‘key tricks’ when you ‘get root’ is to erase your ‘footprints’ in the console log on the disk… can’t do that with paper inside the locked computer room… Caught a couple of folks that way.
Sometimes it’s the simple things 😉
At any rate, I’d give them a few days. If it isn’t done by Friday and back up, then my thesis is probably wrong and they are scrubbing for a different kind of cleaning.
BTW, the Russian announcement that the “stuff” came through their site, but was not them, and they had something to share if folks didn’t stop tossing eggs at them: An anon re-server still has logs. It is only anon because the admin chooses not to talk or scrubs the logs. So my guess is that they did not have the machine set up to ‘not log’, but had chosen the ‘not talk’ option… then the Secret Police showed up 8-{
So Mother Russia ought to have a log of where the traffic originated. And given that they did not just say “It came from Latvia”, it has blackmail value. My further guess from this, especially since much of the Russian Spy Breakin thesis came from English sources, is that their logs say “Came from UEA itself”.
So far, our leaker has done a stellar job. Were I doing this, I’d have “washed” the file through at least 3 places. But one of my favorite ‘distractors’ is to make a fake file of the right size, wash it one way (so things look to originate from A ) then wash it back (so it looks like a ‘from B to C’ ) but in fact, to swap in ‘the real deal’ on B. That lets you claim that the real source, B, must have been hacked too and gives plausible deniability while pointing to A as the ultimate source. One can only hope our real leaker was as careful. (Though they did wash the date stamps so they were clearly thinking…)
Given this, and given that UEA web server went down a day or so of negotiations after the Russian “announcement”, I would further speculate that someone from England talked to the Russians, said they would ‘quiet the finger pointing’ but could a little bit of log file be shared over vodka?…
And the log pointed back to the web server.
Which would then force a full forensic audit and rebuild. Especially if our leaker was careful enough to leave all ‘back pointers’ pointing internally at dead ends. (If you get “inside”, you want to make it look like all traffic originated from inside. But you want the actual box used to be hard to spot. Things like a laptop with wireless login to a lab box, then use the lab box for all the transit and washing, and remove the remote login from the lab box log files… so investigation shows “Phil’s Lab!!!” and not laptop from the parking lot outside Phil’s window… ) Devious? Why, thank you!
The cop has to be at least as devious as the perp to catch him.
(God I miss the chase some times … I can feel the brain lighting up and the slight elevation of adrenaline just remembering it… )