Guest Post by David M. Hoffer
Since both ClimateGate 1&2 there has been considerable confusion in regard to how the emails were obtained, how the FOIA requests were managed, and what was or wasn’t possible in that context. There is no simple answer to those questions.
The ClimateGate emails span a period of nearly two decades. During that time period, email systems evolved substantially in terms of technology, implementation, operational procedures, and the job descriptions of those responsible for them. Other technologies such as backup systems, archive, and supporting technologies for legal compliance also changed (as did the laws themselves). Saying exactly what was and wasn’t possible for such simple actions as deleting an email have completely different answers over time, and also based on the technology that was implemented at any given time. With so many moving targets, it is impossible to draw any conclusions to a 100 percent certainty.
This article is written to cover the basics of how email systems and their supporting infrastructure work, and how they have evolved over time. With that as a common background, we can then discuss everything from the simple questions regarding who could delete what (and when), how the emails might have been obtained, and possibly most interesting of all, raise some serious questions about the manner in which the FOIA requests were handled at the CRU.
EMAIL 101
There are many, different email systems, and many different ways for end users to access them. The basics are common to all of them however. Each user has a “client” that allows them to access their email. It could be an internet browser based client such as the ones used by Hotmail and Gmail, or it could be an email client that runs on your desk top computer like Outlook or Eudora. For the purposes of this discussion I am going to discuss how things work from the perspective of an email client running on a desk top computer.
The email client connects an email server (or servers in a very large implementation). To send an email to someone on a different email server, the two servers must “talk” to each other. In most cases they do so over the internet. How the clients interact with the servers however, is part of understanding why deleting an email that you sent (or received) is not straight forward. The reason is that an email is never actually “sent” anywhere. Once you write an email it exists on the disk drive of the computer the client software is installed on. Press “send” and it goes….nowhere. It is still there, exactly as it was before you “sent” it.
A copy however, has now been sent to the email server you are connected to. That email server makes yet another copy and sends it to the email server the recipient is connected to. That email server then makes still one more copy and sends it to the email client on the recipient’s computer, which in turn writes it to the local hard drive. There are now a minimum of four copies of that one email.
But wait, there may be more copies. When researchers first started exchanging information via email, they were commonly years ahead of the rest of the world. Most large organizations had central IT shops, but they ran financial applications for the most part, email was a curiosity at best. Many researchers were left to run their own email systems, and it wasn’t that hard to do. Solaris was the UNIX operating system in vogue in those days, and Solaris came with a pretty good email system built in called Sendmail. There were many other options too. The bottom line was that early email systems were frequently run by researchers on their own computers.
As time went on, email became more common, and it became more important. The volume of data, performance matters, and security were all becoming beyond the skill set of anyone but someone whose full time job it was to run IT (Information Technology) systems. Researchers began giving up ownership of their own email system and central IT shops took over. Email was becoming mission critical, and a lot of data was being stored in email systems along with records of contract negotiations and other important “paper” trails. Losing email was becoming a painful matter if important information disappeared as a result. As a consequence, the systems that protected the data on email systems also began to mature and be run professionally by IT departments.
The early email systems were just a single server with local hard drives. As they grew in capacity and overall usage, plain old hard drives could no longer keep up. Storage arrays emerged which used many hard drives working together to increase both capacity and performance. Storage arrays also came with interesting features that could be leveraged to protect email systems from data loss. Two important ones were “snapshots” and “replication”.
Snapshots were simply point in time copies of the data. By taking a snapshot every hour or so on the storage array, the email administrator could recover from a crash by rolling back to the last available snapshot and restarting the system. Some storage arrays could handle keeping a few snapshots, others could maintain hundreds. But each snapshot was actually a full copy of the data! Not only could a storage array store many copies of the data, consider the question of deletion. If an email was received and then deleted after a snapshot, even by the central IT department itself, the email would still exist in the last snapshot of the data, not matter what procedure was used to delete it from the email system itself.
What if the storage array itself crashed? Since the storage arrays could replicate their data to other storage arrays, it wasn’t uncommon to have two arrays and two email servers in a computer room so that no matter what failed, the email system could keep on running. What if the whole computer room burned down? Replication to another storage array at a completely different location is also very common, and should the main data centre burn down, the remote data centre would take over. Keep in mind as you think this through that the ability of the storage arrays to replicate data in this fashion is completely and totally independent of the email system itself.
Early email systems were, as mentioned before, most often a single server with internal hard drives. A modern “enterprise class” email system would be comprised of many servers and storage arrays more like this:
If you recall that just sending an email makes, at minimum, four copies, consider what “one” copy on a large email system actually translates to. In the figure above, there are two copies on the storage arrays in the data centre. If snapshots are being used, there may be considerably more. Plus, there is at least one more copy being replicated to a remote data center, which also may have regular snapshots of data. That’s a LOT of copies of just one email! And we haven’t even started talking about backup and archive systems yet.
Let’s return to the question of deleting email. It should be plain to see that in terms of current email technology, deleting an email just from the email system itself is not a simple task if your intention is to erase every single copy that ever existed.
As an end user, Phil Jones is simply running an email client connected to an email server run by somebody else. He has no control over what happens on the server. When he deletes an email, it is deleted from his email client (and hence the hard drive on his computer), and from his view of his emails on the email server. Technically it is possible to set up the email server to also delete the email on the server at the same time, but that is almost never done, and we’ll see why when we start discussing backup, archive, and compliance.
On the other hand, are we talking about what was most likely to happen when Phil Jones deleted an email in 2009? Or what was most likely to happen when Phil Jones deleted an e-mail in 1996? The answers would most likely be entirely different. In terms of how email systems have been run in the last ten years or so however, while it is technically possible that Phil Jones hit delete and erased all possible copies of the email that he received, this would have done nothing to all the copies on the sender’s desk top and on the sender’s email server… and backup systems. Let’s jump now into an explanation of additional systems that coexist along with the email system, and make the possibility of simply deleting an email even more remote.
Backup Systems
Just as we started with email and how it worked at first and then evolved, let’s trace how backup systems worked and evolved. There are many different approaches to backup systems, but I’ll focus here on the most common, which is to make a copy of data to a tape cartridge.
At first, backup was for “operational” purposes only. The most common method of making a backup copy of data for a server (or servers) was to copy it to tape. The idea was that if a disk drive failed, or someone deleted something inadvertently, you could restore the data from the copy on tape. This had some inherent problems. Suppose you had a program that tracked your bank account balance. But for some reason you want to know what the bank account balance was a week ago, not what it is today. If the application didn’t retain that information, just updated the “current” balance as it went, you would have only one choice, which would be to restore the data as it existed on that specific day. To do that, you’d need one tape for each day (or perhaps one set of tapes for each day in a larger environment). That starts to be a lot of tape fast. Worse, as data started to grow, it was taking longer to back it up (and the applications had to be shut down during that period) and the amount of time at night where people didn’t need their applications running kept shrinking as companies became more global.
Several approaches emerged, and I will be covering only one. The most common by far is an approach called “weekly full, daily incremental”. The name pretty much describes it. Every weekend (when the backup window is longest), a full copy of the data is made to tape. During the week, only what changed that day is copied to tape. Since changes represent a tiny fraction of the total data, they could be run in a fraction of the time a full copy could. To restore to any given day, you would first restore the last “full copy” and then add each daily “incremental” on top until you got to the day you wanted.
This worked fine for many organizations, and larger ones bought “tape libraries” which were exactly what they sound like. They would have slots for dozens, sometimes hundreds, of tape cartridges, several tape drives, and a robot arm that could change tapes for both backup processes and for restore processes. The problem was that the tape library had to be as close as possible to the servers so that data could be copied as fast as possible (performance degrades sharply with distance). The following depicts the email system we’ve already looked at, plus a tape backup system:
By making regular copies of data to tape, which was a fraction of the cost of disk storage, the IT department could have copies of the data, exactly as it existed on any given day, and going as far back as the capacity of the tape library (or libraries) would allow. Now try deleting an email from say a year ago. In addition to all the copies on disk, there are at least 52 copies in the tape library. Since we have a tape library however, it is easy to make still more copies, automatically, and most organizations do.
Disaster Recovery
What if there was a major flood, or perhaps an earthquake that destroyed both our local and remote data centers? In order to protect themselves from disaster scenarios, most IT shops adopted an “off site” policy. Once the backup was complete, they would use the copy of the data on tape to make… another copy on tape. The second set of tapes would then be sent to an “off site” facility, preferably one as far away as practical from the data centers themselves.
Consider now how many copies of a given email now exist at any given time. Unlike that financial application whose “current account balance” is constantly changing, email, once received, should never change. (But it might which is a security discussion just as lengthy as this one!). Provided the email doesn’t change, there are many copies in many places, and no end user would have the security permissions to delete all of them. In fact, in a large IT shop, it would take several people in close cooperation to delete all the copies of a single email. Don’t organizations ever delete their old data?
Data Retention
The answer to that question can only be answered by knowing what the data retention policy of the organization is. Many organizations just kept everything until the cost of constantly expanding their storage systems, tape libraries and the cost of housing off site tapes started to become significant. Many organizations decided to retain only enough history on tape to cover themselves from a tax law perspective. If the retention policy was implemented correctly, any tapes older than a certain period of time would be removed from the tape library and discarded (or possibly re-used and overwritten). The copies in the offsite storage facility would also be retrieved to be either destroyed or re-used so that the offsite data and he onsite data matched.
Archive
As email systems grew, the backup practices described above became problematic. How long people wanted to keep their email for was often in conflict with the retention periods for financial purposes. They were designed for general purpose applications with ever changing data. As the amount of data in an email system started to grow exponentially due to ever larger attachments, graphics, and volume, the expense and pressure on even an “incremental” backup window became enormous. That’s where archive started to emerge as a strategy. The storage arrays that supported large email systems were very expensive because they had to be ultra reliable as well as ultra high performance. But 99% of all emails were being read on the day they were sent… and never again. Only if something made an older email important… evidence of who said what and when from a year ago for example, would an email be accessed again after it was a few days old. So why house it on the most expensive storage the organization owned? And why back it up and make a copy of it every week for years?
Many organizations moved to an “archive” which was simply a way of storing email on the cheapest storage available. If someone needed an email from a year ago, they would have to wait minutes or perhaps hours to get it back. Not a big issue provided it didn’t need to be done very often. Some organizations used low performance low cost disk, some even went so far as to write the archive to tape. So, for example, the email you sent and received in the last 90 days might open and close in seconds, but something from two years ago might take an hour. Not only did this reduce the cost of storing email data, but it had the added benefit of removing almost all the email from the email system and moving it to the archive. Since the archive typically wasn’t backed up at all, the only data the backup system had to deal with in its weekly full daily incremental rotation was the last 90 days. This left an email system, with the integrated backup and archive systems, looking something like this:
For most IT shops, if you ask them how many copies of a given email they have if it was sent a year ago, they can’t even answer the question. Lots.
What does that mean in terms of FOIA requests? Plenty.
Compliance
The world was rolling along quite nicely using these general techniques to protect data, and then the law got involved. Enron resulted in Sarbanes Oxley in the United States and similar laws in other countries FOIA came in existence in most western countries. Privacy laws cropped up. Suddenly IT had a new problem, and a big one. The board of directors was suddenly asking questions about data retention. The IT department went from not being able to get a meeting with the board of directors to having the board shining a spot light on them. Why?
Because they (the board of directors) could suddenly go to jail (and some did) because of what was in their email systems. Worse, they could even go to jail for something that was NOT in their email system. The laws in most jurisdictions took what you could delete, and what you could not delete, to a whole new level. Worse (if you were a member of the board of directors) you could be held responsible for something an employee deleted and shouldn’t have…. or didn’t delete and should have. Bingo. The board of directors is suddenly no longer interested in letting employees decide what they can and cannot delete, and when. The same applied in most cases to senior management of public institutions.
Back to the original question. Could Phil Jones have deleted his emails? When? In the early days when his emails were all in a server run by someone in his department? Probably. When the email system moved to central IT and they started backing it up regularly? No. He would only be able to permanently delete any given email provided that he had access to all the snapshot copies on all the storage arrays plus the archive plus all the backup tapes (offsite and onsite). Fat chance without the express cooperation of a lot of people in IT, and the job of those people, based on laws such as FOIA, SOX and others was to expressly prevent an end user such as Phil Jones from ever doing anything of the sort, because management had little interest in going to jail over something someone else deleted and shouldn’t have.
So…did CRU have a backup system? Did they send tapes off site? Did they have a data retention policy and what was it? Did they have an archive? If they had these things, when did they have them?
With all that in mind, now we can look at two other interesting issues:
- What are the possible ways the emails could have been obtained?
- Were the proper mechanisms to search those emails against FOIA requests followed?
Short answer: No.
In terms of how the emails could have been obtained, we’ve seen various comments from the investigation into ClimateGate 1 that they were most likely obtained from accessing an email archive. This suggests that there at least was an email archive. Without someone laying out a complete architecture drawing of the email systems, archive system, backup system, data retention policies and operational procedures, we can only guess at how the system was implemented, what options were available, and what options not. What we can conclude however is that at some point in time, an archive was implemented. Did it work like the description above about archives? Probably. But there are many different archive products on the market, and some IT shops refer to their backup tapes as an archive just to confuse matters more.
In addition, without knowing how the investigators came to the conclusion that the emails were obtained from the archive, we don’t have any way to assess the quality of their conclusions. I’m not accusing them of malfeasance, but the fact is without the data, we can’t determine if the conclusions are correct. Computer forensics is an “upside down” investigation in which the “evidence” invariably points to an innocent party. For example, if someone figured out what Phil Jones username and password was, and used them to download the entire archive, the “evidence” in the server logs would show that Phil Jones did the deed. It takes a skilled investigator to sort out what Phil Jones did (or didn’t do) from what someone using Phil Jones credentials did (or didn’t do). So let’s put aside what the investigators say they think happened and just take a look at some of the possibilities:
Email Administrator – anyone who had administration rights to the email system itself could have made copies of the entire email database going back as far as the oldest backup tapes retained with little effort. So…who had administration rights on the email system itself? There’s reason to believe that it was not any of the researchers, because it is clear from many of the emails themselves that they had no idea that things like archives and backup tapes existed.
Storage Administrator – In large IT shops, managing the large storage arrays that the application servers are attached to is often a job completely separate from application administration jobs such as running the email system. Since the storage administrator has direct access to the data on the storage arrays, copying the data from places such as the email system and the archive would be a matter of a few mouse clicks.
Backup Administrator – This again is often a separate job description in a large organization, but it might be rolled in with storage administration. The point being however, that whoever had backup administration rights had everything available to copy with a few mouse clicks. Even in a scenario where no archive existed, and copying the data required restoring it from backup tapes that went back 20 years, this would have been a snap for the backup administrator. Provided that the tapes were retained for that length of time of course, the backup administrator could simply have used the backup system itself, and the robotics in the tape library, to pull every tape there was with email data on it and copy the emails to a single tape. This is a technique called a “synthetic full” and could easily run late at night when it would just look like regular backup activity to the casual observer. The backup administrator could also “restore” data to any hard drive s/he had access too… like their personal computer on their desk.
Truck Driver – yes, you read that right, the truck driver. Google keywords like “backup tapes stolen truck” and see what you get. The results are eye popping. The companies that specialize in storing tapes off site for customers send a truck around on a regular basis to pick up the weekly backup tapes. There have been incidents where entire trucks (and the tapes they were carrying) were stolen. Did anyone steal CRU’s tapes that way? Probably not. The point is however that once the tapes leave your site and are entrusted to another organization for storage, they could be copied by anyone from the truck driver to the janitor at the storage site. Assembling 20 years of email from backup tapes could be a real hassle of course. On the other hand, an offsite storage facility frequently has as part of the service it provides to clients…great big tape libraries for automating copying of tapes. Encryption of backup tapes was a direct response to incidents in which tapes with valuable (and/or embarrassing information) wound up in the wrong hands.
But encryption has only been common for a few years. That raised an interesting theoretical question. The last email release ends in 2009, and the rest of the release is, in fact, encrypted. One can only wonder, does the CRU encrypt their backup tapes, and if so, when did they start doing that?
Administrative Foul Up – One of the biggest “cyber crimes” in history occurred when a company doing seismic processing for oil companies cycled the tapes back to their customers for the next round of data, and sent old tapes to different customers. One of their customers figured it out, and started checking out the data they were being sent which was from their competitors. It wasn’t the first time it happened, and it wasn’t the last time.
Janitor – Let’s be clear, I’m not accusing anyone, just making a point. There’s an old saying about computer security. If you have physical access, then you have access. Anyone with physical access to the computer room itself, and the right technical skills, could have copied anything from anywhere.
The FOIA Requests
There are dozens of emails that provide glimpses into both how the email systems at CRU were run, and how FOIA requests were handled. Some of them raise some very interesting questions. To understand just how complex compliance law can be, here’s a brief real world story. Keep in mind as you read this that we’re talking about American law, and the CRU is subject to British law which isn’t quite the same.
In the early days of compliance law, a large financial services firm was sued by one of their clients. His claim was that he’d sent instructions via email to make changes to his investment portfolio. The changes hadn’t been made and he’d suffered large losses as a result. His problem was that he didn’t have copies of the emails he’d sent (years previous) so his legal case was predicated upon the financial firm having copies of them. To his chagrin, the financial firm had a data retention policy that required all email older than a certain date to be deleted. The financial firm figured they were scot free. Here’s where compliance law starts to get nasty.
A whistle blower revealed that the financial firm had been storing backup tapes in a closet, and had essentially forgotten about them. A quick inspection revealed that a number of the backup tapes were from the time in question. The financial services firm asked the judge for time to restore the data from the tapes, and see what was on them that might be relevant. The judge said no.
The judge entered a default judgment against the financial services firm awarding the complainant $1.3 Billion in damages. The ruling of the court was that the financial services firm was guilty by virtue of the fact that they had told the court the data had been deleted from that time period, but it hadn’t been. They had violated their own data retention policies by not deleting the data, and were guilty on that basis alone. Wake up call for the financial industry…and everyone else subject to compliance law, which includes FOIA requests.
Suddenly deleting information when you said you hadn’t was a crime. Not deleting information when you said you had, was a crime. Keeping information could wind up being used against you. Not keeping information that it turns out you were required to keep (by the tax department for example) could be used against you. No one serious about compliance could possibly take the risk of allowing end users to simply delete or keep whatever they wanted. From their own personal accounts certainly, but not from the company email server. Ever.
In that context, let’s consider just a few words from one email in which Phil Jones, discussing with David Palmer whether or not he’d supplied all the email in regard to a specific FOIA request, says “Eudora tells me…”
These few words raise some serious questions. Eudora is an email client, similar to the more familiar Outlook. So, let us ask ourselves:
Why was David Palmer relying on Phil Jones to report back all the emails he had? Compliance law in most countries would have required that David Palmer have the appropriate search be done by the IT department. This would have captured any emails deleted by Phil Jones that were still retained by the CRU based on their data retention policy.
Was David Palmer aware of the proper procedure (to get the search done by IT)? If not, was he improperly trained and who was responsible for properly training him in terms of responding to FOIA requests? If he was aware… well then why was he talking to Phil Jones about it at all?
Phil Jones specifically says that “Eudora tells me” in his response to Palmer. Since Phil Jones evidently did the search from his own desk top, the only emails he could search for were ones that he had not deleted. But, that doesn’t mean he found all the emails subject to the FOIA request, because email that he did delete was more than likely retained on the CRU email server according to their data retention policies. As in the case of the financial company, the CRU may well have said they didn’t have something that they did. In fact, we can surmise this to be highly likely. There are multiple emails showing up in which, for example, Phil Jones says he is going to delete the message right after sending it. But we now have a copy of that specific message. Did he send it and then forget to delete it? Probably not. The more likely answer is that he did delete it, not realizing that the CRU data retention policy resulted in a copy being left on the server. If the CRU responded to an FOIA request and didn’t include an email that met the FOIA parameters because they failed to search all their email instead of just the email that Phil Jones retained in his personal folder… well, in the US, there would be some prosecutors very interested in advancing their careers…
“Eudora tells me” is even more curious from another perspective. Why “Eudora”? Why didn’t he say that he’d searched all his email? Why specify that he’d used the search capabilities in Eudora? Personally, I have three email systems that I connect to, and a different email client for each one. Searching all my email and searching all my email in just one email client are two completely different things. Most interesting!
While I am comfortable discussing with IT shops how to architect email systems to protect data and properly service legal requirements such as FOIA requests, I have to admit that I wouldn’t know where to start in terms of submitting one. If I did, I just might ask for all the emails they have pertaining to FOIA requests, and I’d be specific about wanting all the email, regardless of it being in the main email system, the archives, or on backup media, and all the email that ever existed, regardless of having been deleted by end users. Then for the capper, I’d ask for their data retention policy and see if they managed to meet it in their response.
Just sayin’
dmh
Discover more from Watts Up With That?
Subscribe to get the latest posts sent to your email.
@davidmhoffer
“BTW, even if you are using a thin client, the email servers themselves still operate via a store and forward mechanism.”
That’s certainly an option but they only need to store until delivery is confirmed which might be from milliseconds to never. If policy is to trash everything more than say 6 months old then it might get trashed that often. Reality doesn’t always match policy and it’s awfully easy to configure the mail server to save a copy of everything. Fourteen years is a long time for “The Team” to not have a clue about it, isn’t it?
“Also, the correspondence I have turned out to be in relation to formatting hard drives and whether or not the data could be recovered afterward (I’m talking low level format here, not an O/S format).”
The problem with that is when a zero-fill format is done it’s almost aways in conjuction with a test of the media as well to locate & mark bad sectors. That means all bits will be raised to one before being lowered to zero as part of the integrity test. I worked for a couple of years in internal diagnostics at Dell and have written more low level disk utilities than I care to remember except for first-to-market things which are cool. I wrote and sold the first hard disk cache (LRU sector algorithm) for an IBM PC or compatibile using LEM (Lotus Intel Microsoft) expanded memory for instance just about exactly 30 years ago.
-Oops, that should be LIM not LEM. I guess there’s too much stink of neo-NASA around here and I was thinking Lunar Excursion Module. 🙂
davidmhoffer says:
December 1, 2011 at 9:24 am
John Whitman;
I am left with whether the self-named “we” was a single person (an “I”) wrt the CR1 and CR2 releases or an actual “we”.>>>
Having zero evidence in regard to how anything was or wasn’t done, exactly how their environment was set up, etc, I am nonetheless inclined to agree. Consider simple things like a backup administrator setting the backup system to cut an extra set of tapes for off site storage, and a truck drived from the off site storage company simply taking them out of the box. There are many ways in which an individual in the right place could pull this off themselves, but two or three people working together…and you have a crazy number of possibilities.
==============================================================
This is why in science, where there are very often many possible explanations, we have the principle called Occam’s Razor which states that the simplest explanation is usually the correct one and you should not add things that are not strictly necessary.
The simplest explanation is there was a 13-year continouous inclusive email archive on a departmental mail server configured to keep just such a continuous inclusive record and then someone with legitimate credentials to access the archive made a copy of it or allowed someone else to make a copy of it. This has whistle blower written all over it or, barring that, someone on the inside was paid off. I’d really like to know if anyone with legitimate access bought any expensive cars lately if you get my drift.
Hackers and tape libraries and anything else I’ve seen mentioned are unnecessary complications.
I mean it could have been intercepted by extraterrestrials described in Erich Von Daniken who beamed it to their hidden base in Area 51 who then transferred it by mental telepathy to an army of elves who transcribed and emailed to a 17-year old Russian kid who knew a guy who could post it. But that’s a lot of unnecessary complication.
Dave Springer;
It also doesn’t matter what kind of client Jones used. I never claimed he was using one or the other.>>>
No. You claimed my article was in error because thick clients were obsolete. I answered that issue and now you are casting my answer in the context of something else completely.
Dave Springer;
This you wrote under your heading EMAIL 101. Now you’re trying to say it wasn’t a general introduction to email (hence 101)>>>
Anyone who read the artcile knows it was a general discussion of email with a specific focus on the most likely technologies that were in use.
Dave Springer;
You REALLY need to learn to stop digging>>>
LOL.
Dave Springer;
That’s certainly an option but they only need to store until delivery is confirmed which might be from milliseconds to never. If policy is to trash everything more than say 6 months old then it might get trashed that often>>>
Which is why I explained what a data retention policy is, how data retention from the end user perspective differs from the email server perspective, how various data protection mechanisms such as snapshots and backup systems work, and how they typically integrate with each other?
Is there any purpose to your sniping? Are you adding any value by attacking snippets of what I said and casting them in an ill light by presenting them out of context? Is there something to be gained by pretending you are adding new information when the article itself covers exactly the information you claim to be adding? Have you spent one second trying to be of value to the discussion rather than smply trying to find fault with my explanations?
Seems to me your are pretty fixated on your dislike for me. Tiresome frankly. Don’t go away mad.
Just go away.
REPLY: BOTH OF YOU JUST STOP- PLEASE- DON”T MAKE ME THROW YOU IN THE TROLL BIN – Anthony
Dave Springer;
The simplest explanation is there was a 13-year continouous inclusive email >>>
The prupose of the article was to explain the main technologies involved in order to illustrate the possibilities. Applying Occum’s Raiser to arrive at any conclusion at all simply by pointing the finger at the simplest possible method displays a total and complete misunderstanding of the complexity of the systems involved, the security systems that were very likely in place for the specific purpose of preventing access by the simplest and most obvious methods, and your fixation on attacking me personally for whatever reasons you have to do so.
Hugs and kisses.
This article also does not discuss the difference between email protocols, the two most prevalent being POP and IMAP. This is important since how your email client interacts with the email server and how emails are retrieved and deleted is very different. Most home users are probably using webmail clients (browser based) and need to check if it is using POP or IMAP. For example Comcast, Hotmail and Yahoo! use POP3, while Gmail uses both.
dmacleo;
dmacleogravacleo says:
November 30, 2011 at 10:27 am
question on exchange-outlook relationship where non-cached is used.
where is the local copy kept?
there is no ost file like cached mode, is it a temp file deleted at some point?>>>
I pinged one of my contacts and he explained it this way. In “online” mode, also called “cached” mode, the email is never stored on the hard drive of your desk top computer. So, while Outlook itself is installed on your desk top computer, when you read an email, it is downloaded to your desk top computer, but only stored in RAM (memory). When you close the email, from the perspective of your desk top computer, it is “gone”. No permanent copy was ever made to the desk top computer’s hard drive.
This is slightly different from what a web client (thin client) does, but has a very similar over all result. If you want to make a copy of your emails to (for example) a USB drive, you would have to “export” them to a .pst file and store it on your USB drive. However, you cannot do that unless the email server itself is configured to ALLOW that to be done. A lot of IT shops shut that feature off for the specific purpose of preventing end users from walking off site with all their emails on a USB key.
Whoever these DUQU folks are, they sure showed us how to wipe your trail out properly!!
http://www.computerworld.com/s/article/9222293/Duqu_hackers_scrub_evidence_from_command_servers_shut_down_spying_op?taxonomyId=85#disqus_thread
From A Public Servant’s account of UK FOI laws, they adhere to the commandment, “Thou shalt remember the legal fictions, and keep them holy.” Deemed deletion => real deletion? Bizarre indeed, as davidmhoffer sez..
BTW, david, there is not, thankfully, any such word in English as “becomed”. Irregular ol’ Anglo-Saxon verb, past tense “became”. Honest.
CRS, Dr.P.H. says:
December 1, 2011 at 11:08 pm
Whoever these DUQU folks are, they sure showed us how to wipe your trail out properly!! >>>
Read the full article and some related articles, thanks for the link!. That is one seriously nasty viscious brilliant piece (or pieces) of code those guys wrote. At the beginning of the article when they theorized that there were government agencies behind it, I thought… yeah right.
Then I started reading a bit more on how the darn thing worked. That’s one seriously heavy duty chunk of code. It works by tunneling across operating systems, network protocols, even invades versions of one piece of code with one exploit, and then used that to upgrade to a newer version in order to take advantage of a different exploit. WOW!
This isn’t the kind of thing that you can write on a PC. You’d need an entire network with all the operating system variants, protocol variants, edge network devices like firewalls, load balancers, network sniffers and on and on just to test it and see if it works as designed. You’d even need another whole environment to represent the “target” environment otherwise when you did payload testing you’d wipe out your own computers.
One can’t help but be both impressed and scared sh*tless at the same time.
Brian H;
BTW, david, there is not, thankfully, any such word in English as “becomed”. >>>
Sigh. You are correct. You’d think that with “unbecoming” and “becoming” and “become” all being real words that “becomed” would be too? Or would it be “becamed”? 😉
There’s no such word as “discombooberated” either, but when I read the ClimateGate emails…it seems appropriate. After all, the emails were appropriated in some manner from their proprieters, and when one reads the explanations of “the team” it is clear that they have become discomboobriated. Quite unbecoming.
offline is cached, it generates the ost file.
online (non cached) is similar to thin client, i suspected and you confirmed held in ram.
I appreciate your looking into that.
I have set server (SBS 2011 now) up to allow export to pst, small network so I can get away with it,
as far as the disc space I was wondering about…well lets just say I am too embarrassed to say what it was. just accept I was being really stupid and failed to actually look at what I needed to LOL LOL
dmacleo;
as far as the disc space I was wondering about…well lets just say I am too embarrassed to say what it was. just accept I was being really stupid and failed to actually look at what I needed to LOL LOL>>>
Never be embarrased about this stuff. As a couple of commenters pointed out, IT is now so complicated that nobody on earth understands all the technologies and all the options. Get good at the stuff you do every day, and reach out for expertise on the “once in a while” stuff. The real trick is knowing when to reach out because you don’t know what you don’t know. At least, that’s my philosophy. Here’s a brief anecdotal story that ought to make you feel better.
I was in a meeting with a very large ($ Billions/year) company that was having trouble with their high performance compute farm. They’d bought entire racks of servers from three different manufacturers, and, in their words, “none of them work”. The complaint was that they were getting about 1/8th the performance they thought they should be getting. I asked to see the list of software they were running.
The problem was that their software was all “single threaded”, but their server blades had 8 cpu cores on them each. So…the maximum they could get out of a single threaded application was 1 cpu core out of the 8 actually doing any work. Talk about some red faces. Then I suggested that they run virtualization software (and there are free versions that suited their purposes) and then they could run 8 O/S instances on each blade, and hence 8 threads on each blade. More red faces.
I won’t tell you who the company was, but I will tell you that they are one of the largest software companies in the world.
Feel better now? LOL
HA LOL
a little better, still embarrassed at myself though 🙂
I just remembered a better story. I was travelling, and forwarded one email account to another so I would get both sets of inbound email on my blackberry (this was before blackberries could have multiple email accounts). Then I forgot that I had done it. I moved my blackberry service from the 2nd account to the first, but wanted all my email visible on my blackberry, so forwarded all my inbound email from the second account to the first account…
32,000 emails later, I exceeded some limit or other on one of the accounts , or a firewall rule woke up, I’m not even sure what broke the infinite loop. Woke up in the morning, and had 32,000 unread emails. Actually, 64,000 (32k in each account).
The razzing didn’t last as long as the time I wrote a piece of code with an infinite loop that had a print command in it that went to a 400 page per minute line printer….
David,
I wonder if any outsourcing has been involved here? We do not even realize how lax security has become through outsourcing. I lost my job 2 years ago, shortly after demonstrating to management how they were allowing US military secrets to be accessible to employees (administrators) residing in foreign lands. After I documented how it could be done, it was taken to the very top of this VERY large corporation. My manager was forced to sign of that there was no security risks even though I could prove it!
They did not want to hear about this, because they could save money by using the cheaper labor. This just goes to show what a lack of leadership we have today!
Oh, and while I am not certain, I think this data is still sitting available today.
http://www.sympa.org/documentation/article_smime/sympasmime.html
It seems that CRU,s MIME device left his footprint to email , some cryptic strings.
For exampe. ??——————————————
Maybe they didn´regognize that string keeps inside a message ID, and the orginal
message can be found with that ID.
Is that an accident, are they really so silly.
And the MI5 investigating what happened.
Im´ just taking a closer look at their MIME device, some UAX or ….
Hold on, is this only a dream??, i must check it.
Ilkka
Mike Wilson says:
December 5, 2011 at 7:09 am
David,
I wonder if any outsourcing has been involved here? We do not even realize how lax security has become through outsourcing.>>>
Yes, outsourcing is a huge security risk. In most cases, outsourcing contracts rely on contractual obligations to enforce security. The client gets legal commits and then frequently washes their hands of the risk, as it is up to the supplier of the outsourced services to put the proper security in place. Oddly, many clients will put the outsourcer through the ringer to prove they are capable of providing the services being contracted for, and then just sign a piece of paper without any due diligence on the security aspects.
Was outsourcing part of trhe mix here? I don’t know. From the various snippets of information that we get, it looks like UofEA had their own central IT department running the email system. But off site storage of backup tapes is almost always an outsourced service. Implementation of new technologies not familiar to the IT group is often outsourced to a contractor with experience and expertise who comes on site to get things up and running. I don’t know that anything at all was outsourced by UofEA, but it certainly is a possibility, and in fact likely for at least some tasks.
David,
Not says this is what happened at CRU, but the following did happen:
Imagine this scenario. Large outsourcer is taking over data centers for many customers. Some customers have extremely sensitive data of national security interest and some don’t. Outsourcer wants to offer ATL(s) (automated tape libraries) to customers, but these libraries cost 1 or 2 million apiece. So to save money, a management & marketing decision (with customers approval at the time) is made to place multiple customers data in a shared ATL. The ATL is configured by the storage administrator as directed in such a way that nobody on customer A system can access data in the ATL that resides on customer B (it is actually also customers c,d,e,f,g & h) except the storage administrator(s) that holds the keys. Now imagine after this is all setup and running quite well, management starts thinking these storage administrators are very expensive and we have cheaper people overseas. Management starts making decisions w/o input from current storage administrators. They know good and well that customer A and C has very sensitive data that can not be legally administered overseas. But due to many internal reorgs and management changes, they really have no clue as to how the systems are setup or even that the ATLs are shared or even what that means. So the plan becomes to divvy up support for individual customers according to who can be managed overseas and who cannot. But the problem is a storage administrator must have complete control of all the data due to the nature of his job and to manage the hardware and data. So now when storage administrator learns of these plans, he raises a red flag, but by then, the ball has already rolled a long ways down hill & decision makers have much crow to eat if plans are halted. THAT MUST NOT HAPPEN SAYS MANAGEMENT! WE DON’T LIKE CROW!
So now you have ATLs and VTSs with multiple companies data in them with storage administrators in multiple countries that (if they know what they are doing) can access the tape data of any system in the ATL or VTS.
And do you think any of the companies that outsourced their systems even know about this?
If you dont believe me, why dont you take a call to Phil Jones?
I almost hear MIME devices dial tones.
Click the link.
http://www.ecowho.com/foia.php?search=+______________________________________________________________
Oh yea, if you are reading this and you are in the United States. Chances are good your data is in there too. That is if you have credit or if you have had medical tests performed. Otherwise nothing to worry about. Actually I think the medical data may have been removed, not sure though, have not been there for a while now. I was the trouble maker that went 1st.
And yes, David is absolutely correct. We storage administrators have access to EVERYTHING! Nobody has more access on the system to the data. Generally not even the data security admin (although they could grant themselves access if they wanted too).
Mike Wilson;
Your example of an outsourcer and changing administrative roles that result in a security risk happens… way too often. Not only does management not want to eat crow, most customers don’t even think to ask the question in the first place, let alone demand a contractual obligation to implement best practices that would protect them.
To underline just how much access to data a storage admin has, and how they can cover their tracks, here is an amusing story involving a telephone company who shall remain nameless. I’ll cal them Telco for easy reference.
Telco had a study done for them on their customer satisfaction versus their competition in the market for cell phones. One company came out WAY on top, so Telco decided to take a closer look at that company’s web site to see what made them so “good”. To Telco’s surprise, the packages being offered on the web site were identical to their own. The website seemed to be a one man operation, and to top things off, the prices were well below Telco’s cost of operation. How could this one man show have replicated Telco’s services right down to the last feature and offer them for less than Telco themselves could despite being a $10 Billion/yr company that owned their own cell tower infrastructure.
A little sleuthing turned up the fact that the one man band was, in fact, a storage administrator that worked at Telco. Every evening, he would take a “snapshot” of the data in all the admin systems. He would enter all the cell phone orders that he had gotten from his web site that day, and initialize them. This would trigger the admin systems to contact the phone switches themseleves and provision services to the cell phone being set up (including the number). This is where the power of the storage admin in that position becomes clear.
Once all the services were provisioned in the switching network, the storage admin would interrupt the admin systems, kill the current copy of the data, and then promote the “snapshot” he had just taken previously to be the current copy of the data. The net result was that all the services that the rogue sys admin had provisioned for his customers were now up and running in the cellular network itself, but there was no record of them ever having been initialized in the billing system… and hence, no bill.
I have way too many of these stories, but here’s a couple more quick points:
Question: Can you preserve a document that you wrote on a computer by printing it and saving the hard copy?
Answer: Not unless you want to go to jail. Evidentiary law in both the United States and Canada requires that all documents be preserved with all their original attributes. If you print a hard copy it is no longer searchable, it no longer has meta data (date of creation, modification, author, etc) and hence is a violation of compliance law.
Question: If you are a Canadian publicly traded company, are you subject to Sarbanes Oxley in the United States even if you do not trade on an American stock exchange?
Answer: Possibly. SOX applies regardless of where your shares are traded provided that 20% (I think) or more of your shares are held by American citizens. I did work for one Canadian company who was sure they were exempt until I asked if they had an employee share program, and how many of their American employees held how many shares….ooops!
Question: If you are in Canada and are an executive of a company that has violated SOX, can you avoid arrest and prosecution in the United States simply by never going there? SOX is not an extraditable offense, so one ought to be safe, right?
Answer: Without naming names, there’s a fellow in jail in the United States right now who figured he could get avoind prosecution by simply not travelling to the US. But, he decided to take a direct flight from Vancouver to Mexico. The moment his plane crossed into American air space…hello… where’d all those fighter jets come from? They forced the plane down in the US and arrested him.
The fact those of us who work in IT don’t simply lose our minds trying to figure this stuff out and architect systems that both work and are secure… wait a sec…everyone who knows me says I lost my mind a long time ago…
I have always been fond of the old “The Butler Did It!” solution. In this case the ‘butler’ is any long-standing, trusted but underpaid employee with almost unlimited access, good attention to detail, and a strong moral sense. The motivation would be righteous indignation of the employee against what he perceives as wrong-doing by his employers.
His trusted standing with his employers would defer any suspicion, partially because people who react emotionally before they think (the employers) are likely to look for a scapegoat before they will buckle down and analyze the problem. By that time it is easy to plant and nurture suspicion against your chosen nemesis.
You see in case after case of criminal law that once a suspect has been identified, all efforts turn to finding evidence to convict that suspect rather than finding evidence of exactly what happened.
The anonymous leaker will remain so unless he publicly confesses, simply because the Climategaters *want* to believe they were ‘hacked.’