Figuring out who was behind the hack of the Climatic Research Unit (CRU) at the University of East Anglia requires some digital forensic skills – and an insight into the mindset of those who were trying to get at CRU's files at the time.
Analysis by the Guardian and digital forensics experts suggests that an outside hacker gained access to a server at the UEA which held backups of CRU emails and a collection of staff documents. It also suggests the access occurred over a period of days, if not weeks, and was carried out from a computer based on the east coast of north America.
The release of hacked emails and documents came just months after climate change sceptics had filed more than 50 freedom of information requests querying the CRU's refusal to release of raw data and program code during the summer.
Egged on by a group of sceptical bloggers, the requests almost all began with the words "I hereby make a EIR/FoI request in respect to any confidentiality agreements restricting transmission of CRUTEM data to non-academics involing the following countries." Others sought "a copy of any digital version of the CRUTEM station data set that has been sent from CRU to Peter Webster and/or any other person at Georgia Tech". All were refused under FoI exemptions because of commercial confidentiality.
Into that silence came the release of the archived "zip" file by someone with clear hacking skills: first they grabbed the files, then they broke into the RealClimate blog to upload the archive and prepare a draft post; then, when that was thwarted, they uploaded it to a Russian website, and posted links to it on climate sceptics' blogs using web servers located in Saudi Arabia and Turkey.
That sequence of events led Sir David King, the government's former chief scientist, to say that it must have been "carried out by a team of skilled professionals, either on behalf of a foreign government or at the behest of anti-climate change lobbyists in the United States". But he quickly backed away from that statement, admitting he had no inside information.
The Guardian's analysis shows that a small group of just four of the scientists from among the dozens employed at the CRU were targeted in the sifting of email. They are: Phil Jones, the head of the CRU; Professor Keith Briffa, who studied tree rings; Tim Osborn, who worked on climate modelling for modern and archaeological data; and Mike Hulme, director of the Tyndall Centre for Climate Change Research. All are either recipients or senders of all but 66 of the 1,073 emails, and almost all the rest are sent from mailing lists, such as the Met Office's "scenarios" listing, to which at least one of the four would certainly belong.
A few remaining emails are sent by, or to, other CRU staff – indicating that the hacker had access to a backup server holding CRU emails dating back to 1996. That it is a backup is confirmed by the presence of a duplicate sent to Osborn: separated by one second, both have the same document attached, but from different machines. That suggests that the UEA's system administrators had backed up emails from CRU staff's machines onto a server – and that the hacker got into it, and also at a set of documents held on the same machine.
Jones, Briffa, Osborn and Hulme had been the focus of sceptics' ire because their high-profile scientific papers had been used to back the IPCC's reports on global warming. At the same time they had declined to release either the data (citing commercial agreements with suppliers) or the computer code they had used to analyse that data and draw their conclusions, to the frustration of many outside academia who wanted to repeat – or discredit – the work.
Early speculation that the release of the emails and documents came from a one-off hack also appear to be wrong. Digital forensic analysis shows that the zipped archive of emails and documents was not produced on a single date. Instead it was created by copying the files over a number of weeks, with bursts on 30 September 2009, 10 October and 16 November. On the last date a folder of computer analysis code by Osborn was added to the package.
The digital forensics on the files indicate that they were created on a computer set at some times four hours behind GMT, and at others five hours behind – plants the hacker on the eastern seaboard of Canada or the US.
Then early on 17 November, RealClimate's blog was hacked, locking out legitimate administrators, and the hacker tried to create a blogpost claiming that global warming was a myth, and enclosing the emails and documents.
Gavin Schmidt, one of the RealClimate administrators, says that "my information is that it was a hack into [CRU's] backup mail server".
But who was the hacker, and what were they after? Jeff Condon, who runs the climate-sceptical Air Vent blog – which posted one of the links to the archive – told the Guardian that the content of the emails and documents actually points to someone who is not expert in the topic.
Referring to an email it includes from Tim Osborn which says "we usually stop the series in 1960", Condon says that: "The only interesting detail in that email was the data, but that's not what the person wrote. What that means to me is that whomever posted these emails doesn't have a terribly deep understanding of the issues in paleoclimate science. Although the emails themselves featured some scientists who do know the issues and had some very nice details in them.
"Therefore if it's an inside job, it's likely not by a paleo or climate grad student, definitely not by a scientist," Condon said, adding: "If it's an international conspiracy I would have guessed someone on the team would know the science better than that."
But how would an outside hacker get in? Although UEA has security in place, it has seen a number of accidental security breaches of the UEA system in the recent past. On one occasion a server was configured wrongly, so that anyone outside doing a search would "fall through" to directories of files. (UEA closed that hole after being alerted about it.) A misconfigured server could have left just the hole that a capable hacker with a determination to find the data being denied via FoI requests could have exploited. But they are not government-class skills.
So what was the hacker looking for, and how? Besides the clear targeting of the four scientists, it is obvious that this is not the entirety of the CRU's emails: there are none of the routine administrative messages about fire alarms, holiday reminders and so on. Therefore the emails have been filtered. One quick way to see into the hacker's mind is to use "concordance analysis" - examining what the common words or phrases are in the emails and documents. Though usually used in linguistics to compare translations or the frequency of words, concordance software can be used to demonstrate authorship of papers, by combining a "stoplist" of words to be ignored (such as "the" or "and") with a straight analysis of the frequency of words in the text.
Concordance analysis of the emails suggests that the hacker did some careful sifting. But working out precisely what is complicated by the fact that this is the wheat – not the chaff. For instance, the hacker has clearly removed standard words such as "holiday" – except where they appear in emails to or from Jones, Briffa, Osborn or Hulme. There's no other way to explain how such a comprehensive catalogue has so few emails about time off.
Instead, emails with the words "data", "climate", "paper", "research", "temperature" and "model" prevail, according to a concordance plot. That may have been precisely what the hacker was looking for – and the fact that he also ignited a controversy over techniques might have been a surprise to him as well as the rest of the world.
(Note 5 Feb 12:42GMT: the concordance analysis that was here has been moved to a separate file. We will also post a graphic of the analysis in due course.)
