Home Office security officials will rely on overseas-based social media companies such as Facebook and Twitter to comply voluntarily with requests from British police and security services to collect and store personal data tracking web, email and mobile text use.
The detailed draft communications data bill published on Thursday by the home secretary, Theresa May, includes provision for civil court injunctions to be obtained against British-based internet and phone companies that fail to comply with a scheme that critics have dubbed "a snoopers' charter".
Home Office officials appear to be relying on their claim that most overseas communication companies operate under "similar, if not more intrusive legal regimes without the rigorous safeguards" provided by the proposed UK legislation. But precise details of the powers to seize data and force companies to monitor it have been deferred until regulations are published.
May said that not all companies would be required to store all records of internet use but declined to name which companies would be included. However, an official privacy impact assessment published alongside the draft legislation acknowledges that giving police and security services access to personal communications data that is not already collected for billing purposes by the web and phone companies does involve risks to personal privacy.
The Home Office says there are risks that data may be accessed without the necessary approvals, that inaccurate data may be supplied and that it may be insecurely stored, but argues that new safeguards in the legislation mitigate the risks.
A general impact assessment also confirms that the Home Office has given a "blank cheque" to the communications industry and will reimburse all the costs of collecting and storing the personal data for up to 12 months. It does, however, disclose that the minimum price tag to the taxpayer is expected to be at least £1.8bn over the next 10 years and that this does not allow for VAT, inflation or depreciation. It appears that £859m will be spent in the earlier years of the project to invest in the systems needed to capture, retain and transmit the data.
The Home Office admits this is likely to understate the final costs of a scheme that will have to adapt to the rapid pace of technological development, such as the introduction of mobile 4G, and changing behaviour patterns in using the web.
Ministers claim that the cost of the scheme, which they say is vital to the fight against crime and terrorism, is outweighed by benefits, valued at £5bn to £6.2bn, from preventing tax fraud, seizing criminal assets and safeguarding lives.
More than 550,000 requests for communication data were made by the police and security services last year to assist in investigations. This figure can be expected to increase if the powers are expanded.
The home secretary insists that the internet state snooping scheme will involve only the collection and storage of the "header data" – who sent what to whom, when and from where – and not the actual content of the message. But a debate is raging among technical experts about whether the system of "automatic filters" that the legislation envisages is to be put in place to collect the data can do that without grabbing the content too.
Charles Farr, the head of the Home Office office of security and counter-terrorism, said he was confident the technology existed and was reliable enough to achieve that. Interception of the content of email, phone and other communications needs a warrant signed by the home secretary.
May said communications data was a vital tool for the police to catch criminals and to protect children. She said: "If we stand by as technology changes we will leave police officers fighting crime with one hand tied behind their backs.
"Checking communication records, not content, is a crucial part of day-to-day policing and the fingerprinting of the modern age. We are determined to ensure its continued availability in cracking down on crime."
The home secretary said the plan was a coalition policy and now had the backing of the deputy prime minister, Nick Clegg. A special scrutiny committee is to examine the detail of the bill before it is introduced into parliament in its final version.
But former Conservative home secretary David Davis sharply criticised the scheme as "incredibly intrusive" and recalled that David Cameron himself had attacked a very similar scheme when it was put forward by Labour in 2009.
"If they really want to do things like this – and we all accept they use data to catch criminals – get a warrant. Get a judge to sign a warrant, not the guy at the next desk, not somebody else in the same organisation," said Davis.
He was backed by Rachel Robinson of Liberty, the human rights organisation, who said: "The argument for the snooper's charter is that we must all compromise our privacy because crime sometimes happens on the web."
A leading Liberal Democrat critic, Julian Huppert, who is to sit on the scrutiny committee, said that his immediate concern was that clause 1 of the bill would allow "pervasive black boxes that would monitor every online information flow – an idea which is clearly unacceptable".
Burgess Powers builds and designs web sites and applications for high availability