Three years ago the former Whitehall security and intelligence chief Sir David Omand warned that the privacy rights of the public would have to be sacrificed to give the police and security services the access to a sweeping range of sensitive personal data they needed to combat terrorism and serious crime.
A key architect of the government's national security strategy, he gave the most candid assessment yet of the scale of Whitehall's ambitions to be able to mine the confidential personal data of the innocent as well as the suspect so that pattern-recognition software could assist criminal investigations.
"This is personal information about individuals that reside in databases such as advanced passenger information, airline bookings and other travel data, passport and biometric data, immigration, identity and border records, criminal records and other governmental and private-sector data, including financial and telephone and other communications records," he said in his Institute of Public Policy Research pamphlet.
When Omand highlighted the then little understood privacy dangers of the database state in 2009, the Labour government was pressing ahead with plans to track and monitor every citizen's use of email, mobile phone texts and social media. It was also proposing to reverse a basic principle of the Data Protection Act to allow far wider sharing of the public's confidential personal data across Whitehall.
It was forced to abandon both plans in the face of strong opposition from those worried about the unchecked growth of the database state, including from leading Conservative and Liberal Democrat politicians. Now both plans are about to be put back on the table.
The phone-hacking scandal has highlighted how intrusive surveillance methods can destroy individual personal privacy. Omand argued then that public trust would only be maintained in the state's use of data-mining techniques if it was carried out within a strong ethical human rights framework. Many of the databases that hold sensitive personal details have been built with the best of intentions, such as identifying children at risk of abuse by bringing together data from health, welfare, police and education records.
The coalition government came to power on a mandate of rolling back the "surveillance society". The national identity card scheme was abandoned but databases continue to proliferate across Whitehall, even before the extension of data-sharing powers. Now the Cabinet Office minister, Francis Maude, says goverment must be "smarter and more effective" at sharing such sensitive data.
A newly created Home Office database holds anonymised information on more than one million drug-misusing individuals identified through contact with treatment services or criminal justice agencies between 2005 and 2009. Smarter matching of health, criminal justice and employment records is already being used to ensure that individuals are "sustaining their recovery and getting back into work" after leaving treatment. The project is also being used to deepen the official understanding of the impact of drug treatment. Researchers are to be given access to the data warehouse.
The government is also identifying unemployed families for action by its troubled families units. Ministers acknowledge that data protection rules still apply in these cases, and the units will have to seek the informed consent of each family before detailed case histories are passed over. The information cannot be shared with any other government agency and a secure email account has to be established by jobcentres for the sole purpose of identifying troubled families.
The government is pushing ahead with plans to enable patient data from GP practices to be more widely shared with the drug industry. Under a scheme launched three weeks ago, anonymised patient data is to be shared with drug company and NHS researchers on an unprecedented scale.
Last week the European parliament ratified plans to allow airline passenger records, including credit card details, for all transatlantic flights between Europe and the US, including in and out of the UK, to be handed over to the US department of homeland security to be stored for 15 years.
It is six years since the then information commissioner, Richard Thomas, first warned that Britain was "in danger of sleepwalking into a surveillance society". He portrayed Britain in 2016 as a country where technology was extensively and routinely used to track and record our everyday activities and movements, fostering a climate of suspicion and undermining trust. Not long to go now.
