Clear guidelines must be introduced in law to allow the security services and police to intercept social media such as Facebook and Twitter while avoiding the "chilling effect" of state surveillance, according to a former Whitehall intelligence chief.

The explosive growth in social media has provided criminals, paedophiles and terrorists a "secret space", warns Sir David Omand, former director of GCHQ, the government's electronic eavesdropping centre, and Whitehall's former security and intelligence coordinator.

However, the ability of state security agencies and the police to intercept social network communications such as tweets must be placed on a clear legal footing, he says.

A report entitled #Intelligence, published by the thinktank Demos and co-authored by Omand, says existing laws regulating the interception of communications by security agencies and the police needed to be overhauled to meet the new challenges and opportunities presented by social media.

Intelligence gathered from social media – which it calls "Socmint" – could be important in identifying criminal activity or threats to the public. But the public needed to be confident it was not being abused.

"Democratic legitimacy demands that, where new methods of intelligence gathering and use are to be introduced, they should be on a firm legal basis and rest on parliamentary and public understanding of what is involved, even if the operational details of the sources and methods used must sometimes remain secret," says the report.

"In respect of Socmint these conditions of democratic legitimacy are presently absent," it adds. There was a pressing need to distinguish between information obtained through open sources and information that can only be gathered by eavesdropping – that is, between information that people make openly available on social media sites and information that should be considered private and should be accessed only through warrants issued by a minister or a judge. "Moreover, new and emerging technology potentially allows more invisible and widespread intrusive surveillance than ever before," it adds. "Without an explicitly articulated approach towards generating Socmint based on respect for human rights and the associated principles of accountability, proportionality and necessity, there is a serious risk that this vital confidence and trust will be undermined."

The report continues: "There is also the danger that Socmint could result in a chilling effect on the use of social media itself, which would have negative economic and social consequences for the country as a whole."

The report is published amid huge controversy over government plans to extend the monitoring of phone calls, emails and internet traffic to social media and Skype.

"Socmint must be based on a publicly argued and sound legal footing, with clarity and transparency, over use, storage, purpose, regulation and accountability," says the report, whose three authors include Jamie Bartlett and Carl Miller.

Introducing the report, Omand suggested that the 2000 Regulation of Investigatory Powers Act, the existing statute governing communications intercepts and state surveillance, is out of date.

He also emphasised the need to ensure that intelligence is not gathered for political reasons. "After Iraq, we must be sure that if people are engaged in this kind of monitoring, they are doing it for the reasons set out in the authorisation [and that] it has not been politicised," he said.

Omand referred to a number of principles he has already laid out governing the activities of the security and intelligence agencies. A version has been adapted and incorporated in the Guardian's own code of practice covering investigative journalism.

The principles include sufficient cause and integrity of motive, the methods used must be proportionate to the seriousness of story and its public interest, there must be proper authority – any intrusion must be authorised at a sufficiently senior level and with appropriate oversight – and there must be a reasonable prospect of success.

The report's authors point out that last year's riots demonstrated that the police failed to appreciate the opportunities provided by social media used by members of the public who wanted to alert them.