What is big data?

It's a bit of a misnomer: "Ex-Guardian writer Simon Rogers once said, 'Big data is data that is one bit too much for you to be comfortable', and this is probably the best definition I've read.
"The volume of data is not irrelevant, but not as important as it sounds. More important is the ability to link diverse datasets with each other."
– Giuseppe Sollazzo, senior systems analyst at St George's, University of London, and member of the open data user group

It's about time we dropped the "big" jargon: "It's just data. Data volume and velocity is exponentially increasing, but has always been too big to easily store and process … One thing that has changed in the last few years is the recognition from decision makers – not just analysts – that data is a valuable resource."
– Tom Smith, director of the Oxford Consultants for Social Inclusion

Why is it important for government?

"Joining up public sector data sources can make government more efficient, save money, identify fraud and help public bodies better serve their citizens."
– Claire Vyvyan, executive director and general manager of public sector at Dell UK

"Data can enable government to do existing things more cheaply, do existing things better and do new things we don't currently do."
– Tom Heath, head of research at the Open Data Institute

Examples of good uses of big data in public services

Data on prescription drugs: "By analysing who prescribed what, when and where, Nesta/Mastodon C were able to show that it took GPs several years to catch up with NICE guidelines – in other words you are unlikely to be getting the most up to date medicine prescribed for you by your GP."
– Alexander Stevenson, author of The Public Sector: Managing the Unmanageable

A great example of joining up data: "During recent bad weather a local authority blended data about services, such as road gritting rotas, with services for people at risk, such as 'meals on wheels'. This made sure that the services were connected."
– Paul Brook, EMEA programme manager on big data and cloud solutions at Dell UK

Which areas of public services stand to gain most from big data?

The opportunity for adult social care: "Big data can help by analysing data in new ways to help social workers make better decisions about when and how to intervene, with the specific aim of helping people live independently for as long as possible."
– Alexander Stevenson

Health, education and transport: "The biggest potential is the development of new, emergent data-led behaviours. If by analysing health data I know I am statistically more at risk of heart disease, I will modify my lifestyle."
– Giuseppe Sollazzo

Health, personalising public services, fraud and error: "Particularly detection at the point of origin (my credit card company manage this pretty effectively – I know government is more complex but the principle is the same)."
– Chris Yiu, director of digital participation at the Scottish Council for Voluntary Organisations

The police is a sometimes overlooked area of innovation: "There are obvious win-win approaches, such as developing systems to use social media as early warning systems after a major disaster. On the more questionable side, there is using social media data for intelligence gathering and surveillance."
– Ben Taylor, funded by the Arts and Humanities Research Council to write a parliament research briefing on social media and big data

Combining data from the public and private sectors: "For example, if I were empowered to combine data from my bank statements with open regulatory data about the environmental impact of various companies/shops/suppliers, I could make much more informed choices about how I spend my money."
– Tom Heath

Data and analysis should be able to help most with: "Better targeting services, evaluating what works, identifying early interventions (especially important given the squeeze on budgets) and service design (lots we can learn from the private sector here)."
– Tom Smith

A very knotty challenge about ownership of personal data: "There's big potential here, and possibly our best hope for some seemingly intractable problems like obesity (access to real-time personal data might be our best shot at enabling the behaviour change required to alleviate the burden on our healthcare system)."
– Chris Yiu

What are the barriers to better use of big data?

The technology part is easy but there are some practical issues: "Is the data open so it can be used, is the public comfortable with data being used in this way (and some data is clearly more sensitive than others)?"
– Claire Vyvyan

The problem is often in resistance to change: "Managers are generally people managers, not data analysts. Getting them to adopt technology for data-driven decision-making is often the largest obstacle."
– Lillian Pierson, founder of Data-Mania

Complexity of government makes it difficult: "Data analysis often needs multiple parts of government (central, multi-tiered local, arms length) to work together to deliver the desired outcome. That's a challenge that the private sector initiatives don't face at anything like the same level of complexity."
– Peter Wells, project lead on Labour's digital government review

Lack of investment in skills: "Policymakers are dazzled by possibilities but don't invest in the basics, such as a solid grounding in statistics, properly articulating the question and interrogating the conclusions."
– Chris Yiu

What are the risks?

Privacy and data protection: "The UK Data Protection Act is quite vague. Consent is important, and any model that involves simply harvesting personal data will run into problems if people feel they have not been given a choice."
– Ben Taylor

Data protection doesn't feel very strategic: "We seem to have extraordinary laxity in areas of national security, and extraordinary strictness in other areas such as knowing when people have their bins collected. The Snowden leaks will (quite reasonably) undermine public confidence which in turn will make it harder to make the case for sharing data about bin collections."
– Alexander Stevenson

A community surrenders some privacy: "I grew up on a big housing estate. Privacy is important but sometimes the community surrenders elements of this and benefits from being, well, a community – from knowing who stole your bike to knowing when a neighbour has a birthday. Big data need not be so different."
– Paul Brook

It's a question of trust and public acceptability: "It would be interesting to see which types of big data analysis are more accepted by the general public. I'd imagine quite a lot might be to do with who is undertaking the analysis and what their motivations are for doing so."
– Ed Parkes, senior programme manager at Nesta

We are heading for a national debate on data and privacy: "The question is, which (if any) of our politicians are brave enough to take it on?"
– Chris Yiu

How can government get better at using big data?

Use more statisticians, analysts, or data scientists: "IT departments classically serve a vital role but they're generally not populated with great analysts. IT departments are generally very threatened by government (internal) businesses having coders and programmers working on and making sense of their data."
– Lillian Pierson

Licensing: "The two most important changes are releasing data under an open licence and guaranteeing a minimum updating frequency for data that changes."
Promote champions of data: "Many times the use of a data-centric approach is stopped by "blockers" at senior level because of a lack of understanding."
– Giuseppe Sollazzo

Incentivise use of data and analytics: "Sort out digital and data skills, and basic scientific literacy, and put this into the core civil service competency framework. Find new ways to incentivise the use of data, including (but not limited to) payment-by-results."
– Chris Yiu

Give more support on legal aspects: "The technology issues are simple by comparison."
– Paul Brook

Open up safe access to data held by government: "Then others can use this information to target and improve services. Safe access to this data does not need to allow access to the underlying personal level microdata."
– Tom Smith

Hackdays: "I see hackdays as a great accelerated brainstorming/prototyping tool. They are suited to digging into a dataset for the first time, or to try and discuss novel ideas about it."
– Giuseppe Sollazzo

Publishers should consider their data engagement strategy: "This can be relatively simple in terms of making sure you've got the right metadata and publish guidance, but can stretch to things like making sure you listen to feedback from users of your data. A straightforward thing you can do is to ask people what data they actually want to get access to."
– Ed Parkes

• Big data video – register your interest

• Want your say? Email us at public.leaders@theguardian.com.

Join the Public Leaders Network for more comment, analysis and job opportunities, direct to your inbox. Follow us on twitter via @Guardianpublic.