We consider open data to be part of a broader trend towards “open government” where open data combines with social media, mobile technology and other feedback mechanisms to transform the relationship governments have with citizens, delivering better, more relevant public services (which we could broadly term “citizen-centric” open data). Open data also has the power to improve individuals’ lives through private or third sector innovation on the back of publicly-available data sets, resulting in valuable services and economic growth (which we could term “consumer-centric” open data).
Our analysis of these global trends and our experience has underlined that the US government is right to think of transparency and open data as an evolutionary process that should adapt flexibly to changing priorities. Public sector managers and citizens have changed the way they use the available information:
– Citizen-user feedback has diminished over time, but there is increasing external interest in more specialized data sets.
– Performance dashboards provided public servants with a good start to allow better management and comparison between agencies, and they now have appetite for more detailed analytic capabilities to conduct sophisticated analysis and drive change at agencies.
Where the publication of raw data sets has been useful for specialist developers, in our experience it has not always reached its full potential. Experience so far shows that data is most useful when it is released in a consumable fashion, and usability (for both the specialist developer audience as well as the non-specialist citizen audience) should be a key consideration in making open data real. Under-exploitation of the data already in the public domain may be for two reasons: the data is not always in a standardized format, and the market for the information may be underdeveloped. Rather, information should be easily accessible, well organized, intuitive to use and understand and data sets should be complete. In order to realize the full potential of open data – for both citizens and entrepreneurs (and to minimize the burden on public services), we think it is important to:
– Compromise for short-term publication, but focus on rigorous data hygiene standards for data currently being collected: The data is only as good as its quality. Inaccurate datasets can be at best misleading and at worst, harmful. Data quality is central to the open data agenda – but we agree that the cost of completely “clean” data may be prohibitive, and therefore a compromise needs to be found between quality, cost and timeliness.
– Reduce, reuse, recycle: the burden on public service providers can be minimized by prioritizing the publication of data sets that are already being collected – for example management information about service quality and outcomes rather than collecting new data sets. Provision of context will also be important for the publication of service performance data so they can fully understand this information and to avoid any misinterpretation of this data by the public.
– Make it valuable for citizens: Open data has been most empowering and successful both in the US and internationally when it has been real for citizens, i.e. where individuals can relate to the information provided and can affect real change. Successive examples show that open data that is provided at a local (e.g. city), neighborhood and even street level can be most effective.
– Standardize for easy comparability and interoperability: standardized datasets in consistent formats will enable citizens / users / developers to compare and analyze the information easily – this is particularly important for organizations like police or local authorities where citizens may want to benchmark organizations against their peers.
– Turn data into intelligence: Many public sector organizations are data-rich, but marshaling the different data sets to create usable, analytical intelligence to provide insights (either for managers, commissioners or the public) can be more challenging. Widespread adoption of analytic and predictive modelling techniques that analyze and visualize information would provide more easily consumable insights for those inside and outside organizations.
– Make targeted investment to turn “data” into a “service”: the Government should make limited and targeted investments to transform a few crucial datasets into “services” by making them easily consumable – good examples are mash-ups like the crime maps. This is particularly important for the citizen-centric open data agenda and will help establish a market and stimulate demand for further publication of datasets.
As part of the prioritization process and as discussed in our overarching perspective, we think the government is right to think about transparency and open data as an evolutionary process that responds to changing needs (from users, developers and public services) over time. Based on our experience we have developed a framework for approaching open data in this way and adapting to user needs as they become more sophisticated and which could be useful in developing an organization’s open data strategy and identifying the steps that should be prioritized.
In some circumstances it may be prohibitively expensive to gather and process a raw dataset – so it may be worth assessing the viability of encouraging the private sector to pay and commercialize the data. Under such circumstances Government should consider making it an obligation for the purchaser to publish the dataset in its raw format. If the Government is able to make targeted investments to turn the data into a service, there may be scope to sell these services to the public and to create a revenue stream. Although beyond the scope of the consultation, the Government may wish to consider charging for access to anonymized datasets if direct commercial gain may result – for example pharmaceutical firms and health data.
The proposals should also apply to non-public sector providers of frontline public services such as charities or private sector businesses for the transparency agenda to be meaningful and accountability to be real. It would also help build the case for mixed provision of public services. We would urge the government to implement measures to ensure consistency of data collected and its presentation to allow comparison across providers.
One approach may be to include data publication as an assessment criterion in tenders to incentivize providers. As a further step, the Government may want to consider the feasibility of making data collection and consistent presentation a contractual obligation. If this were pursued, it is worth considering transition arrangements for current contracts until the contract is completed.
Although the legal and regulatory framework exists, it will be important to give further consideration to guidelines and standards to protect personal data under the open data agenda. These should include clear guidance on what constitutes personal data in this context (for example taking into account the many decision notices under the Freedom of Information Act) and standards for anonymization and pseudonymization (which have great potential in the context of health data). Individuals should be protected on a level playing field and consideration given to how data sets could potentially be merged or reconstituted across government so standards should be consistent and robust.
To improve accountability and public service performance, the publication of existing management information should be prioritized – which should not place an additional burden on organizations at they will be collecting and reporting this information as a matter of course. We do, however, recognize that different levels of management information are currently collected; from operational data, through to service performance measures, and then output and outcome information. We recommend prioritizing publication of outcome, output and service performance measures. Most operational data is unlikely to be meaningful to the public and will need contextualizing to ensure it is not misinterpreted.
An effective way to ensure open data standards are embedded would be to make them a contractual obligation. In addition, we agree with the government’s aim to ensure the public sector is an intelligent, demanding customer. As part of this, it would be helpful to create focus groups that allow suppliers and other stakeholders to contribute to open data standards in a productive way.