We met with Clare Moriarty on Friday 6 January to run through the progress with the Data Programme. This is a short summary of what we discussed.
Defra has suffered a similar problem to all large organisations: data has not historically been treated as one of its most valued assets, leading to datasets being insufficiently maintained and reducing sharing across the group. This reinforces silos that present barriers to optimal delivery of services.
Data, once collected, has not been used as much as it might be to drive decision making and improve services. There has also been the possibility that inaccurate or inappropriate assumptions might be drawn from poorly maintained data. To address this, the Data Programme is creating a data framework that sets outs out how data should be managed across the Department: reducing duplication and divergence; helping users find what they need when they need it; and providing the necessary guidance on how it should be appropriately used.
Reducing duplication of data is a key principle, since managing duplicate copies both creates unnecessary expense and potentially leads to divergent versions of the data. Avoiding this happening is important, particularly when responding to major incidents. During a flood event, the Environment Agency has historically duplicated data across teams with specific remits, but the development of the ‘Flood cube’ will establish a single central source that can be queried both internally and by key external partners – creating a single definitive, accurate source that can be used to plan response and design sensible mitigation strategies.
In a similar vein, the Data Programme is looking at how geographic data from across Defra can be brought together. The Department holds a huge volume of geospatial datasets, and pulling these into a single location will help integrate with other sources and make them available. In particular, as field workers are equipped with modern tools such as tablets there is a need to allow them to directly access the data that they need, wherever they are.
Defra has been very progressive in exposing a huge proportion of its data externally. However, there remain numerous internal assets that have not been shared due to commercial issues or sensitivity around the potentially ‘personal’ nature of the data. An example area here is animal disease and particularly Bovine TB, where Defra are trying to improve access to the disease history on individual farms. This disease alone costs government around £100 million per annum with additional cost borne by the industry itself. Building on earlier work by the APHA, the data programme is running a Proof-of-Concept data access trial, looking at how the data holdings held by Defra can be presented to farmers and vets to help tackle the spread of the disease.
The launch of the Copernicus Satellite Sentinels and particularly the free availability of data from these is opening up a huge range of options for Defra doing things differently. Data from satellites is extremely powerful for detecting subtle changes in the environment, helping to target inspection and monitoring visits to sites that indicate change, reducing costs in these areas. A key issue to tackle to enable this change is cleaning the raw satellite data; the original signal is distorted as it passes through the atmosphere. The Data programme is helping JNCC deliver its contribution to the EOCoE by building automated mechanisms to clean the data once centrally. This avoids individual organisations having to do the processing themselves hence saving costs but also providing easy access to this will immediately unleash a huge range of potential innovation across Defra.
By integrating various data sources, including satellite imagery, we can gain new insight. To demonstrate this the Data Programme is working with the Forestry Commission and the Earth Observation Centre of Excellence to integrate data on the location of forests where logging has been licensed, along with satellite data, to identify locations where trees have been felled without the necessary permission and potentially provide alerts almost in real time.
The goal is to use this sort of mechanism much more broadly across Defra to spot a range of other environmental problems and issues. The potential has already been demonstrated (see BBC Countryfile - iPlayer link, 7 min, 21 sec in) by the Environment Agency identifying where bare earth occurs on sloping fields as this is likely to lead to high levels of soil erosion and resultant river pollution. Again being able to spot the problems early allows very targeted field inspections and potential early intervention to mitigate the potential consequences.
Defra has already been lauded for its successes in publishing open data last year, but the real challenge is making better use of it. This year's work starts to tackle this, by improving data quality, building the data framework and adopting novel approaches to deliver excellent outcomes.
1 comment
Comment by Cee posted on
Lovely to hear how all the work is getting on. Glad to hear Flood Cube is creeping closer to reality!