Over the last 18 months I've been building a new Data and Information Access team in Defra’s Digital Data & Technology Services (DDTS) function. The team brings together data sharing and library functions and has three broad purposes, to:
- ensure that quality data and information assets are available to Defra group users
- ensure that collections, catalogues and platforms are well maintained and managed
- support the publication of Defra data and research
Maintaining catalogues of good quality metadata is critical to them all. But over the last few months I've come to realise I have a love/hate relationship with metadata and the hate is winning. But there might be some simple principles, which, if applied, could make our metadata better and our data easier to find and reuse.
Why I love metadata
I love metadata because it helps people find and share our data and information and to understand if it meets their needs.
I love metadata because it helps make services work for users. For example, metadata gives users of our Flood Information Service confidence that they are looking at up-to-date information.
I also love metadata because I am a fan of a good standard (I realise this is quite geeky). There are lots of good standards in the metadata domain and these standards drive the sharing and reuse of data.
Why I hate metadata
It feels like there are too many standards, people seem to focus on complying with a standard and not the quality of the content, and everyone interprets standards differently.
Metadata is often disconnected from data, it is created in laborious, manual ways and often we aren’t great at keeping it up to date.
We have many ‘metadata catalogues’, 16 at the last count in Defra group I think. They are all different. Some assets are recorded in multiple catalogues, some are not recorded at all. But worst of all I don’t think the catalogues we have got make it easier for users to find the data and information they need!
The metadata problem is growing as we increasingly move towards greater use of real time data, for example from internet connected devices, telemetry systems and Earth Observation sensors, and as we move to online services like Office 365 which treat our information (documents, emails, tasks, and calls/messages) more like data.
It doesn’t feel like our current approaches to metadata are sustainable or working well.
The good news is that lots of work is already happening
People across government have realised there is a problem. The drive to increase the sharing of data across government means we need to be better at metadata.
There is already a lot happening in the government metadata space:
- Data Standards Authority has set a standard for metadata when sharing data across government and has started to clarify what standards should be used when
- Geospatial Commission are working hard to improve the guidance on metadata for Geospatial data through their Geospatial Data Discoverability projects and have made good recommendations on geospatial
- we are increasingly understanding the importance of making metadata ‘search engine ready’ using things like schema.org standards
- National Data Strategy Framework talks about the need to modernise the way we manage and share data across government to generate significant efficiency savings and improve services - better metadata will be critical to success
Learning to love metadata again
I’ve been thinking about how I could start to love metadata again.
It feels like a lot of the current work is focusing on what metadata we should capture and not how we should do it. Focusing on Defra’s metadata, catalogues and related processes, a lot of the pain is around how we ‘do’ metadata. We could do metadata better, resulting in improved content, quality and currency.
So I want to start a conversation about how we do metadata.
I've developed an alpha vision for metadata in Defra group and a set of principles. I'm sharing these to test them and I’d love your feedback - feel free to contribute to this collaborative document or by commenting below. If you’d like to talk to me about this work let me know via Twitter if you’d be interested joining.
Once we have an agreed vision and principles we can then start to think about the guidance, processes, and patterns that would be needed to apply them.
If this work develops into something useful, I will share the outcomes in a future post, if it doesn’t, I will try and explain why.