What the National Data Library should look like

The Labour government had a National Data Library as a manifesto commitment. But what should this look like in practice? Wellcome and the Economic and Social Research Council recently set up a Technical White Paper Challenge, seeking views on exactly that.

We submitted a paper called A National Data Library: the Modular Approach, which was picked by them for publication.

Seb, Pete and I wrote it to be readable, so the tone is deliberately less formal than some of our other papers. This isn’t a formal scientific study: it’s a practical view on how the state can use all its enormously rich data to drive research, innovation, and better services for citizens.

The paper is in three sections.

Firstly, we describe the current state of play: what’s going well and badly in national data infrastructure. In short, we make one case: big “black boxes” full of all the data about all the citizens - run by one big organisation behind closed doors - is a model that has now been shown to fail.

Next, we set out a technical vision for more productive and more secure data infrastructure. We propose a model of single purpose teams, delivering single purpose services that are replaceable, to maximise accountability and delivery.

We also propose a “take only what you need” model for data movement: data is prepared in situ, by expert teams and tools, and only the minimum amount of data needed for each analysis moves around the system. We give detailed worked examples of how this would work.

Lastly, we set out how to drive a culture of delivery for data and digital in government. That kind of content can often deteriorate into waffle, so we have kept it very practical: how to fail quickly and informatively; how to build capability fast; how to sequence work, from small wins to automated services; and more.

We think the National Data Library should be the central hub that stitches together a network of tightly focused component parts, that each do one job well. We think that’s better for delivery, better for accountability, better for scaling, better for resilience, and better value for money.

Others might disagree. But our model has one big advantage: unlike the approach of building single giant monoliths - whether inside government or through external procurement - our approach hasn’t failed, and it can be tested quickly. As we describe!

➡️ You can read the paper, and download a PDF copy, on zenodo.org.