Openness and Transparency: why is openness and transparency important for good data analysis and how does OpenPrescribing embody it?

At the Bennett Institute we value openness and transparency as we believe open discussion of ideas and methods are the key ingredients for high quality data analysis. We think it is unhelpful that so much NHS data analysis is outsourced, or done behind closed doors. This blog sets out the steps we take with OpenPrescribing.net to ensure all our analysis and methodology is freely available for inspection, review, and importantly re-use, by anyone. We hope other teams will adopt these approaches so we can all collaborate to optimise analysis, generate insights and improve care for patients in the NHS.

How do we share our methodology?

All our code is shared openly and is accessible on our Bennett Institute GitHub; for example all of our measures are here. For those unfamiliar with GitHub, it is a website that makes it easier for everyone to; share code, work together, solve challenging problems and create useful and important technologies like OpenPrescribing. The underlying code, on a site like OpenPrescribing, is the best method for assessing our methodology, as at any given time anyone can see exactly how the site is retrieving and displaying data for our users. To repeat, all our code is freely available for inspection, review, and importantly re-use, by anyone.

How do we build our prescribing measures?

For every single measure on OpenPrescribing we describe the numerator and denominator in the chart title, and when you click into the chart you can see the breakdown of the numerator items that have been prescribed in the practice or CCG you are reviewing (Islington NSAID example). We also describe for every measure “why it matters” to assist prescribers to reflect on their practice and makes changes if appropriate.

Additionally, and for the avoidance of any doubt, all our code that accurately describes a numerator and/or denominator is publically and freely available on our GitHub for anyone to review. For example, you can see our NSAID measure here. If there are specific measures anyone feels are poorly described please get in touch at feedback@openprescribing.net and we will review it and improve if we can.

How do we choose what to display?

We visualise and display data on OpenPrescribing in lots of different ways including bar charts, maps, medians and deciles. Our small team of academics, researchers, clinicians and software engineers are very thoughtful about everything we put on the site and how we present it, as we know a lot of our users (over 110 000 in last 12 months) use it to inform decisions about medicines that are prescribed to patients. If people find it useful we will blog more about the decisions we make and if interested you can read Alex’s blog about why we use deciles on OpenPrescribing.

Our methodology more broadly

In addition to all our code being open and accessible, our description of measures and our extensive FAQ page, we also publish our methodology in academic journals for discrete tools on the site. A good example is our innovative price-per-unit tool (Bexley CCG example) that we developed for every practice and CCG in England. The team published all their methods in BMJ Open here, as well as all the code and results here, and subsequently published follow-up measuring the impact of the tool in the Journal of Medical Internet Research, where we estimated the tool had saved the NHS £243 000 at a practice level and £1.47 million at a CCG level in its first four months of operation.

Other people’s methodology

Sometimes we come across NHS data from organisations that is not well documented or described and we have to invest a lot of time understanding it. In these instances we make all our research freely available on our GitHub and we also write blogs about things that we believe others will find useful. A good example of this is our blog on BNF codes that are used throughout the NHS. We invested time in understanding huge swathes of technical data on BNF codes and once we understood them we wrote an accessible blog so everyone can share and learn from our experience.

Now when you google “what is a BNF code”, our blog is the number one result on google (yes I have switched personalised search off!). Although this is great, it is a little bit frustrating for us as there are large teams (with large budgets) across the NHS that really should be documenting and sharing their work and making it accessible to everyone for open discussion and further collaborative development of ideas. So much NHS data analysis is outsourced, or done behind closed doors and it is left to our tiny team to convey a lot of technical nerdy information but hardly anybody will fund our work.

Could we be more open and transparent?

Yes. Now, having said all of the above we know we can improve and we also know that sometimes the openly accessible code can be inaccessible if you don’t have some understanding of Python and SQL (if you don’t, then come on the courses we will shortly be running!). Over the coming months we plan to write Jupyter notebooks (see the gabapentinoids measure example) for every single measure on the site in addition to the detailed text descriptions on the website and openly accessible code on GitHub. We are also increasing the commenting on our GitHub measures code, making it easier to understand and check. Finally we are planning on increasing robustness by making our quality assurance process completely open, including review dates and BNF code complexity, which will be embedded as part of the Github code. We’ll write more about it when it’s completed. You can see our initial experiments with it here.

We also plan to write some more blogs on our design choices we have made for OpenPrescribing and our other sites like the EU Trials Tracker. If there is a particular topic you are interested in or an area of our site that isn’t well documented please tell us and we will write about it. As you know we are a tiny (but mighty) team and doing all this takes time and money!

Open discussion of ideas, methods and flaws can only help improve the tools available, not only from us, but from everyone to ultimately improve care for patients. Please get in touch at bennett@phc.ox.ac.uk if you would like to know more.