{phsmethods}: an R package for Public Health Scotland

Background

On Wednesday 1st April 2020, Public Health Scotland (PHS) came into being. It was formed as a result of a three-way merger between NHS Health Scotland and the Information Services Division (ISD) and Health Protection Scotland (HPS) sections of Public Health and Intelligence (PHI) (which was in itself a strategic business unit of NHS National Services Scotland (NSS)). It’s fewer acronyms to remember at least.

The Transforming Publishing Programme (TPP) was devised in 2017 in an attempt to modernise the way in which ISD produced and presented its statistics. Traditionally, work within ISD had been undertaken using proprietary software such as Excel and SPSS, with output presented predominantly in the form of PDF reports. It required a great deal of manual formatting, caused an even greater deal of frustration for analysts with programming skills, and resulted in more copy and paste errors than anyone would care to admit.

Over time, ISD gradually (and at times begrudgingly) came to embrace open source software – predominantly R and, to a lesser extent, Python. TPP were at the forefront of much of this: creating style guides; working with producers of official and national statistics to convert PDF reports into Shiny applications; producing R packages and Reproducible Analytical Pipelines; and introducing version control, among other things. Now, with the move to PHS complete, TPP’s purview has broadened: not only to further the adoption of R, but to ensure it’s adopted in a consistent manner by the hundreds of analysts PHS employs.

Introducing {phsmethods}

Analysts working across a multitude of teams within PHS have to perform many of the same, repetitive tasks, not all of which are catered for by existing R packages: assigning dates to financial years in YYYY/YY format (for example 2016/17); formatting improperly recorded postcodes; returning quarters in plain English (e.g January to March 2020 or Jan-Mar 2020) rather than the slightly restrictive formats offered by lubridate::quarter() and zoo::yearqtr(). The list is endless, and every analyst has their own workaround, which quickly becomes a problem. Time is wasted by multiple people devising an alternative way of doing the same thing – and how can anyone be sure that everyone’s method actually does the same thing?

The {phsmethods} package was conceived to address (some of) these concerns. At the time of writing, it contains eleven functions to facilitate some of the more laborious tasks analysts regularly face, with more in the works. None deal with any statistical methodology, nor are they likely to provoke any controversy or consternation over their methods of implementation; they are simple functions designed to make routine data manipulation easier.

{phsmethods} isn’t on CRAN. Perhaps it will be at some point, but there hasn’t been any need to make it available more widely thus far. {phsmethods} does, however, come with many of the features one would expect from a CRAN package: function documentation; unit tests; continuous integration; code coverage; and releases. No hex sticker yet, but maybe one day.

Using {phsmethods}

Practical examples featuring all of {phsmethods}’ functions are available in the package’s README and no one will make it to the end of this blogpost if they’re all regurgitated here. But hopefully one demonstration is okay. Consider the following fictitious dataset containing postcodes:

df <- tibble::tribble(
  ~patient_id, ~postcode,
  1,           "G26QE",
  2,           "KA8 9NB",
  3,           "PA152TY ",
  4,           "G 4 2 9 B A",
  5,           "g207al",
  6,           "Dg98bS",
  7,           "DD37J    y",
  8,           "make",
  9,           "tiny",
  10,          "changes"
)
df

# A tibble: 10 × 2
   patient_id postcode     
        <dbl> <chr>        
 1          1 "G26QE"      
 2          2 "KA8 9NB"    
 3          3 "PA152TY "   
 4          4 "G 4 2 9 B A"
 5          5 "g207al"     
 6          6 "Dg98bS"     
 7          7 "DD37J    y" 
 8          8 "make"       
 9          9 "tiny"       
10         10 "changes"    

This is a problem that analysts across the NHS and beyond face regularly: a dataset containing postcodes arrives, but the postcodes are recorded inconsistently. Some have spaces; some don’t. Some are all upper case; some are all lower case; some are a combination of both. And some aren’t real postcodes. Often this dataset is to be joined with a larger postcode directory file to obtain some other piece of information (such as a measure of deprivation). Joining these datasets tends not to be a trivial task; sometimes a {fuzzyjoin} may suffice, but often it requires an arduous process of formatting the postcode variable in one dataset (or both datasets) until they look the same, before combining using a {dplyr} join.

In PHS, postcodes typically follow one of two formats: pc7 format (all values have a string length of seven, with zero, one or two spaces before the last three digits as necessary); or pc8 format (all values have one space before the last three digits, regardless of the total number of digits). phsmethods::postcode() is designed to format any valid postcode, regardless of how it was originally recorded, into either of these formats. Consider the earlier dataset:

library(tidyverse)
library(phsmethods)

df %>%
  mutate(postcode = phsmethods::format_postcode(postcode, format = "pc8"))

# A tibble: 10 × 2
   patient_id postcode
        <dbl> <chr>   
 1          1 G2 6QE  
 2          2 KA8 9NB 
 3          3 PA15 2TY
 4          4 G42 9BA 
 5          5 G20 7AL 
 6          6 DG9 8BS 
 7          7 DD3 7JY 
 8          8 <NA>    
 9          9 <NA>    
10         10 <NA>    

df %>%
  # The format is pc7 by default
  dplyr::mutate(postcode = phsmethods::format_postcode(postcode)) %>%
  dplyr::pull(postcode) %>%
  nchar()

 [1]  7  7  7  7  7  7  7 NA NA NA

Updated package

The following paragraphs refer to the original function postcode() which has since been deprecated and warning messages suggest using format_postcode().

In order to ensure this blog renders the newer function has been used but the corresponding blog text has not been changed.

In hindsight, maybe it should have been called something other than postcode() to avoid the whole postcode = postcode(postcode) bit, but everyone makes mistakes when it comes to naming functions.

The NHS-R package {NHSRpostcodetools} does check for whether a postcode exists but is currently only for England.

phsmethods::postcode() isn’t designed to check whether a postcode actually exists; it just checks whether the input provided is a sequence of letters and numbers in the right order and of the right length to theoretically be one and, if so, formats it appropriately. If not, it returns an NA. Handily, phsmethods::postcode() comes with some warning messages to explain how many values were recorded as NA, why they were recorded as NA, and what happens to lowercase letters:

warnings <- testthat::capture_warnings(df %>% dplyr::mutate(postcode = phsmethods::postcode(postcode)))

writeLines(warnings[1])

## 3 non-NA input values do not adhere to the standard UK postcode format (with or without spaces) and will be coded as NA. The standard format is:
## • 1 or 2 letters, followed by
## • 1 number, followed by
## • 1 optional letter or number, followed by
## • 1 number, followed by
## • 2 letters

writeLines(warnings[2])

## Lower case letters in any input value(s) adhering to the standard UK postcode format will be converted to upper case

Next steps

The seven functions which comprised the 0.1.0 release of {phsmethods} were all written by members of TPP. However, the package is not intended to be a vanity exercise for the team. Those initial functions were designed to get the package off the ground, but now contributions are open to be made by anyone in PHS. With that in mind, the 0.2.0 release contains four functions written by PHS analysts not part of TPP.

Unlike many R packages, {phsmethods} will, at any given time, have two or three maintainers, each with an equitable say in the direction the package takes. When one maintainer moves on or gets fed up, someone else will be sourced from the pool of analysts, and the show will go on. In theory at least.

There are moderately extensive guidelines for anyone who wishes to contribute to {phsmethods} to follow. Proposed contributions should be submitted as GitHub issues for the maintainers to consider and discuss. If approved, the contributor makes a branch and submits a pull request for one or more of the maintainers to review. The hope is that, by creating an issue prior to making a contribution, no one will devote time and energy to something which can’t be accepted, and that no duplication of effort will occur in the case of multiple people having the same idea.

It’s not expected that everyone who wishes to contribute to {phsmethods} will understand, or have experience of, all of the nuances of package development. Many won’t have used version control; most won’t have written unit tests or used continuous integration before. That’s okay though. The only requirement for someone to contribute code to {phsmethods} is that they should know how to write an R function. The package maintainers are there to help them learn the rest.

Hopefully {phsmethods} will provide a safe incubator for analysts who wish to develop their skills in writing, documenting and testing R functions prior to making their own open source contributions elsewhere. And hopefully the knock-on effect of more analysts developing these skills will be improved coding standards permeating throughout PHS. If not, it’ll at least look good having a package pinned on my GitHub profile.

This blog has been formatted to remove Latin Abbreviations and has been edited for NHS-R Style and to ensure running of code in Quarto.

The original comments form the WordPress site:

David Henderson

1 May 2020

Please, please, please get this on CRAN. That way it’ll be available in the National safe haven for linked data projects. Some of these functions would be super handy.

David McAllister

2 May 2020

The transforming publishing team have done amazing work at both the “back-end” and front-end of modernising work at ISD. This has great potential to lead to even more in the future for Public Health Scotland in terms of reliability, efficiency, openness and quality.

Peter Hall

29 December 2020

I too would love to see this on the CRAN - can’t underestimate the value of doing this.