Step 6a - Data Over Time - Run Charts
In QI data collection is continuous. It is always going on in the background, regardless of what else we are doing in the project. This is one aspect where QI differs from an audit cycle. In an audit cycle a data set is collected before an intervention, and then a further set after. However having only 2 data points makes it very difficult to evaluate the impact of our changes to practice. For example if things appear to go up after the intervention, how do we know it’s down to the change, and not some other unknown factor? How do we know the first data point wasn’t unusually low (and most of the time higher), or the second unusually high (and most of the time lower)?
You can hopefully see why we need the data collection to be continuous, it gives a truer picture of the performance of the system we are looking to improve. This means you should define a time period between data points and always collect at these set points. One data point per week is usually a good target, and gives you nearly 20 data points (which is ideal) over a 4 month period. Your project may allow daily points, or fortnightly or monthly; and this is also fine. It depends on what is available to you. Collecting this much data may seem daunting, but don’t worry – each data set can be very small and we’ll tell you how you actually go about getting this easily in the next step (step 6.2).
Run Charts
What multiple data points gives is a Time Series, or ‘data over time’. This is plotted as a type of line graph, known as a Run Chart.
Figure 6.1, Example of a Run Chart. In this project, one data point was collected each week for just over 4 months. Baseline data was collected in the first 4 weeks, before any changes to practice had been made. The red line represents the median - it is explained when to 'shift' it below.
A run chart is a line graph where time is along the X-axis, and whatever you are measuring is up the Y-axis. What exactly are we measuring? – it’s whatever constitutes the ‘M’ part of your SMART aim.
Say your SMART aim is “For 90% of patients at your GP practice newly diagnosed with vitamin B12 deficiency to be investigated for pernicious anaemia, by December 2018” – the metric measured up the Y-axis would be ‘the % of patients newly diagnosed with vit B12 deficiency to be investigated for pernicious anaemia”.
In another example, say your SMART aim is “To reduce the number of inpatient falls on ward C4 (medicine for older people) to below 1 per week, by 31st October 2018” – you would measure “the number of inpatient falls on ward C4” over time with weekly data points.
Baseline Data
It is important to get some baseline data, before making any actual changes to practice. In a way this is common sense; if you are looking to improve you need to know where you are starting from. In an ideal world you would get at least 6 data points, although this is not always possible. However you definitely need more than one, to avoid the problems found in data used in audit cycles as we’ve described above.
Plotting the Median
Once you have your baseline you plot the median of those points. The median is used, rather than the mean (average) for 2 reasons:
– if you have any extremely unusual data points it prevent them from skewing the data
– The fact that the definition of the median is that ‘50% of points are above the line and 50% below’, allows us to statistically determine when things have improved/worsened (called a ‘shift’, which we’ll cover below)
As you undertake your project, remember you are always collecting data. You should add each new data point on the end of the chart as you go. Keep the median line at the same value for now.
It is important to realise that the individual data points are not really representing the true current performance of your measure, the median line is. There will be day-by-day or week-by-week fluctuations in the data points, but this is the normal ebb and flow of any system. For example, the A&E 4-hr target won’t always be exactly 87% every day. It may be 88% as a median, but Monday might be 84%, Tuesday 92%, Wednesday 89%, Thursday 86% etc. You can see that to say there was a meaningful improvement from Monday to Tuesday is nonsense, it is just the normal fluctuation of the system. In QI this is called ‘normal variation’. What we are looking for is to bring about an improvement in the data that is more significant than this ‘normal variation’ and we have rule that allows us to do this, called a ‘shift’.
Shifts
In research it is convention to determine statistical significance by looking for a p-value <0.05, which is analogous to a confidence interval of 95%. To see if any statistically significant changes have occurred in our data on the run chart, we need to look for patterns which have a <5% likelihood of occurring simply by chance if nothing has really changed at all. To do this we use the property of the median described above, that if there is really no significant change 50% of points will be above the median line and 50% below. Using this we are looking for a ‘shift’, which is 6 consecutive points either all above or all below the existing median.
Once you see 6 consecutive points either all above or all below the existing median, you ‘shift’ the median line. Take the median of these 6 points and starting from the first of these points, plot the median line as your recalculated value forwards in time.
Take another look at the chart near the top of the page. You can hopefully now see why the median shifted up at week 6, it’s because weeks 6 – 11 were all consecutive points above where the median had previously been (from the baseline). In a similar way it shifted down again at week 14, as points 14 – 19 were all consecutively below where the median was at.
This is the proof that an improvement has been made and is what you are looking for in your project. Whatever the ‘M’ part of your aim is, is where you are hoping to ‘shift’ your median to. If your SMART aim is to increase the percentage of patients on the acute medical ward to get VTE risk assessments within the first 24 hours of admission to 90% in 4 months – then you are looking to shift the median to above 90%, not just get a single data point there.
We have made some videos which explain run charts with animations and we highly recommend you watch them, as run charts are a concept best understood by seeing them in this way:
The statistics behind requiring 6 consecutive points fro a shift is determined as follows – the likelihood of any one point being above the median is 50% (or 0.5), of 2 consecutive points being above the median is 0.52 (=0.25), 3 consecutive points is 0.53 (=0.125), 4 is 0.54 (~0.06) and 5 is 0.55 (~0.03). But because a shift can go both up (improvement) or down (worsening) the real likelihood of 5 consecutive points on either side of the line is 2×0.03=0.06 (this is the equivalent to a 2-tailed hypothesis test in statistics). Therefore to be significant we need 6 consecutive points on either side of the median as: 0.56+0.56=0.015+0.015=0.03 (this being less than 0.05). See the video 2 below for a visual explanation.