OBSERVATIONAL STUDIES LIMITATIONS AND STRENGTHS |
Robert Wolfe, Ann Arbor, USA
|
Chair:
Friedrich K. Port, Ann Arbor, USA
|
Carmine Zoccali, Reggio Calabria, Italy |
|
Prof R. Wolfe
Scientific Registry of Transplant Recipients Arbor Research Collaborative for Health Ann Arbor, MI, USA |
Slide 1
Thank you very much Doctor Port and the organisers of the session. I’m going to go over some of the different tools that can be used with observational studies to help us understand the strength of the evidence that maybe present in those observational studies. I will say that not all observational studies are made equal.
Slide 2
What we need to do as scientists, as clinicians is evaluate the evidence that’s available to us. I’m going to talk about 3 tools that are extremely important for observational studies most important of which is statistical adjustment. I’ll give an example of that. I’m also going to talk about analysis by facility rather than by patient as a way to avoid treatment by indication bias and I’m also going to go over a technique which we’ve labelled here the delta-delta analysis and it looks at changes over time as a way to further increase the evaluation of evidence from an observational study.
Slide 3
Now what I want you to do is to step back in time to 1915. I’m going to talk about an example which has nothing to do with kidney disease but that puts us all on an equal footing. I’m a biostatistician, so here I get to talk about numbers. I’m going to talk about the incidence of Down’s syndrome per 10.000 live births. These are data from Michigan in the United States, an entire population. The reason 1915 is important is because at that time there was very little selection and decision making about live births we suspect compared to now. What I’ve shown here are rates based upon an entire state in the United States in 1 year and we’ve related the rate of Down’s syndrome in the right hand column, the rate per 10.000 live births to mother’s age. You can see we have approximately 5 year intervals of age 20-24, 25-29 and so on until 40 plus and what you see is a very clear gradient, a very strong association between the rate of Down’s syndrome and mother’s age. Many people would look at that and say, ‘Ah ah older mother’s age causes Down syndrome’. Well, keep this slide in mind.
Slide 4
Here’s another slide from exactly the same population based upon the same data. We’re again looking at rates of Down’s syndrome and here we’re relating it to the parity of the birth that is whether this was the first born or the second born child to the mother and this is really 5 plus. This is all births after the 5th and what we see is a very strong gradient. The rate of Down’s syndrome is increasing with birth order. So some people might look at this and say, ‘Ah ah clearly the birth order causes Down’s syndrome and later birth order leads to a higher rate of Down’s syndrome’.
Slide 5
The only way we can sort this out, this conundrum of an apparent association with two different factors is to look at them simultaneously. One of the major concepts of science in general is when you’re looking at the effect of one factor to make all else equal, we have many tools for making all else equal, one of the simplest is a simple table. By all else equal what I’m proposing is if we want to look at the effect of mother’s age which is shown in the row columns here, in the rows, we would like to be able to look at what happens as mother’s age increases while birth order is held constant or held the same so we can chose a particular value. We’ve calculated here birth order by mother’s age and in the body of the table we have the incidence rate of Down’s syndrome and it says adjusted for age and parity that just means accounting for birth order and for mother’s age simultaneously.
When you read the language, when adjusted for, it was adjusted for that’s all they’re talking about, it just means looking within columns of a table and holding the thing that is being adjusted for, in this case birth order, holding it constant and looking down a column in which it is held constant to see the effect of changing another factor holding one constant. So we’ve adjusted for birth order and looking at the association between Down’s syndrome and mother’s age and it is there.We say that Down’s syndrome is independently associated with mother’s age, independent of birth order. Let’s look at the other direction.
What happens if we adjust for mother’s age and look for the association with birth order? That would correspond to looking across a row here. If we look at mothers who are 30-34 years old at the time of birth and look at the incidence of Down’s syndrome, it’s relatively flat right here it’s 9 or 8-10. Similarly, if we look at mothers who are 25-29 and see whether or not the incidence of Down’s syndrome increases or declines with the birth order, the answer is no it doesn’t. If you look across any row there, you’ll see virtually no association between the incidence of Down’s syndrome and birth order, if mother’s age is held constant. So here we’re saying that when adjusted for mother’s age, there is no relationship between birth order and Down’s syndrome. This is really all that adjustment does. And you can see how natural it is, how appropriate it is. It is exactly what we use in science, in the laboratory when you try and control the environment and make all else equal when we’re adjusting one factor at a time.
You can ask ‘Now how in the world did this happen? How was it we saw in the first table there was a strong association between birth order and Down’s syndrome. How can it possibly be constant across every row? And the answer is down in the bottom here you actually see what happens when you collapse all the numbers above into the rate for each birth order column. You can see at 6 for first borns through to 17 for fifth borns. That’s the same table we looked at in the very second slide. The reason that’s happening is because when we compare this 6 to this 17, we’re not just comparing first borns to fifth borns, we’re comparing first borns to young mothers compared to fifth borns to older mother’s because there is an association between the order of birth, the number of children that a mother has had and her age. So women who have had five or more children tend to be older than women who are having their first child. So the interpretation of this number is exactly correct, it is true but the incidence rate of that amount is much lower among first borns than it is among fifth borns but it’s not because of the number of siblings, prior siblings it has to do with the age of the mother. So many people would look at this table and say yes, I knew all of that, we’re done.
Slide 6
Here’s the language I just went over to say that mother’s age is associated with Down’s syndrome even when adjusted for birth order but not by ---. We can also say that comfortably and know we understand the problem completely but we must never stop, we must always be vigilant with observational studies because we always have to look for the potential for other confounders.
What else might be causing the association that we see between mother’s age and Down’s syndrome? Well, today there have been many studies. We do know the answer, it has to do with genetic effects in the DNA but at this time they didn’t have a clue about that.
Slide 7
Let me go on to something closer to us right here. This is a table that shows by year from 1985-2002 it shows mortality rates in the United States amongst dialysis patients. It’s shown on a vertical scale with something called ratio, a mortality ratio and these numbers correspond to higher numbers greater than one means greater than the average, numbers lower than one means less than the average and the interpretation actually is that this death rate here is 11% higher than the average. This death rate because it’s 1.11 that the 1.1 here makes the 11% higher. The 0.93 if you subtract that from 1 that means 7% lower than the average. What you see is a trend going down in mortality. Now, actually that’s not the truth, that’s not the pattern of death rates. This is the pattern of adjusted death rates. Why did I do that? Because over this period of time we’ve been treating older and older patients in the United States, more and more diabetic patients in the United States. It’s extremely important to account for that. If I hadn’t done that, in fact the death rates might even be increasing because we know that death rates are higher amongst older patients. So what we did was we adjusted for the same concept that I used in the 2 way table before. So that as we compare the death rates across these different years, we’re comparing it all else equal with regard to age, with regard to diabetes, we are looking at a similar mix and that makes it very interpretable to understand what’s really going on with the time effect. We can pool out the effect of different kinds of patients and look at just the effect of whatever else is changing beyond there and we do see a decline in death rates.
Slide 8
Now here I’d like to look at part of the reason that might of happened in the United States. This is a chart. I’d like you first to focus upon the gold bars here. What these reflect are different groups of dialysis facilities in the United States. They are distinguished from each other with regards to the fraction of patients at those dialysis facilities with dose of dialysis as measured by URR greater than 65%, that’s approximately the KDOQI guideline with regard to URR. On the left we have facilities where less than 78% of their patients have achieved this guideline and on the right 95% or more have achieved this guideline and actually these are quintiles, a fifth of the facilities are in each group you can see there are approximately the same number of facilities in each group 570-590 and what we show on the vertical axis is the mortality amongst those facilities. So what happens to patients who are at facilities where relatively a lower fraction of their patients are achieving the dose of dialysis guideline? Their mortality is 15% higher than the norm here. At the facilities where they have a very high fraction achieving the guideline the mortality is 7% lower than the norm. All together that’s about a 20% difference in mortality, 1 in 5 deaths approximately.
So what are the possible explanations for this? One is maybe dose of dialysis is good to achieve. Another is maybe my adjustment didn’t account for everything. Maybe there are some things I couldn’t measure and I didn’t account for. In fact, these are just sicker patients over here in these facilities, facilities are stuck with them there’s nothing they can do about them, it’s hard to deliver a high dose of dialysis to them and they’re going to die anyway so why worry?
Well, there was a grand experiment that went on in the United States between 1999-2002. These facilities did not just continue with those practices untouched. In fact, there was a massive shift towards compliance with the KDOQI guidelines and what we have in the turquoise bars are the same data for 2002 from the same facilities, these are the same facilities followed through this 3 year period. What you see is that there are many more facilities almost twice as many as there were in the previous era achieving many more of their patients with a higher URR. Nearly twice as many facilities are in this highest category here. Here we have less than half as many facilities. Many facilities moved to a better practice with regard to achieving the KDOQI guideline.Slide 9
What happened to their patients? They took their patients with them although they are different patients to a large extent in a different era but it may well be the same kind of patient population and we have adjusted so that these are adjusted to be similar kinds of patients before and after. These patients that were supposedly before encouragable, difficult to treat and who are going to die anyway when their facilities changed the practice and moved somewhere to the right, magically the patients stopped dying.
Slide 10
Now this is an observational study. At the same time we’ve tried to make as much else equal as we can, we have adjusted for the patient characteristics, we’ve kept them at the same facility and actually, we’ve done that more formally in another table. This is the same kind of chart which you can see with regard to anaemia management and here I’ve put them together and we’ve classified facilities according to the amount of improvement they’ve had with regard to URR ranging in thirtiles, these are a third of the facilities in each column with very little improvement with regard to the dose of dialysis guideline up to a substantial improvement with regard to that. Facilities are characterised with regard to that. They are also characterised with regard to the amount of improvement they had with achieving the anaemia management. The body of the table shows the change in mortality, the improvement in mortality amongst those facilities that have achieved different levels of improvement with regard to these 2 guidelines. You can see that the facilities that have had the greatest improvement both with regard to dose of dialysis and with regard to anaemia management have improved their mortality by 12.4% over this era in the United States while those that changed little with regard to either, actually if anything their mortality got worse by 3%. All together it’s about a 15% differential here. If you look across any row, you’ll see a gradient. If you look down any column, you’ll tend to see a gradient although it’s a bit noisy. With a suggestion that each of these components, practice patterns have independent effects, independent of the other with regard to mortality.
Slide 11
So somehow the facilities that change their practices ended up with changes in mortality. And when I say not all observational studies are made equal, I would suggest that the level of evidence here, at least to me and to many people I have talked to, is much more convincing than the level of evidence from a case series of 15 patients at a single institution which is a good demonstration perhaps but not nearly as convincing in terms of the weight of evidence as we see here where there was an intervention of a type at some facilities. Some facilities truly did change their practice and there was a corresponding change in mortality. Other facilities did not change their practice very much and there was a correspondingly lower change in mortality.
Slide 12
We have put this together with some other practice guidelines including albumin and the use of catheters.
Slide 13
We’ve built up an overall index for all four of those factors simultaneously and we are examining others and there is a gradient in mortality with regard to each of these factors
Slide 14
And when we have looked again at a delta-delta analysis, the change in mortality associated with a change in practice patterns, we see a significant and substantial impact on mortality associated with an overall index in practice patterns.
Slide 15
So in summary, it’s extremely important to make all else equal. How do you do it? The best way by far is randomisation with a randomised controlled clinical trial. It’s guaranteed 99%. Another way is strict entry criteria, you can effectively make all patients the same to begin with. Statistical adjustment. This is by far the most important tool for observational studies. Instrumental variables is another tool. I’ve mentioned this with regard to classifying facilities rather than patients. I didn’t classify patients in these recent charts according to whether or not they achieved the DOQI guideline, I classified the facilities, entire practice patterns achieving the guidelines quite well or achieving them very poorly. Intent-to-treat concepts are related to that in a sense patients at a facility with a low level of achievement of the guideline have a low expectation of achieving that guideline and all the patients in the facility are assigned to that treatment group. The idea of a delta-delta design I think increases the weight of the evidence that a single facility changes their practices and you see some change in the outcome that’s a little bit stronger than just seeing between facility relationships and in some cases we use weighted analyses. I go with these because all of these are tools that we have used in DOPPS and I think all of these are tools that are considered by serious investigators who are carrying out well designed observational studies. Thank you very much.