Slide 1

Now, I will try to focus you on how to assess study quality. In my presentation you will see I will overlap a bit of that of Doctor Jager but I believe this may help you to follow this course.

Slide 2

Now, the doctor-patient relationship is a one-to-one relationship. In this relationship the Doctor spends his empathy, his ability to properly inform, his ability to aliment hope but perhaps the most important things that the Doctor puts in this relationship is knowledge.The Doctor needs various types of knowledge to treat the patients. He needs the etiologic knowledge, infectious disease, genetic disease, metabolic disease, which kind of disease does the patient have? The Doctor should be able to make diagnosis that is to differentiate between different diseases. Prognosis that is to predict the likely cause of the disease, to choose the best treatment for the patients in that particular context and finally, the Doctor should be able to put into effect prevention that is to reduce the chance of the disease occurring or recurring. Now each question requires a different type of study; etiologic, diagnostic, prognostic, therapeutic and preventive studies. We as doctors are demanded that we make the best choice in selecting the studies where upon we base clinical decisions.

Slide 3

Now, in order to help Doctors and study reviewers checking the quality of studies, very long check lists have been prepared and good doctors and good reviewers go through this checklist from the top to the bottom. Now, preparing for you the full check list would be tantalising. I will focus just on so many items you should apply in the different types of studies. Now let’s start with etiologic/harm studies and prognostic studies. These are together because in general the design in these studies is cohort studies or case-control studies. Of course, randomised trials are important to establish a casual role for a given risk factor, for example, angiotensin converting inhibitors as related to renal disease progression tells us that the system is causally implicated in renal disease progression. But randomised trials are inadequate to pick up rare adverse events that is hard. For this reason I will deal with this type of these studies later on. Now, let’s start with cohort studies.

Slide 4

As Kitty Jager has shown you, cohort studies are ideally suited to pick up the effect of exposure because exposure always precedes the event. Smoking, the habit of smoking precedes the development of myocardial infarction. In cohort studies the investigators remain neutral, external to the study, he just observes. In these studies two groups are formed an exposed group and you count the events that happen in the exposed group and the unexposed group and again, you count the events and then you calculate a relative risk. Now what should we check in these studies? We should pay attention to the definition of disease. Myocardial infarction can be defined in various ways by ECG, by biomarkers by advanced techniques, by PET for example, and the sensitivity of these methods is not the same. Second we should pay attention at verifying that exactly the same definition is applied to exposed and to unexposed individuals because if you apply different criteria you end up with wrong study interpretation. Now, we should make sure that exposed and unexposed individuals are comparable. Now in observational studies it may happen that at baseline exposed and unexposed are not comparable and multivariate adjustments are made in order to allow for the baseline difference. Now we should pay attention in reading these studies that multivariate adjustment is correctly made. The follow up should be long enough for the event to develop. If we would like to study the progression of renal disease to ESRD phase, the study cannot last just one month, of course. Second we should pay attention to the loss to follow up because sometimes the loss to follow up may influence the results. In this kind of study we should pay also attention to the plausibility of the hypothesis you are testing, that is you should make sure that the link that you find in a study of this kind is a truly causative link.

Slide 5

In other words, you often need an external type of evidence, experimental evidence, for example, an experiment in rats exposed to smoking condensate.

Slide 6

Now case control studies. Case-control studies are radically different from cohort studies because they start from the outcome, from myocardial infarction and they trace back exposure in the past. You assemble two groups, a group with myocardial infarction and a group without myocardial infarction and then in the past you try to find the individuals who were exposed to the risk factors, for example, for a year, you do the same in the control group, 1 year. Then you calculate the odds ratio of exposure. Now in case-control studies you should pay attention to the case definition because the case definition should be broad enough to ensure that true cases are not missed but yet you should make sure that the case definition is specific enough that only true cases are included. Secondly you should pay attention to the case source, general population, specialist centre. As you know specialist centres cure the most severe cases. So what you pick up with studies done in specialist centres do not reflect what happens in the general population or may not reflect what happens in the general population. You should pay attention if the patients enrolled in the study are prevalent or incident because prevalent studies are less likely to pick up the most severe cases who die early. We should pay attention at checking whether controls are similar in terms of age, sex, social class and area of residence to the cases. An important point is when we assess retrospectively the exposure to make sure that the assessor of the exposure is blinded to the case-control status because if the assessor is not blinded he maybe hypothesis driven. Secondly, if we assess the exposure by interviewing patients, we should make sure that we apply a formal interview, a form identical to case and control in order to avoid the so-called recall bias or rumination bias because patients who are diseased people are more likely to remember exposures. Second, in this kind of study you should pay attention whether or not the authors dealt with the possible confounding factors.

Slide 7

Now, let’s move to diagnostic studies.

Slide 8

Diagnostic studies are usually summarised in a contingency table. On the horizontal line you have the disease, present or absent which is a subject according to a golden standard. Then on the vertical axis we have test positive and test negative. The first square, true positive that is patients who have the disease and are positive at the test. False positives, those where the disease is absent but the test is falsely positive. True negatives, those who are not diseased and have the test negative. Finally, false negatives that are those who are diseased but turned out to be negative at the test. On the basis of this contingency table you can calculate sensitivity, specificity, positive and negative predictive value and positive and negative likelihood ratio. Likelihood ratio is the most useful indicator of test accuracy in clinical practice because a positive likelihood ratio tells you how more likely is the disease when the test is positive. Conversely, it tells you how less likely is the disease when the test is negative.

Slide 9

Now, when you are going to test a new assay, you should first of all ask yourself how golden is the standard against you which you have validated the test because, for example, we know that creatinine clearance is far less reliable than inulin clearance. So if you are going to propose a new test for measuring the GFR, please don’t use creatinine clearance. In this kind of study we should avoid also the bias that may derive from the observer when he is not blinded. So we should pay attention that both the standard and the diagnostic tests be interpreted by the observer blindly. This is, for example, maybe important in a test aimed at testing a new immunofluorescence assay. Next we should pay attention at verifying the spectrum of patients where the test has been evaluated. For example, the test may perform better in people with more severe disease because a test is more likely to be positive in patients with advanced disease than in patients with early disease. We should pay attention also at verifying if in the study population patients with diseases that mimic the original disease are included. Finally, we should ask ourselves whether or not the test has been applied in the typical type of patients that we see in clinical practice. We should pay attention to reproducibility and finally, we should pay attention on whether or not the likelihood ratio has been reported.

Slide 10

Now studies on therapy and prevention. These studies use the randomised controlled trials design.

Slide 11

As Kitty Jager has emphasised, the randomised controlled trial is a particular type of prospective cohort study, that is a study where non-pharmacological intervention, for example salt intake or pharmacological intervention are tested. Here the investigator is active. Here he is not a mere observer. He makes the choice who is treated and who is not treated and the choice is made randomly. Randomisation is a great equaliser. This is the baseline data of the ALLHAT study. As you can see, in this study the number of white non-Hispanic individuals enrolled to the study were identical in all the study arms, 47, 47, 47, 46.4, the same. The number of patients who were already on anti-hypertensive treatment, identical, 90, 90, 90, 90. Smokers, identical and patients with a history of diabetes, identical. So this is the beauty of the randomised controlled trials. Randomisation is an equaliser. Now, once you have randomised patients, you follow them up during time and pick up intervention. But in this kind of study you should pay attention on whether or not the randomisation list was concealed to clinicians because if the doctors are not kept blinded but probably blinded they may consciously or unconsciously distort the balance between the groups. But these are real threats.

Slide 12

For example, in a highly reputed study which was published in the Lancet, the CAPP study, the doctors were allowed to open the envelopes containing the anti-hypertensive treatment before patients were actually entered into the study. Let’s see what happened. As you see, it was quite a large study, more than 10000 patients. In baseline conditions blood pressure was higher in patients allocated to captopril than in those allocated to conventional treatment and the difference was + 2,2 mmHg systolic, 1.7 mmHg in diastolic. It may seem trivial but it is not trivial because this difference entails a risk increase of 2.75%. So foreknowledge may be an obstacle to proper randomisation.

Slide 13

Now, we should make sure that doctors and patients are blinded and we should check whether the study was or not a placebo controlled study. Often people believe that all randomised studies are placebo controlled. This is not true. For example, the HOT study, a very good study on the effects of anti-hypertensive treatment was not placebo controlled. So, pay attention on whether or not the study was placebo controlled. Again, we need to check whether or not the definition of the study outcome was clear and measurable. We should pay attention to the drop out rate and also to the length of the follow up to make sure that the follow up was long enough to capture cases. Finally, we should pay attention to the type of analysis. You often read intention-to-treat analysis, that is to analyse the patients as they were originally allocated. This is fundamental because this is the only way for making certain that randomisation is respected. Intention-to-treat analysis is fundamental.

Slide 14

Now apart from check lists, doctors are also demanded to grade the evidence they use. As Kitty Jager has shown later on evidence has been prepared. A hierarchy of study types. At the top is the systematic review followed by randomised controlled trials and then non-randomised intervention studies, observational studies and finally, non-experimental studies and expert opinion.

Slide 15

Now, systematic reviews are at the top and rightly so. If you start a systematic review, it’s like finding a pearl in an oyster bunch. You find the pearl but pearls should not be considered isolatedly or scatteredly. You need to look at pearls well combined in a necklace and equally when you do a systematic review, you should combine the study to extract the old picture to understand what is going on combinedly in all studies. What is the advantage of systematic review? They use explicit methods and using explicit methods they limit bias in identifying and rejecting studies. This usually occurs in the so-called narrative reviews. When I think this is important this is not important but this is highly subjective. If the methods you apply are explicit, this will help to limit bias. Second, systematic reviews allow comparison of different studies in order to establish the generalisability of findings they get and also to test the consistency of the various studies. Once in a systematic review you pick up an inconsistency, that is heterogeneity between studies, you may build up new hypotheses about particular subgroups. Finally, systematic reviews can be quantitative, meta-analyses and this of course, will increase the precision of the overall results.

Slide 16

Now, said that metanalysis and systematic reviews are of paramount importance we should not forget the anecdotal evidence; case reports are important. This book has collected the world literature on adverse drug reactions and interaction in the year 2000 at the end of the millennium. 3000 citations, 1/3 of these citations were anecdotes, were case reports and there were only 45 systematic reviews. You recently have read about the rosiglitazone problem and the problem of pharmovigiliance rosiglitazone has raised. Probably you have also had the opportunity to see the meta-analysis in the New England Journal of Medicine but you should not forget that all this started with a case report. Case report that was published 7 years ago in the Annals of Pharmacotherapy. This was a study where oedema in patients taking rosiglitazone was noted for the first time and this increased the alertness of investigators on this potentially dangerous side effect. As emphasised by Kitty Jager case reports are important to generate hypotheses, elucidate mechanisms, to educate and to enable, like it was the case of rosiglitazone, systematic review.

Slide 17

Now we should not take the hierarchy of study types as a ten commandments table. Because grading various systems exist and there is no way in the existing grading system to accommodate new research design and new research design that are most important and of papers that are accepted in highly reputed journals. For example, this paper by Marcello Tonelli dealt with the effect of pravastatin on the rate of kidney function loss in people with or at risk for coronary heart disease. This study was based on three studies. It put together three different studies and it reanalysed this data. So it’s a sort of a restricted meta-analysis. How would you put this study in this hierarchy? Immediately below systematic reviews or below randomised controlled trials because after all this was a secondary analysis?

Slide 18

But there is more. For example, in the ‘80s there was much interest in aspirin in the prevention of pre-eclampsia. 6 clinical trials were produced, all small clinical trials and they collected 394 patients and the Imperiale and Petrulis made the meta-analysis. Let’s see the results of this first meta-analysis. In the 6 studies the relative risk was between 0.2 and 0.7 so benefit of aspirin. The combined relative risk was 0.035 that is 65% risk reduction. After this study a new trial was made. A quite huge trial with 9000 patients, the CLASP study and the result of the CLASP study was that the relative risk was 0.82 that is just 18% risk reduction. Now, how would you put this study in the ten commandment table? First Imperiale meta-analysis analysis and second CLASP, this is irrational. So, I’m sure that all of you will value more the CLASP than the previous analysis.

Slide 19

Now, what is the problem with grading studies? The problem is that the grading system collapses all categories in just one while there are other relevant dimension. For example, the validity of measurement this point was already made by Kitty Jager. If you are doing a study about blood pressure is blood pressure measured by sphygmomanometer in the doctor’s office or by 24-hour ambulatory pressure monitoring which is a far more precise way of measuring blood pressure. The blinding to outcome assessment is important and again, in a study about blood pressure there maybe also the observer bias. So was blood pressure measured by the sphygmomanometer or by the random 0 sphygmomanometer that keeps the observer blinded to the measurement? Finally, we should grade the study also taking into account the loss to follow up. It’s fair to say that none of the current hierarchies of evidence include all these dimensions.

Slide 20

We have different types of studies and then different types of questions. Recently an epidemiology by Oxford, Glaziou, suggested we should approach the problem like the Michelin guide. The Michelin guide keeps separate hotels, restaurants, local attractions, museums. If you are in Rome for example, and you would like to select a restaurant, the guide will tell you the restaurant in the area but also gives you the grade in spoons 1, 2, 3, spoons of the restaurant.

Slide 21

Now, we should keep these questions separate and we should formulate pertinent questions in the specific dimension of each type of study. Then grade these studies. 3, 2, 1, stars.

Slide 22

To conclude different types of research are needed to answer different types of clinical questions. Check lists are useful to verify the quality of various study types. Irrespective of the type of research systematic reviews are necessary but hierarchy of study quality should not be seen as a ten commandment table. If you have questions to pose please take note of my e-mail address. Thank you for your attention.