Evaluate the
therapy study | Are the results
valid? | Return to the Patient
|
Note: For criteria for
other types of studies, see the following |
Evaluating the Validity
of a Therapy Study
|
The evaluation |
4. Appraise that evidence for its validity (closeness to the truth) and applicability (usefulness in clinical practice) |
We have now identified current information which can answer our clinical question. The next step is to read the article and evaluate the study.
There are three basic questions that need to be answered for every type of study:
-
Are the results of the study valid?
-
What are the results?
-
Will the results help in caring for my patient?
This tutorial will focus on the first question: are the results of the study valid? The issue of validity speaks to the "truthfulness" of the information. The validity criteria should be applied before an extensive analysis of the study data. If the study is not valid, the data may not be useful.
The evidence that supports the validity or truthfulness of the information is found primarily in the study methodology. Here is where the investigators address the issue of bias, both conscious and unconscious. Study methodologies such as randomization, blinding and accounting for all patients help insure that the study results are not overly influenced by the investigators or the patients.
Evaluating the medical literature is a complex undertaking. It is not the intention of this tutorial to downplay that intellectual process or to make you an expert on evaluating the evidence. This session will provide you with some basic criteria and information to consider when trying to decide if the study methodology is sound. You will find that the answers to the questions of validity may not always be clearly stated in the article and that clinicians will have to make their own judgments about the importance of each question.
Once you have determined that the study methodology is valid, you must examine the results and their applicability to the patient. Clinicians may have additional concerns such as whether the study represented patients similar to his/her patients, whether the study covered the aspect of the problem that is most important to the patient, or whether the study suggested a clear and useful plan of action.
Note: The questions that we used to test
the validity of the evidence are adapted from work
done at McMaster University. See the References/Glossary
unit: 'Users' Guides to the Medical Literature.'
|
The article selected to help answer our clinical question is: The effect of digoxin on mortality and morbidity in patients with heart failure. The Digitalis Investigation Group. New England Journal of Medicine February 20, 1997; 336(8):525-533. The Massachusetts Medical Society has given us permission to make this article available for this tutorial. |
Are the results of this therapy study valid?
1. Was the assignment of patients to treatment randomized?
The assignment of patients to either group (treatment or control) must be done by a random allocation. This might include a coin toss (heads to treatment/tails to control) or use of randomization tables, often computer generated.
Research has shown that random allocation comes
closest to insuring the creation of groups of patients
who will be similar in their risk of the events you
hope to prevent. Randomization balances the groups
for prognostic factors (ie. disease severity) which
eliminates over-representation of any one characteristic
within the study groups. Randomization should also
be concealed from the clinicians and researchers
of the study to help eliminate conscious or unconscious
bias.
Article: This information is found under the METHODS section. The patients in this study were randomly assigned to receive digoxin or placebo. The randomization was stratified by left ventricular ejection fraction as well as by study center. It appears that randomization was concealed and done by telephone through a coordination center. The researchers enrolling the patients did not have any information about the randomization scheme. (Additional details about the study methodology were reported in a previous article.)
2. Were all the patients who entered the trial properly accounted for at its conclusion?
Was follow-up complete?
All patients who started the trial should be accounted for at the end of the trial. If patients are not accounted for, the validity of the study may be jeopardized. A good study will have better than 80% follow-up for their patients. Patients may drop out of a study for various reasons, including because of adverse events related to the treatment. If these patients are not included in the results, they can make the treatment look better than it really is (and vice versa). To be sure of a study's conclusions, lost patients should be assigned to the "worst-case" outcomes and the results recalculated. The results are still valid if the recalculations do not change the end results.
|
Article: Yes, the final status of 98.6% of patients is known at the end of the trial. In addition, the investigators comment on a sensitivity analysis (how sensitive the results would be to the "worst" case scenario if all the lost patients had the worst outcome). In this case, a sensitivity analysis showed that the "lost" patients would not effect the overall mortality results, even if all had died. They do not state how a "worst" case scenario would effect hospitalizations |
Were patients analyzed in the groups to which they were (originally) randomized?
Anything that happens after randomization can affect the chances that a patient in a study has an event. Patients who forget or refuse their treatment should not be eliminated from the study analysis. Excluding noncompliant patients leaves behind those that may be more likely to have a positive outcome, thus compromising the unbiased comparison that we got from the process of randomization. Therefore patients should be analysed within their assigned groups. This is called "intention to treat" analysis.
|
Yes, analysis was done by intention to treat. This information is in the METHODS section, under Statistical Analysis. |
3. Were patients, clinicians, and study personnel "blind" to treatment allocation?
Blinding means that the people involved in the study do not know which treatments are given to which patients. This eliminates bias and any preconceived notions as to how the treatments should be working. When it is difficult or unethical to blind patients to a treatment, such as a surgical treatment, then a "blinded" researcher is needed to interpret the results.
|
Article: The researchers and data analysts who looked at clinical outcomes were blinded to the treatment groups. It is not clear if the patients and clinicians were blinded to the study arms. In some patients, if clinical need arose, a switch was made to open label treatment. Measures were taken to minimize unblinding, however, this was not always possible. |
4. Were the groups similar at the start of the trial?
The treatment and the control group must be similar for all prognostic characteristics except one: whether or not they received the experimental treatment. This information is usually displayed in tables that outline the baseline characteristics of both groups.
|
Article: This information is found in Table 1, which presents the baseline characteristics of the patients in the 2 study arms. Characteristics appear to be evenly distributed within the 2 groups. |
5. Aside from the experimental intervention, were the groups treated equally?
Both groups must be treated the same except for administration of the experimental treatment. If "cointerventions" (interventions other than the study treatment which are applied differently to both groups) exist, they must be described in the methods section of the study.
|
Article: It appears that both groups were treated the same. There are no reported differences in co-interventions, follow-up or outcome measures. |
Are the results of this study valid?
|
Article: This study methodology appears to be sound and the results are valid. |
What are the results of the study?
Main results: Analysis was by intention to treat. Mortality did not differ between groups. 1181 deaths occurred in the digoxin group compared with 1194 deaths in the placebo group. (34.8% vs. 35.1%, P=0.8) The digoxin group however, had lower rates of hospitalizations overall compared with the placebo group (64.3% vs. 67.1%, (P=0.006).
Conclusions: Digoxin did not affect mortality but reduced hospitalizations in patients with heart failure and normal sinus rhythm.
Outcomes:
Outcome |
Digoxin |
Placebo |
RRR |
ARR |
NNT |
Mortality |
34.8% |
35.1% |
nonsignificant p=0.08 |
||
Total hospitalization |
64.3% |
67.1% |
4.1% |
2.8% |
36 |
hospitalization for CHF |
27% |
35% |
23% |
8% |
13 |
hospitalization for CV causes |
49.9% |
54.4% |
8.3% |
4.5% |
22 |
Reprinted with permission from the American College of Physicians. (ACP Journal Club 1997 Sep/Oct;127(2):34)
Are the results applicable to your patient?
In some ways, yes: 86% of the patients in the trial were white, 73% were younger than 70 years, and 84% were either NYHA class II or III. However, only 22% of patients in the study were female. All in all, it is likely that our patient is close enough in characteristics to the study population that the results may be applied to her.
|
Key issues for studies of Therapy:
|
|
Key terminology for estimating the size of the treatment effect
| ||||||||||||||
Source:
Guyatt GH ; Sackett DL ; Cook DJ. Users' guides
to the medical literature. II. How to use an article
about therapy or prevention. A. Are the results of
the study valid? Evidence-Based Medicine Working
Group. JAMA 1993 Dec 1;270(21):2598-601.
Return to the Patient
|
The patient |
5. Return to the patient -- integrate that evidence with clinical expertise, patient preferences and apply it to practice |
It appears from our brief analysis that this article meets the criteria for validity. To complete the analysis you would need to review the results and determine if they are applicable to Pauline.
|
The study population appears to be similar enough to Pauline that we can consider the results applicable to her case. The results of this study indicate that digoxin can reduce the need for hospitalizations in patients with heart failure and normal sinus rhythm. Digoxin may be an appropriate therapy to help keep Pauline at home and out of the hospital. However, if Pauline elects to be treated with digoxin, there will be a need to monitor therapy, draw frequent drug levels, and hold the risk of toxicity. For Pauline, these issues may be offset by the possible benefit of avoidance of hospitalization. |
|
|
Self-evaluation |
6. Evaluate your performance with this patient |
Take a moment to reflect on how well you were able to conduct the steps in the EBM Process.
Did you ask a relevant, well focused question? Do you have fast and reliable access to the necessary resources? Do you know how to use them efficiently? Did you find a pre-appraised article? If not, was it difficult to critically evaluate the article?
|
Note: For criteria for other types of studies, see the following |
