Welcome

What is EBP?

ASK
Clinical Question

ACQUIRE
Literature Search

APPRAISE
Evaluate Evidence

Test
Your Knowledge

Glossary (CEBM)

References

Feedback

Evaluating the validity of a Etiology/Harm study

Are the results of this article valid?

COHORT Studies: Aside from the exposure of interest, did the exposed and control groups start and finish with the same risk for the outcome?

1. Were patients similar for prognostic factors that are known to be associated with the outcome (or did statistical adjustment level the playing field)?
The two groups, those exposed to the harm and those not exposed, must begin with the same prognosis. The characteristics of the exposed and non-exposed patients need to be carefully documented and their similarity (except for the exposure) needs to be demonstrated. The choice of comparison groups has a significant influence on the credibility of the study results. The researchers should identify an appropriate control population before making a strong inference about a harmful agent. The two groups should have the same baseline characteristics.  If there are differences investigators should use statistical techniques to adjust or correct for differences. 

2. Were the circumstances and methods for detecting the outcome similar?
In cohort studies determination of the outcome is critical.  It is important to define the outcome and use objective measures to avoid possible bias.  Detection bias may be an issue for these studies, as researchers may look deeper to detect disease or an outcome.

3. Was follow-up sufficiently complete?
Patients unavailable for complete follow-up may compromise the validity of the research because often these patients have very different outcomes than those that stayed with the study. This information must be factored into the study results. 

CASE-CONTROL STUDIES: Did  the cases and control group have the same risk (chance) of being exposed in the past?

1. Were cases and controls similar with respect to the indication or circumstances that would lead to exposure?
The characteristics of the cases and controls need to be carefully documented and their similarity needs to be demonstrated. The choice of comparison groups has a significant influence on the credibility of the study results. The researchers should identify an appropriate control population that would be eligible or likely to have the same exposure as the cases.

2. Were the circumstances and methods for determining exposure similar for cases and controls?
In a case control study determination of the exposure is critical.  The exposure in the two groups should be identified by the same method. The identification should avoid any kind of bias, such as recall bias. Sometimes using objective data, such as medical records, or blinding the interviewer can help eliminate bias. 

What are the results?

How strong is the association between exposure and outcome?
    * What is the risk ratio or odds ratio?
    * Is there a dose-response relationship between exposure and outcome?

How precise was the estimate of the risk?
    * What is the confidence interval for the relative risk or odds ratio?

Strength of inference:

For RCT or Prospective cohort studies:

Relative Risk (RR) is the risk of the outcome in the exposed group divided by the risk of the outcome in the unexposed group:

RR = a /(a + b) divided by c/(c + d)

RR = (exposed outcome yes / all exposed) /
(not exposed outcome yes / all not exposed)

"RR of 3.0 means that the outcome occurs 3 times more often in those exposed versus unexposed."

For case-control or retrospective studies:

Odds Ratio (OR) is the odds of previous exposure in a case divided by the odds of exposure in a control patient:

OR = (a / c) divided by (b / d)

OR = (exposed outcome yes / not exposed outcome yes) / (exposed outcome no / not exposed outcome no)

"OR of 3.0 means that cases were 3 times more likely to have been exposed than were control patients."

Confidence Intervals are a measure of the precision of the results of a study. For example, "36 [95% CI 27-51]", a 95%CI range means that if you were to repeat the same clinical trial a hundred times you can be sure that 95% of the time the results would fall within the calculated range of 27-51. Wider intervals indicate lower precision; narrow intervals show greater precision.

Confounding Variable is one whose influence distorts the true relationship between a potential risk factor and the clinical outcome of interest.

More information on odds ratios: The odds ratio Douglas G Altman & J Martin Bland BMJ 2000;320:1468 (27 May)

How can I apply the results to patient care?

Were the study subjects similar to your patients or population?
Is your patient so different from those included in the study that the results may not apply?

Was the follow-up sufficiently long?
Were study participants followed-up long enough for important harmful effects to be detected?

Is the exposure similar to what might occur in your patient?
Are there important differences in exposures (dose, duration, etc) for your patients?

What is the magnitude of the risk?
What level of baseline risk for the harm is amplified by the exposure studied?

Are there any benefits known to be associated with the exposure?
What is the balance between benefits and harms for patients like yours?

Source:  Guyatt, G. Rennie, D. Meade, MO, Cook, DJ.  Users' Guide to Medical Literature: A Manual for Evidence-Based Clinical Practice, 2nd Edition 2008.

Note: For criteria for other types of studies, see the following supplements:
 Therapy | Prognosis | Diagnosis| Systematic Review

Revised July 2010