Evidence-Based Medicine and How to Read Research Like a Pro: Part 1

My good friend IrishUp posted an article she wrote on evidence-based medicine in an ED support forum, which will be part of a series on learning how to read and interpret medical research. It’s something we’ve both learned from experience, and Irish graciously allowed me to share her series here at ED Bites. All of my comments or additions are in italics so you can clearly see what was original content and what wasn’t. I briefly edited the intro for clarity. I hope you all enjoy.

Sometimes I feel like a lot of us feel overwhelmed, bewildered, or just plain turned off by the debates around EDs, Evidence Based Medicine, different therapies & treatments, and all the research and data we throw around this joint. In a good way! Even if sometimes our passions make the discussions contentious, *that* the discussions are happening is very valuable, IMHO. But I think they could be even *more* valuable.
What I’d like to do on this thread is give readers a practical tutorial with the following goals:

  • Explain the basics of medical research;
  • Explain the relationship between research and the practice of medicine;
  • Give readers the tools to critique Mainstream Media (MSM) reporting of primary research;
  • Give readers the tools to evaluate a primary research paper for their own understanding.

In parts of the ED world, there is a lot of focus on Evidence Based Medicine and “The Research”. Some might have a vision of a world where all ED research was well designed with clear outcomes that are plainly explained, and these would form a pool of data – evidence – that would lead to uniformly effective treatment. And that would be swell, but that is not the world we live in.

The truth is that no research is perfect, some research is poorly designed or executed, and no treatment for any disease or condition is uniformly effective. We think of Medicine as a Science. It is NOT, it is an Art that is *informed* by science. And actually, it’s a GOOD thing that Medicine does not rely 100% on science – it would be bad if a discipline as complicated and encompassing as Medicine had to wait for and rely on only one source of information. That said, the science behind Medicine is REALLY IMPORTANT!

Step 1: Understanding Medical Research -Different Kinds of Research Give You Different Kinds of Information.

The first thing is to understand that the “knowledge base” of medicine comes from many sources. Each of these has strengths and weaknesses. The FIRST thing to do when you are looking at a piece of research is to identify which category it falls under. Then you have an idea of what kind of information the study design is good at getting, and what the limitations are. This is by no means exhaustive, but it’s a good starting point :

1. Primary bench science: this would be the kind of experiments NOT run on people at all, but more like the stuff most of us did in biology or chemistry in school. This would include experiments on animals.

  • Pros: Valuable for understanding basic mechanisms, lots of control over conditions, you can easily repeat what was done.
  • Cons: hard to know how well what happens in a lab will match what happens in a real live human being, lab conditions do not replicate real world conditions very well.

2. Case reports details a single patient or small series of patients. These describe what the patient(s) showed up complaining of, their medical history, physical exam, symptoms, the treatment(s) administered and the outcome of the patient. Prior to formally applying the Scientific Method to medical research, Case reports comprised the bulk of medical knowledge. They are still extremely valuable for detailing rare diseases and conditions,novel treatments, unusual outcomes, or other anomolous or singular patient care events.

  • Pros: Pretty much the only way to study rare or singular events, good for descriptive purposes.
  • Cons: very small numbers, not repeatable, hard to know how generalizable (how applicable to another person or situation) the results or findings are.

3. Observational studies which involve no “experimental” manipulation of patients in any way, but rely on identifying a set of patients with some shared quality – a diagnosis, an exposure, a treatment, an outcome – and either looking back in time to see what happened to them before the diagnosis/treatment/outcome of interested (in a retrospective study design) or forward to see what happens AFTER your event of interest (a prospective design). The important part here is that you aren’t changing anything that happened or will happen to the people you are studying.

  • Pros: Inexpensive to do, easier to get large numbers, the study is already looking at “real life” situations, low risks to study participants.
  • Cons: Can’t control all of the conditions, can only show relationships but CANNOT show which thing causes what, can be hard to replicate.

4. Randomized, controlled, clinical trials which involve carefully selecting people to get a treatment of your specific design. RCTs involve active participation from the subjects because you are deliberately manipulating what happens to them. Also very important: The treatments administered in RCTs have the PRIMARY GOAL of benefiting OTHER people! Now, the risks to the patient have to be acceptable – you can’t do something worse to a person than what would happen if they *weren’t* in your study. But since you’re conducting an experiment, you don’t KNOW if you’re doing anything better, so really, the people who get the MOST benefit from an RCT are the people who get treatments that come AFTER the results of the study are known.

  • Pros: You can control conditions and participants so you don’t have contamination from unknown sources. You can “prove” causality. Easy to duplicate
  • Cons: Expensive, study participation limits can limit generalizability.

RCTs are called the “Gold Standard of Proof”. What that means is that RCTs are the only experimental design that gives you live people getting real treatments or exposures in such a way that you can *show* that what you did CAUSED the findings you observed.

BUT! Not every problem can be investigated with an RCT. And you DON”T need an RCT if the preponderance of evidence reaches a certain tipping point. Real life example:  – Smoking has never been “proven” to cause lung cancer in humans. This is because it would be unethical to design a study where you make one group smoke. The preponderance of the evidence from primary bench science, animal studies, and observational human studies is that smoking *does* cause lung cancer. This did not stop scientists in the employ of tobacco companies from loudly proclaiming that the connection wasn’t “proven” for decades. They were technically right, but ethically reprehensible.

The other issue is that RCTs participants are often very unlike the larger group of people who would get the treatment, and that the treatment is administered under ideal, and not real world conditions. Because RCTs can be very small in terms of numbers of participants, it can be hard to get detailed data about subgroups of patients – men vs. women, responders vs. not responders, that kind of thing.

{Another mom posted this in response to the part of RCTs: What randomization does is control for variables you haven’t thought of or don’t even know about.  Randomly assigning subjects to treatment groups means that important variables that could impact the study will end up in all comparison groups equally.  Effectiveness of randomization needs to be checked, and it is highly dependent on sample size.  In a small study you can easily find things that pop out as different in the groups despite the randomization.  For example, if a study has an age range for inclusion, you can check and make sure that the age distribution in the 2 groups is the same after the study has been recruited.

This is the hardest part about recruiting a trial for medical care.  Most people want to have the medical treatment that their Dr. thinks is best for them individually.  In an RCT you will be randomized to treatment.  Randomizing to a placebo group is even harder, and here’s where blinding is essential because otherwise subjects who end up in the placebo group tend to drop out.

Blinding is also important, but as someone mentioned, can be difficult.  Not impossible, though.  In some cases, subjects can be blinded to the type of therapy they are receiving if they don’t know much about it (don’t know the difference between CBT and ACT).  Also, the experimenters should make sure whoever is measuring the outcome (for example, administering the psychological questionnaire) is different from the person who supplied the therapy, and is not told which arm the subject was on.  Double blinding (means Drs are also blinded)- impossible for therapy, standard of practice (or should be) for RCT drug trials.}

So, take home of part 1: When reading a report about research, look for what kind of study it was. Think about the plusses and minuses of that kind of study. Then look at the results being reported and see what kind of “yeah but” (or caveat) might apply.

Twitter Digg Delicious Stumbleupon Technorati Facebook Email

3 Responses to “Evidence-Based Medicine and How to Read Research Like a Pro: Part 1”

  1. Neeraj Chowdhury February 24, 2013 at 9:37 am

    It seems that the main stream media has a habit of always sensationalising outcomes of any research or trial involving unusual outcomes or which diapproves traditional thinking leading to un-ncessary confusion in minds of uninformed readers of the publication. Such half baked, half truths and half accurate information create amigiuty in reader’s mind and they are unable to make an informed opinion on the subject. They are always in the dilemma of whom to trust and belive and which information to discard and ignore. Only when just and unbiased reporting starts appearing in mainstream media is when people will trust such reporting and use the information to enrich lives.

    • The misrepresentation and sensasionalistic reporting approach in MSM is certainly a problem. While I wouldn’t deny that certain media outlets have an obvious bias or agenda, when it comes down to it, even the science journalists are often not very science literate. In the modern era, the science reporter is not necessarily chosen for their science “chops” or journalistic craft, and the expected turn-around time from story idea to airing/publication is a fraction of what it once was – leaving little-to-no-time for the reporter to investigate their topic. Reporters themselves are often writing in ignorance of these same issues, at the same time they have a short deadline, and an Editor or Producer who is shaping the “angle” of the story.

      So I agree that it is a situation conducive to creating confusion that needn’t be.

      That said, there is more primary source data available to the general public than ever before. In the US, research with federal funds MUST be publicly available (see here ), and the rules of citation and attribution are still in place. Many people won’t look at these primary sources for fear that it will be too confusing and they won’t understand it.

      This series is intended to teach people about research and how research is presented, so that they WON’T be confused and WILL understand enough of it to make up their own minds, without having a staff of people in the hire of an MSM company do that for them. Underneath all the jargon and Latin words, a research publication is a very basic story of who? why? when, where & how? and what?

      Who was studied? Who did the study?
      Why is this study important? Why do we care about the research question and any answers this study found?
      When & where was the study done? How was the study conducted?
      What were the results of the study? What do those results mean to ME, who is interested in this topic?

      This is information that a regular person can get, once they know how to do it.


  1. Rethinking ED Prevention | ED Bites - March 2, 2013

    […] of people might hope. About five years ago, Stice and colleagues published the first results of a randomized control trial of an eating disorder prevention program that deconstructs the thin body ideal (Stice et al., 2008). […]