We are regularly bombarded with news of the latest scientific research findings, and sometimes it seems like you can find a study to tell you just about anything.  My concern with news reporting of research is that many people (including members of the media) have relatively limited research literacy.  Research literacy refers to the ability to draw upon a body of knowledge about research methods in order to interpret and critically evaluate research findings.  Greater research literacy means, among other things, a more effective bullsh** detector.  In this post I’ll break down some of the terminology that’s often used in research.  The concepts involved are not necessarily intuitive, so it’s important to reflect on how we can most appropriately use the information we are given.  The randomized, double-blinded, placebo-controlled trial is considered the “gold standard” for much medical research, and it’s useful to understand why that is.

Peer review

Peer review is an important quality control step before research papers are published.  Reviewers are independent of the journal the paper has been submitted to, and they remain anonymous (except to the journal editors).  Peer reviewers are selected by editors because of their expertise in the field, and they will give feedback to guide the editor in making a decision about whether or not to publish a manuscript.  Most often reviewers will recommend improvements be made before a paper is accepted.

The peer review process can potentially get a little shakier when it comes to “open access” journals.  The articles in these journals are publicly available without a subscription, and instead the journal’s revenue stream comes from authors, who must pay a publication fee.  Some of these journals are credible and reputable, while others are predatory.  Because I’ve had work published in academic journals, my email address is out there in that world, and I regularly get emails from obscure open access journals asking me to do reviews for papers that are so completely unrelated to my research field it’s sometimes laughable.  Hmmmm.

Quantitative vs qualitative research

Quantitative research collects quantifiable data and generally uses statistical methods to analyze that data.  This type of research is the basis of much of the medical knowledge that we have.  However, numbers can never tell the whole story, and that’s where qualitative research comes in.  Qualitative research uses people’s stories to answer questions like what does it feel like to have mental illness, for example.  I’m particularly partial to a research method called autoethnography, which is a study of the self in the context of culture.  I used this in my masters thesis to examine my own experiences of mental illness, and it’s a really exciting way for the voices of people with mental illness to be heard.

Randomization

There is a great deal of person-to-person variability in health status, genetics, and many other factors.  By randomizing which participants are assigned to the intervention group and which are assigned to the control group, the likelihood decreases that one particular group will be stacked with people having certain characteristics.  Greater homogeneity between participants receiving intervention A and those receiving intervention B improves the validity of the results obtained.

Blinding

Blinding refers to who knows what intervention the patient is receiving.  In a double-blinded study, neither the patient nor the investigators evaluating the outcomes know this.  Blinding is considered desirable because it reduces the chance of bias in the results.  In practical terms, in a study of Drug A, drug A and a placebo might each be contained in identical gelatin capsules.  Each participant would receive a numerically coded dose, and a member of the research team not involved in administering the drug or evaluating the participant’s response would be responsible for managing the codes.  Specifics vary from study to study, but the purpose is still the same.

Controls

Let’s say you give drug A to 20 people.  A month later 15 of them are doing better.  That’s great, but it doesn’t tell you the extent to which drug A is or is not affecting outcomes.  That’s where controls come in.  A control represents a comparison group that allows researchers to make some determination of what is causing what.  Using a placebo control is often the most desirable research design in separating out how much of the response is actually due to the drug being studied.

Sometimes another active drug is used as a control.  One example of this is in studies of ketamine for major depressive disorder.  Since ketamine is an anaesthetic, if a normal saline infusion was given as a placebo any attempts at blinding would go right out the window.  Therefore, studies might use something like intravenous midazolam, a rapid- and short-acting benzodiazepine, as a control.  Other types of controls include “treatment as usual”(the receiving whatever treatment is considered standard practice) or “wait list” (patients who are currently on a wait list to receive the intervention that’s being studied).

Risk

Risk is often quantified as a percentage, but a number needs sufficient context to be meaningful.  Absolute Risk refers to the probability of something happening, full stop, while relative risk refers to the probability of something happening in relation to some other designated population, factor, or situation.  Relative risk numbers are meaningless unless you know the reference point.  Let’s say drug A causes a 500% increase in the risk of cancer X.  Sounds scary, right?  But what if the absolute risk of cancer X is 0.00001%?  That means taking drug A increases the risk of cancer X to 0.00005%, which is a lot less frightening.

Reports on the news may talk about an increased risk of death from drug B.  Again, that sounds pretty scary.  Except the absolute risk of death is 100%, with no exceptions (at least thus far in human evolution).  That means we need more information.  What time frame are we talking about?  Risk of death within 1 year?  Within 20 years?  It makes a difference.

Correlation vs causation

Correlation refers to things that tend to happen together, and this can mistakenly be taken to mean there is a causative relationship.  If we looked at 100 people who had died, we would find that 100% of them had skin.  Does that mean having skin causes death?  Of course not.  This is a common type of mistake in the anti-vaxxer movement.  Autism can show up at around the same age as children receive some vaccines, but that doesn’t mean there is a causative relationship.  Causation can be difficult to establish definitively, which is why it’s important to have well-designed, rigorous, peer-reviewed research studies.

Significance

The term “significant” is used differently in research studies than it is in general usage.  Significant results in a study indicate that from a statistical perspective the results obtained were unlikely to be due to chance.  It does not mean that the results are large in scale (this would be described as a large effect size) and it does not mean that the results are necessarily important or meaningful.

Related to this is the concept of “confidence intervals”, which gives a range that expected results would fall within a certain percentage of the time, such as 95%.  As a completely arbitrary example, say placebo was associated with a 10% improvement in symptom x, with a 95% Confidence Interval of +/- 5% (i.e. 5-15%).  Study drug A was associated with 25% improvement in symptom x with a 95% confidence interval of +/- 5% (i.e. 20-30%).  Because drug A is associated with a greater improvement in symptom x, and the lower end of its confidence interval doesn’t overlap with the upper end of the placebo confidence interval, the results are considered statistically significant.  These numbers are completely made up, but hopefully this example gives you the right idea.

Systematic reviews

Review papers evaluate the existing research literature on a topic.  “Systematic” refers to clearly laid out criteria for inclusion and exclusion of studies in the review and elements used to evaluate the quality of the included studies.  Review articles are a great way to get a lot of information all in one place that has already been carefully scrutinized for you.

Conclusion

So there it is, a quick overview on bringing a critical eye to research.  The best way to improve research literacy is by actually consuming research literature.  This is where Google Scholar can be your new best friend.  Full research papers often aren’t available for free access, but for the most part there is public access to a paper’s abstract, which is a concise summary of the key points including method, results, and conclusions.  PubMed is another great source of information, and often hits on Google Scholar are linked to the corresponding PubMed page.  Papers with the results of research funded by the National Institutes of Health (NIH) in the United States are available for free on PubMed.  I’ve read a lot of research papers over the years, and while they can be a bit dry  they can be also be an invaluable source of information.  And the more you read the easier it gets. Trust me.

Image credits:

GDJ on Pixabay

PublicDomainPictures on Pixabay