Techniques for Finding and Evaluating Research

by Dr. Travis H Johnson, DO, MS

 

Cruising the Internet for Information on PCOS

One of the things I enjoy most about working with the PCOS community is speaking to the many women with PCOS who are intelligent, hard-working investigators—and they have to be. Too many times I've heard the familiar tale about the day a woman was diagnosed with PCOS. Her doctor sat her down and said, "You have PCOS. Here is a prescription for Metformin and birth control. See you in two months." Before she could even form her first question, the doctor was gone. Frustrated, these industrious women take to the Internet to find satisfying answers to their questions. 

There is a lot of information on the Internet for anyone interested in learning more about PCOS. The downside to this is that—as you have doubtless already discovered—the quality and reliability of that information can vary greatly.

Here are some guidelines to help you distinguish fact from fiction: Remember that PCOS is a syndrome.

With a syndrome, each case is different—it may present differently (a different set of symptoms), and it may be caused by a different underlying problem. What works for another woman may not work for you. So, when you read on a forum about a woman who completely reversed her PCOS by taking vitamin X every day, don't feel deceived or discouraged if it doesn't work for you. You may need to try something else entirely. (By the way, there is no such thing as vitamin X.) Consider who wrote what you're reading.

Is the blogger who is raving about the miraculous results of vitamin X also selling vitamin X? Or linking to some other site that sells it? If that makes you leery, then good for you. Also consider that these connections are not always apparent.

Skepticism is healthy, but don't let it paralyze you.

Just because a blogger loves vitamin X and also sells it doesn't necessarily mean that vitamin X is useless. Gather more information. Find original research and primary literature on the topic (more on that in a moment) to decide for yourself whether vitamin X is something worth trying.

Find original research and primary literature.

Find the original sources that our blogger used when putting together her article on vitamin X. The best and most reliable writers will list their sources, either within the article or in a list at the end. If the source is another blog or magazine article, then check that article for the primary source. Once you know the name of the article, you can find it on PubMed—an online library of more than 24 million scientific articles. You can always read the abstract (a summary of the study and its conclusions) of an article on PubMed. Sometimes, you can read the entire article for free!

Original research can be overwhelming at first. But, I have also written a few tips on how to approach primary literature in a systematic way.

How to Critically Read Journal Articles About PCOS

When I talk to health professionals and women with PCOS, I find that many feel intimidated by primary literature and don't like reading it. Some of the most common reasons they give are:

• There is so much of it, and I don't know where to start.
• It uses too much jargon, and I find it hard to understand.
• I know some studies are of poor quality, but I can't tell which ones.
• Some of it doesn't make sense, and I don't know where to go for help.

Don't despair; we have suggestions for each of these issues!

Managing the Volume

Despite the fact that not nearly enough research is done about PCOS, there is still a huge volume of primary literature. A PubMed search I did today using the keyword "PCOS" produced 6,054 results. That is a lot of reading! Getting a grip on all of these articles is a daunting task. (So daunting, in fact, that many physicians only read the abstracts, or even only the titles. Even worse, they may not read anything at all.) Here are a few pointers that I hope will help you deal with information overload:

Start with review articles.
If you are new to reading original PCOS research, then review articles are an excellent starting point. They tend to be written for broader audiences, which means less jargon and clearer explanations. When I clicked the "review" link on my search results page (it is in the left sidebar), there were 1,073 results. Down by a factor of 6, but still too many!

Start with current literature.
Our understanding of PCOS evolves and builds over time. Current literature (with some notable exceptions) tends to reflect our best understanding. When I limited my results to studies published since 2012, I got my total down to 243. Down by a factor of 4, but still too many!

Start with free articles.
It is no good if you find a paper with a really interesting title if you discover that buying a copy costs $54. There are lots of good, high-quality, free journals out there, so start with those. When I clicked the "free full text" button on my search results page, there were 74 results. Down by a factor of 3, but still too many (at least for now)!

Skim the results.
Ignore what seems uninteresting or too technical. The eighth result on my refined search wasthis article. It has a broad title, which usually means it is will be more readable. It covers a lot of interesting topics in PCOS, like vitamin D deficiency and Anti-Müllerian hormone. You can read the whole thing or just the topics that interest you. If you decide you want to know more, you can use the works cited section to find other papers worth reading. I call this "following the bread crumb trail" and I think it is one of the most fun and informative things you can do when reading up on any topic. Citations can lead to surprising discoveries.

Managing the Jargon

"One of the most frustrating things I encountering an unfamiliar word and going through the trouble of looking it up, only to discover it means nearly (or exactly) the same thing as a common word people use every day."

Jargon helps scientists be specific and succinct, but it also makes communication harder in some ways. It is here to stay.

Here are some things you can try to manage it:

Start with review articles.
Again, they are usually the most clearly explained publications on any given topic.

Read articles in translation.
This may sound strange, but you could start with foreign or international journals that are published in English. Sure, there will be some funny sounding sentences here and there, but for the most part these journals are written with simpler sentences and less of what I call "smartest guy in the room" explanations of things—those moments in an article where it seems like the author is more interested in sounding smart than being understood. They drive me bananas!

Look up strange words.
Yes, it slows down reading the article you have in your hand. But, don't be afraid to get out a dictionary or Google something. It will make reading every future article easier. It is an investment in yourself that I think is worth the time.

Skip it.
If a sentence or section is total gibberish, skip it. If you get to the end of the paper and you feel like you got the gist, then that one piece may not have been so important.

Assessing Quality

Is this a high-quality study? Is it garbage? Can I trust the conclusions? Answering these sorts of questions is difficult, especially for laypersons. The good news is that as you read more and build a body of knowledge, you'll get better and better at this. Here are some tips to help you get started on your journey:

Start with the materials and methods section.
Some people skip this section entirely, but I always read it first. People skip it because it is dry, technical, and painfully boring. But, it is the key piece in deciding whether to believe a paper's conclusions. Were the materials junk? Were the methods flawed? If so, you can skip the rest of the paper. You wouldn't live in a skyscraper made out of mud and sticks, so why believe the conclusions of a poorly constructed study?

Read the conflict of interest and funding statements.
"Funding bias"is a real problem. Good research should make perfectly clear where the money for the research came from. Researchers should disclose any financial interest in the outcome of the research. This doesn't always happen, so check and see before you spend a few hours really digesting the entire study.

Consider the size of the study.
This is often just called the n (n is for number) of the study. The number often set as a threshold for being meaningful research is 30, meaning there should be at least 30 participants in the clinical trial. This isn't a hard and fast rule, but keep in mind that bigger isn't always better. Huge clinical trials with subjects in the hundreds or thousands are not necessarily better than small, well-conducted trials. The FDA itself recommends small trials for establishing efficacy. Large trials are good for finding rare side effects and adverse reactions. But wait, then why do drug companies always brag about their huge clinical trials?

Well, that brings us to the next point: Consider the significance.

First, you must appreciate the difference between statistical significance and clinical significance. We say that a result is statistically significant when a difference in results (between vitamin X and placebo, for example) is too great to be purely by chance. This is determined by some complicated mathematics. Clinical significance is about outcomes, not mathematics.

When we ask: "Is it clinically significant?" we are really asking: "Does it really make a difference?" 

Here's a disturbing fact to illustrate the difference: the larger the study, the more likely that small changes can be (mathematically) proven to be statistically significant. So, when Company X sponsors a 1,000-person study of their new blood pressure medication, the large number of subjects actually makes it easier to show a small change was statistically significant. Company X gets really excited, because now they can say that the 2-point reduction in blood pressure is statistically significant. But, is a 2-point reduction in blood pressure clinically significant? In other words, will it matter to the patient? Probably not. But that won't stop a pharmaceutical representative from bragging about the "significant drop in systolic pressure" to any doctor that will listen. Believe it or not, things like this drive the prescribing habits of physicians in the US.

Another example, in case the blood pressure example wasn't absurd enough for you: A new study shows that Sunscreen X protects the skin from burning for longer than Sunscreen Y. The math shows that the difference is statistically significant—it's definitely not due to chance; it's all there in black and white. But when we look at the data, we see that Sunscreen X is effective for an average of ninety seconds longer. Is ninety seconds significant to you?

So, when you see researchers claiming that one treatment is significantly better than another, always always always look to see what the difference actually is. And then answer for yourself: "Does it really make a difference?"One last bit on this subject before we move on. The methods of analyzing statistics are myriad and too complex to cover here. But, you should know those different methods of analysis yield different results. That is, if a researcher with dubious intentions isn't satisfied with the results of his first analysis of the data, he could re-analyze by another method. A small change that was determined to be statistically insignificant by one method could be ruled significant by another method of analysis: yet another reason to be aware of any funding bias, and yet another reason to always consider the clinical significance.

Decide whether the evidence supports the authors' conclusions.
If the paper didn't have any red flags and the conclusions seem appropriately stated, then it is probably something to consider moving forward.

Did the authors find only a small change but write a very strongly worded conclusion about how great the treatment was? That should make you leery and want to look for corroborating evidence—in other papers.

 
Getting Help

What do you do when you read a paper and you're still just not sure what it all means?

Send us an email! I can track down almost any paper. And, if it is related to PCOS, I will read it, and we will post a summary and review for you (and the whole world) to read.

Click here to read my comments and opinions on some PCOS related studies