Systematic Reviews

Systematic reviews have proliferated greatly in past years. Unfortunately, this larger number of articles means a wider range of quality. Researchers should be informed about what goes into producing a high-quality systematic review before starting one.

Systematic reviews require a team of researchers, a long lead time to publication, and major time commitment. Rush librarians are here to help – starting with this informational guide.

Is a systematic review appropriate for your research question?

The following article is from "Liblog: Newsletter of the Mayo Clinic Libraries" May 1, 2013, by Melissa L. Rethlefsen, MLS, AHIP

I Want to Do a Systematic Review

“I need to do a systematic review.”  “I want a systematic review.”  The refrain is increasingly common, but it’s not always clear what someone means by a systematic review. Before our reference librarians will start on a “systematic review” literature search, they will work with you to make sure that what you really need–and want–is a systematic review.

So, what is a systematic review?  In basic terms, a systematic review is a protocol-driven, comprehensive literature review, usually designed to answer a specific clinical question.  What’s different about a systematic review than a regular literature review?

• Systematic reviews generally answer very focused, PICO-based questions.
• Systematic reviews have a protocol in place prior to the literature review beginning, including:
  • The clinical question
  • Specific inclusion and exclusion criteria
  • Methods for assessing bias
  • Methods for combining the data (e.g., via a meta-analysis)
• A systematic review literature search is pre-specified and designed to find all relevant materials; a literature review does not follow a pre-specified protocol, nor does it need to be truly “comprehensive”.
• A systematic literature search attempts to reduce bias by searching in all languages; searching across multiple databases, including subject-based and regional databases; and searching “grey literature” sources like clinical trial repositories, the web, and unpublished trials.
• Systematic review protocols generally dictate that two or more individuals independently review each retrieved article separately to determine whether it meets inclusion criteria. Conflicts are solved by a third party.  A literature review, on the other hand, could be done by a single person.
• Systematic reviews commonly produce large amounts of search results, sometimes even in the tens of thousands, because they are designed for high sensitivity. A typical literature review would aim primarily for specificity–that is, all of the results would be focused on the topic.
• Systematic reviews publish, as part of their methods sections, the details of the search strategy. Systematic review literature searches are ultimately replicable; the search strategies (including database names and platforms, dates of the search, all search terms, and any limits used) are published so that others who want to redo the searches can find the same information.
• Regular literature review searches do not need to be replicable or even reported, though on occasion, you may see a published search strategy or a list of keywords the author used to search as part of a regular review’s methodology. Just because a literature review does not publish a list of search terms (which is not particularly helpful without the logic behind it) or a full strategy does not make it a bad or “loosey-goosey” search–it just means the authors did not need to make the search replicable for a systematic review.
• Systematic reviews are often the basis for a meta-analysis, where the data from the materials fitting the pre-specified criteria are pooled and statistically analyzed. Traditional literature reviews do not apply additional statistical methods to the materials found.
• Systematic reviews take exponentially more time to do, from the search strategy creation itself, to going through each retrieved citation in duplicate or triplicate, to analyzing the data from the included articles.

Before you begin a systematic review, ask yourself:

• Do I have a clearly defined clinical question with established inclusion and exclusion criteria?
• Do I have a team of at least three people assembled?
• Do I have time to go through as many search results as we might find?
• Do I have resources to get foreign-language articles appropriately translated?
• Do I have the statistical resources to analyze and pool data?

If you answered “No” to any of the first four questions, a traditional literature review will be more appropriate to do. If you answered “No” to the last question, a meta-analysis will not be an appropriate methodology for your review.