Guidance for systematic reviews

This guide aims to round up the most frequently asked questions (FAQs) on conducting systematic reviews

Building search plan

Systematic reviews seek to collate evidence that fits pre-specified eligibility criteria in order to answer a specific research question. They aim to minimise bias by using explicit systematic methods documented in advance with a protocol.

The PICO framework is used the most commonly for health-related reviews. These components + types of studies that will be included form the basis of the eligibility criteria for the review.

P: Population Target population: this group can be the general population or a specific group defined by age, socioeconomic status, risk factors or location
I: Intervention Intervention of interest: this can be therapy, test, or strategy to be investigated
C: Comparator Comparator group to compare results against such as placebo or control groups
O: Outcome Outcomes of interest

For instance, an intervention review will compare two or more interventions or a range of different outcomes or compare a particular intervention with no intervention at all. Then your search would be formed to answer the research questions below:

1) What is the value of intervention X compared with intervention Y for a particular group of people?

2) What is the value of intervention X compared with controls?

To find the relevant systematic reviews on a research area of interest, you can also find them using PICO filter in Cochrane library, which allows users to filter into systematic reviews based on specific conditions, interventions, outcomes or patient population: Cochrane PICO search | Cochrane Library 

Alternative frameworks could be considered depending on your research topic.

PECO Population | Environment | Comparison | Outcome Effect of exposure
SPICE Setting | Population | Intervention | Comparison | Evaluation PICO+setting
CIMO Context | Intervention | Mechanisms | Outcome Management and organisation studies
ECLIPSE Expectation | Client group | Location | Impact | Professionals | Service

Health policy/ Management searches
SPIDER Sample | Phenomenon of Interest | Design | Evaluation | Research Type Qualitative and Mixed-methods research

Others: Here is a guide to question frameworks: PICOS/T, PEO, PICo

For more guidance on identifying synonyms and alternative terms, see the LibSmart I course, and Module 5: Health Information here in LibSmart II. Remember, not all essential concepts are good search terms. Some may work better when they have applied during screening as selection (inclusion/exclusion) criteria.

Before finalising your search plan, it will help to read good-quality published systematic reviews on topics related to your own research question to see what methods they employed, and you’ll need to do quite a bit of exploratory testing.

  1. Write down your systematic review research question.
  2. What type(s) of data do you need to address your question?
  3. What study design(s) would yield those data? Or what types of publications or reports are most likely to contain the data you need to address your question?
  4. Where are those publications or reports likely to be published? e.g in which journals, or by which organisations?
  5. If you think there might be unpublished but relevant reports, which organisations or people do you think would know about them?
  6. What literature databases, research registers, or organisation websites are most likely to contain the records (or full reports) of the publications you need?

Reference: Chapter 1.5 Protocol development. Cochrane training

Because comprehensiveness is so important, searchers look at several databases. In every review, searches used to find clinical trials, for instance, are documented for each of the databases searched.

Reproducibility of search results is an important criterion of searches and is one of the reasons for documenting.

  • The ASL developed “Subject guides” to help you get the best out of the library with information on finding academic literature, referencing and more. The subject guides list relevant resources for each research subject and useful courses (Subject guide)
  • Or you use multiple databases (access via Databases A-Z webpage), as they index different journals (Databases A-Z)

Key databases

* PROQUEST and OVID are interfaces/platforms rather than databases. Ovid provided Medline, Embase, CAB abstracts for the University

Search strategy

This training video demonstrates how to build complex searches

Once you have identified the literature databases most relevant to your search, you need to become familiar with:

  • The information included in each database's records
  • The available searchable fields
  • Whether there is a useful controlled thesaurus from which subject headings are used to "tag" records of articles
  • The search interface (bear in mind some databases are available from more than one platform: use the platform with the features most useful to your review)
  • What search functions are available (e.g. phrase searching)
  • The syntax to use for the search functions you need
  • What options there are for exporting results in bulk

To learn about these aspects, you need to spend time exploring and testing searches in each database. You will also need to check the Help documentation for each database. You may want to use tutorials and other support documentation provided by each database publisher.

You can practice translating a search in LibSmart II model systematic review.

Useful Resources

 

Q. What is a subject heading?

A subject heading is like "a tag" or "a label" of a topic/item in a book or an article. Subject headings are in general, words or short phrases that can represent important concepts of literature in the database. For instance, when a paper about “health-related behaviours of people with diabetes” enters a database, an indexer will decide which topics are coved by the article and choose several subject headings to be tagged.

Subject headings are the way databases such as MEDLINE or CINAHL Plus organise and categorise their records. Not all databases have subject headings. 

They are added to records to describe the content of the article a record describes and because they are consistently applied, they smooth out variations in terminology (e.g., changes over time, geography, discipline and author differences in how concepts are described). 

Subject headings are indexed in the databases' thesauri. The thesauri are searchable, allowing you to find the subject headings related to your search terms and from which all the articles assigned, that heading can be retrieved.

Databases for health subjects make greater use of subject headings than databases for some other subjects and learning how to use subject headings will greatly improve the relevance of your search results.

As described above, the subject headings encompass a standard list of terms or thesaurus to be used. Here is an example in Ovid EMBASE. When the paper “Rosiek A et al. (2016) Health behaviours of patients diagnosed with type 2 diabetes mellitus and their influence on the patient satisfaction with life. Ther Clin Risk Manag.” was searched in Ovid EMBASE, you can see the following subject headings:

 

So if you included a subject heading “health behaviour” in your search, then the search is likely to retrieve this paper. What if you included another subject heading “quality of life”? Again, your search will pick up this paper.

The subject heading is also known as a “controlled vocabulary”, which means that all items (articles or books) about “Diabetes” for instance, would be tagged with the same, standard subject heading “Diabetes Mellitus” regardless of the terms or phrases that were used in the title or abstract.

Here is an example. Suppose you’re looking for articles about “health behaviour” and then you search for “health behavio?r*” in the title and abstract in Ovid Embase (the wildcard symbol (?) can be used as a substitute for one character or none. It is useful for retrieving records with British and American spelling variations. So here, it will retrieve records containing “behaviour” or “behavior”).

There are 32544 articles with the word “health behaviour” or “health behaviour” in the title or abstract.

What if you search for the correct subject heading for health behaviour?

Then you will have 466382 articles. As you can see, the subject heading exploded to retrieve all results using the selected term and all of its more specific terms. Then 466382 articles were retrieved here.

Therefore, we always recommend using both the most relevant subject headings as well as free-text searches that reflect usage/terminology to get the best from both the database’s standardised descriptors as well as authors’ words from fields like title and abstract. If a database has subject headings (e.g., Ovid EMBASE, Ovid MEDLINE, etc.), then use them! There’s no good reason not to utilise subject headings. And then adding free-text searches that reflect usage/terminology would increase the sensitivity of your search.

  • Please note that the thesaurus (list of subject headings) will be different for each database. For example, the subject heading for sociocultural factors is “Sociocultural Factors” in PsycINFO whist no subject heading in Medline.
  • Sometimes, available subject headings are not precise enough or do not reflect variation or range in the terminology used by authors. In addition, sometimes the most appropriate subject headings are not applied to article records.
  • Sometimes, subject heading hierarchies use highly specific terms that can be more easily captured with subject heading searches.
  • Subject headings should be normally truncated unless included as part of a free test term search

Proximity operators are sometimes called adjacency operators. They function as precision-maximisers as they enable you to define how closely you want your search terms to be found in relation to one another. 

Proximity searching can be important when there’s a possibility of having another word between two words.

For example, some papers might use other terms “social psychological determinants”, “social structural determinants” or “social-health determinants”. To deal with such variations, we can use ADJn (e.g., adj3) between two search terms.

For instance, "social" ADJ2 "determinants" on the Ovid platform, will search the following search terms.

“social determinants”

“social psychological determinants”

“social structural determinants”

 “social-health determinants”.

 

Useful Syntaxes: 

1. Ovid: ADJn (ADJ=adjacency),

n specifies the number of words between two search terms in any order.

2. EBSCO: Nn (N=Near), Wn (W=Within),

N searches for instances of the search terms in any order;

W searches for instances of the search terms in the specified order,

n specifies the number of words between the search terms.

3. ProQuest: NEAR/n (or n/n), PRE/n (or p/n)

NEAR searches for instances of the search terms in any order;

PRE searches for instances of the search terms in the specified order;

if n isn't specified, 4 is set by default.

4. Cochrane library: NEAR/n, NEXT

n specifies the number of words between the search terms.

if n isn't specified, 6 is set by default.

NEXT searches for instances of the search terms next to each other and in the specified order

5. Web of Science: NEAR/n

           n specifies the number of words between the search terms.

if n isn't specified, 15 is set by default.

 

Please note that not all databases allow proximity operators. 

If you have too many/small records, it could be due to a number of reasons. There are several points you might need to check.

  1. Did you apply filters?
  • Please do comparisons of the results before/after applying filters.
  • Remove unnecessary filters
  • Reconsider applying filters 
  1. Your search terms might be too broad/narrow
  • Include some synonymous keywords
  • Use truncation to include word variations
  • Consider British/American English spellings
  • Consider various locations of a hyphen.
  • Spell out abbreviations and include abbreviations
  1. Did you choose appropriate databases?

Peer review of search strategies is recommended by many organisations including Cochrane.

PRESS checklist for appraisal of searches

The PRESS 2015 Guideline Statement is helpful to guide and improve the peer review of electronic literature search strategies. The aim of this checklist is to attain comprehensiveness of coverage while maintaining a moderate degree of precision in the records retrieved (McGowan et al., 2016). There are six domains and the following questions that you should consider: 

Table 1. PRESS 2015 Guideline Recommendations (cited from Table 2. McGowan et al., 2016)
Domains Questions
Translation of the research question
  1. Does the search strategy match the research question/PICO?
  2. Are the search concepts clear?
  3. Are there too many or too few PICO elements included?
  4. Are the search concepts too narrow or too broad?
  5. Does the search retrieve too many or too few records? (Show the number of hits per line)
  6. Are unconventional or complex strategies explained?

Boolean and proximity operators (these vary based on search service)
  1. Are Boolean or proximity operators used correctly?
  2. Is the use of nesting with brackets appropriate and effective for the search?
  3. If NOT is used, is this likely to result in any unintended exclusions?
  4. Could precision be improved by using proximity operators (eg, adjacent, near, within) or phrase searching instead of AND?
  5. Is the width of proximity operators suitable (eg, might adj5 pick up more variants than adj2)?
Subject headings (database specific)
  1. Are the subject headings relevant?
  2. Are any relevant subject headings missing; for example, previous index terms?
  3. Are any subject headings too broad or too narrow?
  4. Are subject headings exploded where necessary and vice versa?
  5. Are major headings (“starring” or restrict to focus) used? If so, is there adequate justification?
  6. Are subheadings missing?
  7. Are subheadings attached to subject headings? (Floating subheadings may be preferred.)
  8. Are floating subheadings relevant and used appropriately?
  9. Are both subject headings and terms in free text (see the following) used for each concept?
Text word searching (free text)
  1. Does the search include all spelling variants in free text (eg, UK vs. US spelling)?
  2. Does the search include all synonyms or antonyms (eg, opposites)?
  3. Does the search capture relevant truncation (ie, is truncation at the correct place)?
  4. Is the truncation too broad or too narrow?
  5. Are acronyms or abbreviations used appropriately? Do they capture irrelevant material? Are the full terms also included?
  6. Are the keywords specific enough or too broad? Are too many or too few keywords used? Are stop words used?
  7. Have the appropriate fields been searched; for example, is the choice of the text word fields (.tw.) or all fields (.af.) appropriate? Are there any other fields to be included or excluded (database specific)?
  8. Should any long strings be broken into several shorter search statements?
Spelling, syntax, and line numbers
  1. Are there any spelling errors?
  2. Are there any errors in system syntax; for example, the use of a truncation symbol from a different search interface?
  3. Are there incorrect line combinations or orphan lines (ie, lines that are not referred to in the final summation that could indicate an error in an AND or OR statement)?
Limits and filters
  1. Are all limits and filters used appropriately and are they relevant given the research question?
  2. Are all limits and filters used appropriately and are they relevant for the database?
  3. Are any potentially helpful limits or filters missing? Are the limits or filters too broad or too narrow? Can any limits or filters be added or taken away?
  4. Are sources cited for the filters used?