| |
The results of a systematic review of a health care
intervention are liable to be influenced by systematic error
(bias), as is the case with any observational study. There
is a risk of bias influencing a review if the
characteristics of, and distribution of reports (concerning
a particular treatment) within and outside the medical literature
are associated with the nature and direction (degree of
harm or benefit, statistical significance, precision) of
the results. In other words, there is a risk of bias at
the secondary level if account is not taken of bias in the
primary research literature.
Any action taken by the reviewer should be appropriate
to the type of bias and could involve, for example, a careful
search for trials from a certain source, or the exclusion
of trials from a certain chronological period, or a cautious
appraisal of results selectively reported by trialists.
Definitions
The characteristics of reports include the
outcome estimator used (e.g. Odds Ratio), the primary outcome
selected by the investigator (e.g. freedom from symptoms,
death) and the sub-group results actually appearing in the
report (e.g. patient status after three months, patients
under 40 years of age). The distributions of reports relates,
for example, to the differential publication and non-publication
of reports according to their results, to the accessibility
of reports (e.g. their appearance in some databases and
not in others), and to the appearance of reports in English
language journals or in languages other than English in
a pattern that is related to their results.
Types
of Bias
Bias has the potential to affect reviews of both experimental and
observational studies. Some of the different types of bias are
summarized below. However, work-to-date has focused mainly on the
reports of trials and the potential impact of bias on meta-analyses of
trials. A prime example is "publication bias", an issue in social
science research for over forty years (Sterling 1959; Smith 1980).
There is now considerable evidence that trials which are not formally
published ("grey trials"), and have to be accessed from sources such as
conference reports or contact with trialists, have results that differ
systematically (showing less benefit of treatment) from those published
in journals (Simes 1986; Dickersin 1990, Easterbrook 1991; Dickersin
1997; McAuley et al. 2000; Bartlett et al. 2000; Pham B 2000; Sterne et
al. 2000; Sutton et al. 2000). Trials with less beneficial results,
moreover, tend to take longer to achieve publication than trials with
more optimistic results ("time lag bias") (Stern and Simes 1997; Clarke
and Hopewell 2000). Many reviewers endeavour to avoid publication bias
by attempting to identify and include all relevant trials from the grey
literature. However, this is a time-consuming process and estimates of
the average impact of the grey literature have varied in magnitude.
Further investigation is required if pragmatic guidelines for reviewers
are to be drawn up.
Forms of bias are also connected with the
following: language of journals ("language bias") (Gregoire et al.
1995; Moher et al. 1996; Egger et al. 1997; Moher et al. 2000;
Jüni et al. 2000); with the source of funding for the primary
research ("funding bias") (Davidson 1986; Rochon et al. 1994; Cho and
Bero 1996); with the selective reporting of results within primary
studies ("outcome variable selection bias") (Hutton and Williamson
2000; Hahn et al. 2000; Hahn, Williamson and Hutton 2002)
(also known as "within-study reporting bias"); with the inclusion of
reports within some bibliographic databases and not within others
("database bias") (Zielinski 1995); with inconsistent coding within
databases ("coding bias"); and preferential citation of certain results
by scientific authors ("citation bias"). These last three biases are
also sometimes referred to as "retrieval biases" and a case might be
made for regarding them as constituting a class of bias distinct from
other types of reporting bias.
The nature of the results and outcome of a
study may also be associated with the geographical location in which
the study was undertaken or in which the researchers were based
(Vickers et al. 1998; Pittler et al. 2000). For example, it may also be
more difficult for studies from some regions, most notably developing
countries, to achieve full publication or publication in the most
accessible journals (Zielinski 1995; Wayt Gibbs 1995). "Regional Bias"
and "Developed Country Biases" may affect the results of some
systematic reviews and meta-analyses, and have significant consequences
for the practice of evidence-based health care around the world,
particularly in developing countries.
Complicating
Factors
Additional considerations for the reviewer are the questions of
whether bias and its impact varies by topic area, for example,
by
medical specialty, by type of intervention (e.g. drug or non-drug), or
according to whether the intervention is complementary or conventional
medicine, or according to the subject population (e.g. children or
adults). There is also the issue of whether a bias in primary
studies is inevitably associated with lack of quality. The level of
quality of primary research, (the level of bias in the design and
conduct of primary studies) is a major issue for Cochrane reviewers,
and investigations into bias will, of necessity, involve
consideration of the quality of primary studies.
Observational
Studies and Reviews without Large Meta-analyses
With regard to observational studies, our knowledge of the pattern of
bias and its potential impact on meta-analyses is markedly more limited
than is the case with trials. Knowledge is also very limited with
regard to the influence of bias on systematic reviews without
meta-analyses and on meta-analyses likely to contain a small number of
trials.
Reviewer
Bias
The BMG is concerned with the way factors relating to the results of
primary studies might lead to bias at the secondary level, that is, in
a systematic review. However, the BMG will also investigate
"Reviewer Bias" (Ernst 1994; Cates 1998), subjective bias introduced by
the systematic reviewer at the level of secondary research, in a way
that cannot be wholly attributed to the characteristics of the primary
studies. This consideration might be made because some of the
methodological approaches used would be similar to those used for
investigating primary reporting bias and it is possible that reviewer
bias could compound the effects of the reporting biases.
Types
of Reporting Bias & Definitions
|
Publication
Bias
|
|
(Positive results bias) The tendency
on the parts of the investigators, reviewers, and editors to submit or
accept manuscripts for publication based on the direction or strength
of the study findings (Dickersin 1990).
|
|
Language
Bias
|
|
Languages
of publication depend on the
direction and strength of the study results (Gregoire 1995).
|
|
Funding
Bias
|
|
The biases in the design, outcome, and
reporting of industry sponsored research in order to show that a drug
shows a favourable outcome (Lexchin 2003).
|
|
Outcome
Reporting Bias
|
|
A study in which multiple
outcomes were measured reports only those that are significant, than
those that were insignificant or unfavourable (Song 2000).
|
|
Database
Bias
|
|
Biased indexing of published studies
in literature databases (Felson 1992). The literature search will be
biased when it is based on a database in which the results of indexed
studies are systematically different from those of non-indexed studies
(Song 2000).
|
|
Study
Quality Bias
|
|
Studies of lower or higher quality are
associated with positive or favourable results.
|
|
Grey
Literature Bias
|
|
The results reported in journal
articles are systematically different from those presented in reports,
working papers, dissertations, or conference abstracts (Song 2000)
|
|
|
