The purpose of this dictionary is to describe items in the tool thereby assisting raters to score study quality. Due tounder-reporting or lack of clarity in the primary study, raters will need to make judgements about the extent that bias may be present. When making judgements about each component, raters should form their opinion based upon information contained in the study rather than making inferences about what the authors intended.
A) SELECTION BIAS
(Q1) Participants are more likely to be representative of the target population if they are randomly selected from a comprehensive list of individuals in the target population (score very likely). They may not be representative if they are referred from a source (e.g. clinic) in a systematic manner (score somewhat likely) or self-referred (score not likely).
(Q2) Refers to the % of subjects in the control and intervention groups that agreed to participate in the study before they were assigned to intervention or control groups.
B) STUDY DESIGN
In this section, raters assess the likelihood of bias due to the allocation process in an experimental study. For observational studies, raters assess the extent that assessments of exposure and outcome are likely to be independent. Generally, the type of design is a good indicator of the extent of bias. In stronger designs, an equivalent control group is present and the allocation process is such that the investigators are unable to predict the sequence.
Randomized Controlled Trial (RCT)
An experimental design where investigators randomly allocate eligible people to an intervention or control group. A rater should describe a study as an RCT if the randomization sequence allows each study participant to have the same chance of receiving each intervention and the investigators could not predict which intervention was next. If the investigators do not describe the allocation process and only use the words ‘random’ or ‘randomly’, the study is described as a controlled clinical trial.
See below for more details.
Was the study described as randomized?
Score YES, if the authors used words such as random allocation, randomly assigned, and random assignment.
Score NO, if no mention of randomization is made.
Was the method of randomization described?
Score YES, if the authors describe any method used to generate a random allocation sequence.
Score NO, if the authors do not describe the allocation method or describe methods of allocation such as alternation, case record numbers, dates of birth, day of the week, and any allocation procedure that is entirely transparent before assignment, such as an open list of random numbers of assignments.
If NO is scored, then the study is a controlled clinical trial.
Was the method appropriate?
Score YES, if the randomization sequence allowed each study participant to have the same chance of receiving each intervention and the investigators could not predict which intervention was next. Examples of appropriate approaches include assignment of subjects by a central office unaware of subject characteristics, or sequentially numbered, sealed, opaque envelopes.
Score NO, if the randomization sequence is open to the individuals responsible for recruiting and allocating participants or providing the intervention, since those individuals can influence the allocation process, either knowingly or unknowingly.
If NO is scored, then the study is a controlled clinical trial.
Controlled Clinical Trial (CCT)
An experimental study design where the method of allocating study subjects to intervention or control groups is open to individuals responsible for recruiting subjects or providing the intervention. The method of allocation is transparent before assignment, e.g. an open list of random numbers or allocation by date of birth, etc.
Cohort analytic (two group pre and post)
An observational study design where groups are assembled according to whether or not exposure to the intervention has occurred. Exposure to the intervention is not under the control of the investigators. Study groups might be non-equivalent or not comparable on some feature that affects outcome
Case control study
A retrospective study design where the investigators gather ‘cases’ of people who already have the outcome of interest and ‘controls’ who do not. Both groups are then questioned or their records examined about whether they received the intervention exposure of interest.
Cohort (one group pre + post (before and after)
The same group is pretested, given an intervention, and tested immediately after the intervention. The intervention group, by means of the pretest, act as their own control group.
Interrupted time series
A time series consists of multiple observations over time. Observations can be on the same units (e.g. individuals over time) or on different but similar units (e.g. student achievement scores for particular grade and school). Interrupted time series analysis requires knowing the specific point in the series when an intervention occurred.
C) CONFOUNDERS
By definition, a confounder is a variable that is associated with the intervention or exposure and causally related to the outcome of interest. Even in a robust study design, groups may not be balanced with respect to important variables prior to the intervention. The authors should indicate if confounders were controlled in the design (by stratification or matching) or in the analysis. If the allocation to intervention and control groups is randomized, the authors must report that the groups were balanced at baseline with respect to confounders (either in the text or a table).
D) BLINDING
(Q1) Assessors should be described as blinded to which participants were in the control and intervention groups. The purpose of blinding the outcome assessors (who might also be the care providers) is to protect against detection bias.
(Q2) Study participants should not be aware of (i.e. blinded to) the research question. The purpose of blinding the participants is to protect against reporting bias.
E) DATA COLLECTION METHODS
Tools for primary outcome measures must be described as reliable and valid. If ‘face’ validity or ‘content’ validity has been demonstrated, this is acceptable. Some sources from which data may be collected are described below:
Self reported data includes data that is collected from participants in the study (e.g. completing a questionnaire, survey, answering questions during an interview, etc.)
Assessment/Screening includes objective data that is retrieved by the researchers. (e.g. observations by
investigators)
Medical Records/Vital Statistics refers to the types of formal records used for the extraction of the data.
Reliability and validity can be reported in the study or in a separate study. For example, some standard assessment tools have known reliability and validity.
F) WITHDRAWALS AND DROP-OUTS
Score YES if the authors describe BOTH the numbers and reasons for withdrawals and drop-outs.
Score NO if either the numbers or reasons for withdrawals and drop-outs are not reported.
The percentage of participants completing the study refers to the % of subjects remaining in the study at the final data collection period in all groups (i.e. control and intervention groups).
G) INTERVENTION INTEGRITY
The number of participants receiving the intended intervention should be noted (consider both frequency and intensity). For example, the authors may have reported that at least 80 percent of the participants received the complete intervention. The authors should describe a method of measuring if the intervention was provided to all participants the same way. As well, the authors should indicate if subjects received an unintended intervention that may have influenced the outcomes. For example, co-intervention occurs when the study group receives an additional intervention (other than that intended). In this case, it is possible that the effect of the intervention may be over-estimated. Contamination refers to situations where the control group accidentally receives the study intervention. This could
result in an under-estimation of the impact of the intervention.
H) ANALYSIS APPROPRIATE TO QUESTION
Was the quantitative analysis appropriate to the research question being asked?
An intention-to-treat analysis is one in which all the participants in a trial are analyzed according to the intervention to which they were allocated, whether they received it or not. Intention-to-treat analyses are favoured in assessments of effectiveness as they mirror the noncompliance and treatment changes that are likely to occur when the intervention is used in practice, and because of the risk of attrition bias when participants are excluded from the analysis.
Component Ratings of Study:
For each of the six components A – F, use the following descriptions as a roadmap.
A) SELECTION BIAS
Strong: The selected individuals are very likely to be representative of the target population (Q1 is 1) and there is greater than 80% participation (Q2 is 1).
Moderate: The selected individuals are at least somewhat likely to be representative of the target population (Q1 is 1or 2); and there is 60 – 79% participation (Q2 is 2). ‘Moderate’ may also be assigned if Q1 is 1 or 2 and Q2 is 5 (can’t tell).
Weak: The selected individuals are not likely to be representative of the target population (Q1 is 3); or there is less than 60% participation (Q2 is 3) or selection is not described (Q1 is 4); and the level of participation is not described (Q2 is 5).
B) DESIGN
Strong: will be assigned to those articles that described RCTs and CCTs.
Moderate: will be assigned to those that described a cohort analytic study, a case control study, a cohort design, or an interrupted time series.
Weak: will be assigned to those that used any other method or did not state the method used.
C) CONFOUNDERS
Strong: will be assigned to those articles that controlled for at least 80% of relevant confounders (Q1 is 2); or (Q2 is 1).
Moderate: will be given to those studies that controlled for 60 – 79% of relevant confounders (Q1 is 1) and (Q2 is 2).
Weak: will be assigned when less than 60% of relevant confounders were controlled (Q1 is 1) and (Q2 is 3) or control of confounders was not described (Q1 is 3) and (Q2 is 4)
D) BLINDING
Strong: The outcome assessor is not aware of the intervention status of participants (Q1 is 2); and the study participants are not aware of the research question (Q2 is 2).
Moderate: The outcome assessor is not aware of the intervention status of participants (Q1 is 2); or the study participants are not aware of the research question (Q2 is 2); or blinding is not described (Q1 is 3 and Q2 is 3).
Weak: The outcome assessor is aware of the intervention status of participants (Q1 is 1); and the study participants are aware of the research question (Q2 is 1).
E) DATA COLLECTION METHODS
Strong: The data collection tools have been shown to be valid (Q1 is 1); and the data collection tools have been shown to be reliable (Q2 is 1).
Moderate: The data collection tools have been shown to be valid (Q1 is 1); and the data collection tools have not been shown to be reliable (Q2 is 2) or reliability is not described (Q2 is 3).
Weak: The data collection tools have not been shown to be valid (Q1 is 2) or both reliability and validity are not described (Q1 is 3 and Q2 is 3).
F) WITHDRAWALS AND DROP-OUTS – a rating of:
Strong: will be assigned when the follow-up rate is 80% or greater (Q2 is 1).
Moderate: will be assigned when the follow-up rate is 60 – 79% (Q2 is 2) OR Q2 is 5 (N/A).
Weak: will be assigned when a follow-up rate is less than 60% (Q2 is 3) or if the withdrawals and drop-outs were not described (Q2 is 4)