This template provides an analysis plan for a randomized controlled trial using an analysis of covariance (ANCOVA) as the analytical method.
This template provides an analysis plan for a randomized controlled trial using an analysis of covariance (ANCOVA) as the analytical method. I have placed some text within brackets. These are places where I usually provide text specific to the current study. In brackets, I have attempted to provide, first, a description of the type of information needed and, second, an example from an actual study. For example, the section might refer to the [dependent variable, e.g., reading comprehension]. The bracket comment should be replaced with specific information. Reviewers do not want to read textbook descriptions of analyses. They expect to see how an analysis method will apply specifically to the project at hand.
Analysis Plan
To examine differences between the [intervention and control conditions, e.g., support electronic texts (e-texts) versus usual presentation], we will conduct a randomized trial, advocated as an important standard for research designs (Shadish, Cook, & Campbell, 2002; Society for Prevention Research, 2004). If [key outcomes improve, e.g., if student comprehension improves] for participants in [the intervention condition, e.g., supported e-text condition] over the control students, we infer a functional relationship between [the independent variable, e.g., e-text supports] and [dependent variables, e.g., student reading comprehension]. Within this section, we outline the analysis plan for this experiment.
Statistical Analysis
To compare conditions, we will use analysis of covariance (ANCOVA) procedures. These methods are well known and detailed in many statistics texts (e.g., Cohen, Cohen, West, & Aiken, 2003; Hays, 1988; Tabachnick, & Fidell, 1996). When we compare conditions, our primary interest will revolve around tests of participants who received [the intervention, e.g., e-text supports] versus [controls, e.g., those who received no electronic supports]. Thus, we will test whether outcome measure[s] differ between conditions controlling for baseline [measures, e.g., reading performance] and other covariates [e.g., fluency with the English language]. Covariates, in an ANCOVA, serve to reduce the variability of the outcome measures and, hence, increase the power of the statistical tests.
Analyses will also explore mediational influences that correspond to [dosage or implementation fidelity of the independent variable, e.g., usage pattern of eText supports] (Baron & Kenny, 1986; Judd & Kenny, 1981). Assignment to a treatment condition, by itself, will not directly affect [the dependent measures, e.g., student comprehension]. “Treatment” only influences [the dependent measure(s), e.g., reading comprehension] through [a certain dosage or degree of implementation of the intervention, e.g., students’ use of e-text supports]. Thus we will estimate the degree to which measures of [the mediator, e.g., the use of the e-text supports, such as the electronic transcripts of student eText interactions,] accounts for the effects attributed to the particular [condition, e.g., e-text support condition]. This mediational analysis formally establishes the functional relationship between [the independent variable, e.g., eText supports] and [the dependent variable, e.g., improved comprehension], ruling out influences attributable to other causes or simply chance (Kraemer, Wilson, Fairburn, & Agras, 2002). This type of analysis also allows the investigation of [dose response, treatment integrity, or treatment fidelity, whichever topics have not already been covered] (Kazdin, 1992).
Power Analysis
For this ANCOVA approach, we calculated the power to detect differences between any two conditions in terms of [effect sizes or a raw, understandable metric, such as reading rate per minute] (Cohen, 1988; Rosenthal, Rosnow, & Rubin, 2000). The calculations assumed a single outcome, a Type I error rate (a) of .05, a Type II error rate (ß) of .20 (power of .80), and a two-tailed statistical test, as is appropriate for research purposes (Fleiss, 1981). With [number, e.g., 50] participants per condition and no effective covariate, these experiments could detect an effect size of .57. The effect sizes tell us the smallest difference we can detect in standard deviations units between students who receive [the intervention, e.g., supported text] and those that do not. Cohen (1988) refers to this effect size as d and defines a medium effect size as .50, a small effect size as .20, and a large effect size as .80. Most likely, pretest measures will account for some variability between individuals and improve power. With covariates of .40 or .60 (correlation values), this experiment will detect effect sizes of .52 or .45, respectively. We believe these effect sizes are small enough that the [intervention, e.g., e-text supports] can produce these differences between conditions, but not so small as to be [clinically, practically, educationally] irrelevant.
[In the above paragraph, we have assumed a single test. Power, or the ability to detect true effects, is influenced by the number of tests. A study with multiple tests will inflate the opportunity for Type I errors. To remedy this problem, divide the study-wide Type I error rate by the number of tests and then specify that value as the individual-test a. For example, with 5 dependent measures, replace the second sentence in the above paragraph with the following: “These calculations assumed five outcomes, a study-wide Type I error rate of .05, and hence, a single test a of .05 / 5 or .01. We also assumed a Type II error rate (ß) of .20. . . .”]
Attrition and Missing Data
Participant attrition, also called experimental mortality, causes difficulties in research that requires repeated measurement over time and poses a threat to both external and internal validity of a study (Barry, 2005; Shadish, Cook, & Campbell, 2002). We expect [justifiable amount, e.g., very little] attrition among [participants, e.g., students or teachers], and we do not expect [participant, e.g., student or teacher] attrition to vary by treatment condition. In light of attrition, however, we will attempt to identify key predictors of attrition status, such as [example predictors, e.g., SES, reading level], and test for differences between conditions.
When participant attrition is discovered, several procedures offer effective approaches that may attenuate the problem. Maximum likelihood models, with time as a random variable, allow the use of all available data from all assessments, reducing bias and increasing power (Nich & Carroll, 1997). There are three other accepted approaches to analyses with missing data: (a) multiple imputation procedures like EMCOV (Graham, Hofer, & Piccinin, 1994) which utilize the expectation-maximization (EM) algorithm with bootstrap estimates of standard errors, (b) raw maximum-likelihood analysis, or (c) multiple-group structural equation or latent growth modeling (Muthén, et al., 1987; Duncan & Duncan, 1995). For manifest variable models, these methods produce virtually identical results and will be used to address attrition. In general, the application of these missing data procedures can provide unbiased conclusions, even in the face of substantial attrition (Graham & Donaldson, 1993).
These methods all assume that the data that are missing at random, that the missing data mechanism does not depend on the unobserved determinants of the outcome of interest (Little, 1995; Little & Rubin, 1987, 1989). If this assumption is invalid, then the missing data mechanism is described as non-ignorable. In that case, potential remedies are much more difficult. However, recent methods have been developed that can help the researcher determine whether the missing at random assumption fits the data. These methods were developed by Verbeek and Nijman (1992) and can provide some indication of whether the missing data mechanism is ignorable. An illustration of these methods can be found in Foster and Bickman (1996). We will make every attempt to treat missing data appropriately using these methods.
Software
All major data analysis packages as well as spreadsheets, such as Microsoft Excel, can perform a simple analysis of covariance. We plan to conduct these analyses in [pick appropriate software: SAS (SAS Institute, 2006), SPSS (SPSS Inc., 2005), Stata (need reference), Excel with the Analysis Tookpak that performs multiple regression.]
Disclaimer
Please keep in mind that these templates have been provided for members of the NCSeT community. We hope to help NCSeT members develop a strong line of research across a number of related areas. Because the materials presented here represent the intellectual property of Dr. Keith Smolkowski and NCSeT, please do not distribute these materials without permission.
© 2007 Keith Smolkowski