The research methods section in any proposal must include a large number of design components. This Research Methods overview briefly describes the contents of each section.
The methods section of any proposal must address several fundamental design components. It helps to begin with a respecification of the research hypotheses. Then the research methods must (a) outline the design and present a timeline, (b) describe participant selection and recruitment, (c) explain the procedures for assignment to condition and methods for experimental control, (d) describe the independent variable, the intervention, (e) present the dependent variables or measures, (f) discuss data collection and management procedures, (g) provide the data analysis strategy, including a power analysis, if appropriate, and (h) address attrition and missing data.
Design and Timeline
The research design should include a general overview of the project. Consider this section as an abstract of the methods portion of the proposal, with a few additions. This section often include a figure that helps document when key events take place. These events may include recruitment, assignment to condition, intervention activities, assessments, and any other key features of the design that will help reviewers understand the research plan.
Often designs falls into a standard category, and it helps to explain such designs in standard terminology. Potential research designs include randomized controlled trials, nonequivalent groups designs, single-condition designs, clustered trials, regression discontinuity designs, single-subject trials, and so on. The details will vary substantially by design type. For example, a randomized trial can range from a post-only design to a longitudinal model with multiple assessments before, during, and after the intervention. Similarly, single-subject research covers a wide range of designs. This overview of the research design and all following sections must accommodate the specific research design type chosen for the project.
The choice of a design can be complicated. It often requires an iterative process comparing the advantages and disadvantages of each design in terms of participant selection and recruitment, the independent and dependent variables, analysis methods, and the budget. Because all these factors influence the overall design of the project, the decision process will benefit from experts in (a) the theory of the intervention and processes under study, (b) the pragmatic details of recruitment, intervention, and assessments, and (c) research methods, including design and statistics.
Participant Selection and Recruitment
Participant selection often begins with the identification of the population of interest. This section must then describe how project staff will select a sample and recruit participants.
The sample selection methods depend on the overall goal of the research project. For example, if the results must generalize to all similar people in the population, random sampling from the population will achieve that goal. On the other hand, the choice of a convenience sample would allow for the direct comparison of a specific intervention with a control group, and such a sample might involve reduced costs and simpler procedures. This section, however, should clearly describe the sample selection procedures and the number of students included in the sample.
This section must also include a clear description of the recruitment procedures. This includes information about how contacts are made, the type of consent process, if any, and related information that allows reviewers to judge the value of the final set of participants. Not all people identified as part of the sample will agree to participate, so this process should identify the number of participants expected to take part. This defines the initial sample. Finally, if the design calls for multiple assessments across time, this section should describe the expected rates of attrition. Although the analysis section will describe the details of the analysis in light of attrition, allusion to those methods here can provide the reader with a useful preview.
Assignment to Condition
For studies with more than one condition, assignment becomes an important feature of the research methods. In a randomized trial, research staff must place students into conditions randomly, and this section must state exactly how that will happen. There are many acceptable options, such as assignment via coin flip, a random number table, the use of a statistical program, the roll of dice, and so on. Nonrandomized comparison studies, such as the quasi-experimental nonequivalent groups design, also require assignment. Some researchers will order participants on a key variable of interest, and then randomly assign pairs, assuming two conditions, to treatment or control, working their way down the list. This is useful in small randomized trials or nonrandomized trials to ensure that the two groups are somewhat similar at the beginning of the project. Regression discontinuity designs also require assignment to condition. In this case, participants below (or above) a certain cut point on a predictor will receive treatment. Assignment to condition must be carefully specified and tied to the chosen design.
Experimental Control
Any experimental trial should attempt to control all influences on outcome measures. In a two-condition study, researchers should attempt to control for all differences between members of each condition other than those specified by the independent variable, the intervention. Randomization, for example, controls for the preexisting differences among participants in each condition. It allows for the theoretical assumption that participants in each condition do not differ at the onset of the study. It does not, however, control for differences during the study. If participants in the treatment condition, for example, receive instruction from two teachers and a computer and a single teacher provides instruction to the participants in the control condition, then the project has not established experimental control.
Controls other than assignment, then, can be very important. Participants in each condition should receive nearly identical treatment before and during the study, except for those differences associated with the independent variable. This includes demand characteristics of each condition, the format and structure of materials, the handling of participants by project staff, and so on. Methods for experimental control will differ substantially by the type of research design.
Independent Variable
The independent variable (IV) defines the intervention conditions. A condition represents a set of participants who receive one type of treatment. In a typical randomized controlled trial with two conditions, one condition, the treatment group, will receive an intervention or treatment. The other condition, the control group, would receive usual care or possibly a placebo to control for demand expectancy. The description of the intervention should be thorough. It should include every way that the experimenter manipulates the participants. The control group must also be clearly described. Does the study call for a placebo or a less rigorous comparison treatment? And how does the control group differ from the treatment group?
Thus, the section about the intervention or independent variable should include a discussion of the intervention condition, a clear description of the control condition, and an explanation of how they differ. Again, these vary by the research design, but most designs must include some control condition, also called the counterfactual, and an intervention of some kind. In a single-subject ABAB design, the period of time in the A condition is considered the control phase, where only observations take place. The B condition represents the phase where the treatment is applied. A description of the independent variable, then, would describe the A and B phases as well as how the investigator would transition a single participant between the two.
Dependent Variables and Other Measures
Proposals often include dependent variables (DVs), or outcome measures, as well as all other assessments in a single “measures” section. Measures may be standardized measures, custom tests, study-specific measures, end-of-chapter tests, time to complete tasks or subtasks, observations, and so on. Depending on the design type, measures should include (a) the dependent variables, the outcomes that the research project should change, (b) moderators, often preexisting conditions for which the intervention might work differentially, (c) covariates, such as pretest measures or demographic variables, (d) blocking variables, which are usually moderators, (e) matching variables, used to match participants before assignment to condition, and (f) mediators, variables that intervene between condition and outcomes.
Moderators generally include demographic characteristics or preexisting conditions within levels of which the intervention may work differently. For example, some interventions might work differently for students with a reading disability than students without. Thus, an the presence of an IEP for a reading disability would represent one moderating variable. Although not often ideal, gender might moderate the intervention effect if, for example, incentives in the intervention condition appeal better to girls than boys.
The measures section should also include mediators. Mediators are intervening variables, and treatment fidelity and dosage represent two common mediators. Mediators are often those variables that an intervention is expected to directly impact. In a study testing an intervention intended to provide additional instructional supports, the number of instructional supports used by a student could be one mediating variable. Students who received the intervention should clearly use more instructional supports, but those in the control condition might still have access to some. Nonetheless, condition should clearly predict the differential use of instructional supports. If the study hypothesizes that instructional supports should lead to improved reading, then we would expect instructional supports to predict improved reading. If instructional supports truly mediates the relationship between condition and reading, however, we would then expect instructional supports, as a predictor of reading, to supplant condition as a predictor.
For all measures, investigators should describe them and provide reliability and validity statistics. These can come from either previously published research, as with standardized measures, or pilot work, which might be demonstrated within the proposal. Some measures will not have reliability and validity data, such custom, study-specific tests and measures. This section should then provide a detailed description of the development procedures and psychometric criteria for establishing reliability and validity.
Data Collection and Management
Many investigators describe, often briefly, their data collection and data management procedures. For example, how will research staff track participants across time, maintain data integrity and security, manage demographic information, organize the data, and so on? Participant may be tracking with identification numbers, with data organized into a set of spreadsheets or a database. Often identification numbers are stored separately from participant data, as are consents. This section allows the team of investigators to establish that they know how to work with data and keep it secure.
Analysis Methods
Data analysis methods vary considerably from and even within the types of research designs. Some methods, such as single-subject designs, do not necessarily need a statistical analysis to convey experimental control over the dependent variables. Most “quantitative” designs, such as randomized trials and many quasi-experimental designs, require a statistical analysis. Quantitative designs can vary from one or two assessments in time to longitudinal data collection with numerous assessments across several years. They might assign individual participants to condition or intact clusters of participants, such as classrooms or schools. The analyses may need to compare two or more conditions or pretest assessments to posttest assessments.
In general, the statistical analysis must answer the research questions or address the research hypotheses in a manner that accounts for the overall design of the study. Thus, an analysis section should (a) state or paraphrase the research questions or hypotheses, (b) review key features of the design, (c) describe the analytical approach in detail, (d) address the number and type of tests and study-wide Type I error rate, (e) report a power analysis and describe all assumptions, (f) express how the analysis will account for attrition and other missing data, and (g) list the software that the analysts expects to use for the analyses.
Disclaimer
Please keep in mind that these templates have been provided for members of the NCSeT community. We hope to help NCSeT members develop a strong line of research across a number of related areas. Because the materials presented here represent the intellectual property of Dr. Keith Smolkowski and NCSeT, please do not distribute these materials without permission.
© 2007 Keith Smolkowski