Ninth Edition CoverGraziano & Raulin
Research Methods (9th edition)

Chapter 9 Summary
Controls to Reduce Threats to Validity

General Control Procedures

We define control procedures somewhat broadly as any procedure used by the researcher to counteract potential threats to the validity of the research. Control is achieved when we are able to rule out alternative explanations of results in order to enhance both construct and internal validity and when we reduce barriers to generalization, thereby improving external validity. General control procedures include (1) preparation of the setting, (2) response measurement, and (3) replication.

Preparation of Setting

The most general control procedures are those that clearly define the setting in which the research is to take place. The advantage of the laboratory setting over low-constraint settings is that many extraneous variables can be eliminated. However, controlling the setting may increase internal validity, but there is a risk that it might decrease external validity. This effect can be reduced, for example, by creating a more natural setting in the laboratory.

Response Measurement

Careful selection and preparation of the instruments used to measure the dependent variable will affect both statistical and construct validity.

Replication

Replication is another general control procedure. Exact replication refers to repeating an experiment exactly as it had been carried out before to see if the same results occur. Systematic replication and conceptual replication involves testing some systematic, theoretical, or procedural modification of the original work.

Control over Subject and Experimenter Effects

Experimenter and subject effects are serious threats to validity that must be controlled in virtually all experimentation. Among the useful controls are single-blind and double-blind procedures, automation, use of objective measures, multiple observers, and use of deception.

Single- and Double-blind Procedures

To control for experimenter effects, we reduce the researcher's direct contact with participants, the researcher's knowledge about the conditions under which each participant is tested, and the researcher's knowledge of the hypotheses being tested. This can be done by using an assistant who is blind to those factors (i.e., a single-blind procedure). When both the assistant and participants are blind to these factors, we have even greater control (a double-blind procedure).

Automation

Another way to reduce contact between the experimenter and the participants is to standardize and automate instructions to participants and the procedures for obtaining and recording the responses.

Using Objective Measures

Using objective rather than subjective measures is an important control. Objective measures precisely define the behaviors to be observed and require minimal judgments on the part of the observer. That is, objective measures are based on empirically observable and clearly specified events about which two or more people can easily agree.

Multiple Observers

When there may be questions about objectivity in making observations, multiple observers can be used and their observations compared for agreement.

Using Deception

The most common control for subject effects is to obscure the true hypothesis by deliberately misinforming participants about the true nature of the experiment--that is, using deception. To answer ethical objections to deception, we must fully debrief participants after the experiment is complete. 

Control through Participant Selection and Assignment

The manner in which participants are elected and assigned to groups can affect both the internal and external  validity of a study.

Participant Selection

Careful participant selection ensures external validity and allows us to generalize the results to a population. A population is the larger group of all events of interest from which the sample is selected. The sample is a smaller number of events drawn from the population and used in a specific study as if the sample adequately represented the entire population. The target population is the population in which we are ultimately interested. The accessible population is that set of events available to us, from which we can draw the sample. Most research samples are drawn from accessible populations rather than from the larger populations.

  • Random Sampling. Random sampling means drawing a sample from a population in such a way that: (1) every member of the population has an equal chance of being selected for the sample, and (2) the selections are independent. Numbers for random samples are typically drawn from a table of random numbers or a random number generator.
  • Stratified Random Sampling. Stratified random sampling is a variation of random sampling in which random selection of participants from a population is carried out by drawing randomly from each of several subsets of the population.
  • Ad Hoc Samples. At times we cannot randomly select participants and must use whatever participants are at hand (termed an ad hoc sample). We can use an ad hoc sample if we describe the sample's characteristics and limit generalizations to other people (events, etc.) that have the same characteristics. In human research, we measure such characteristics as sex, age, and socioeconomic class to describe the ad hoc sample.

Participant Assignment

Once participants have been selected from a population or an ad hoc sample has been created, they must then be assigned to the conditions of the experiment. Unbiased participant assignment is critical in all experiments.

  • Free Random Assignment. Free random assignment of participants to conditions is carried out by using a table of random numbers or a random number generator to assign participants.
  • Randomize within Blocks. In this procedure, we randomly assign a block of participants to the groups until every group receives one participant. Then we repeat the procedure with the next block.
  • Matched Random Assignment. In matched random assignment, we decide what variables to use in matching participants, then match them, and then use a table of random numbers or random number generator to randomly assign the first participant of the pair to one of the conditions, with the other matched participant of the pair automatically assigned to the other condition.
  • Other Matching Procedures. An alternative to the participant-by-participant matching is to match characteristics of groups, rather than individuals. Another is to equate groups by holding the matching variable constant. We can also match by building the variable into the study.

Advantages of Random Selection and Assignment

Random selection enhances external validity, and random assignment enhances internal validity. Random assignment controls all kinds of confounding variables, including unknown confounding variables.

Control through Experimental Design

We have discussed three major groups of control methods: general control procedures, control of subject and experimenter effects, and control of participant selection and assignment. The fourth major group of control procedures is control through specific experimental design. 

 All experiments have the following specific characteristics:

  1. a clearly stated research hypothesis concerning predicted causal effects of one variable on another;
  2. at least two levels of the independent variable;
  3. unbiased assignment of participants to conditions;
  4. specific and systematic procedures for empirically testing the hypothesized causal relationships;
  5. specific controls to reduce threats to internal validity.

Ethical Principles

The balanced placebo design raises several ethical issues, including informed consent, deception, and avoiding potential risks. The participants do know that they may receive alcohol, but they are not told that they may be unaware of whether they are in the experimental or control condition. Alcohol is a controlled substance, so people must be legally able to consume and not have physical or psychiatric conditions that make consumption risky. Researchers in this field have been very careful about meeting all of their ethical obligations while carrying out this research.