Graziano and Raulin (8th ed)Graziano & Raulin
Research Methods (8th edition)

Research Design Checklist Examples

In this unit, we will walk you through a couple of examples using the research design checklist. As you will see, the checklist only alerts us to issues that must be considered. It does not indicate what decisions we should make. The information necessary to make those actual decisions was covered in earlier chapters in the textbook. We recommend that you follow along with the research design checklist as we go through these examples. If you need more details on any element of the research design checklist, you can consult the tutorial elsewhere on the website.

Examples Using the Research Design Checklist

We will walk you through two examples of research studies in this unit. One involves a differential research study, and the other is an experimental research study. To help organize our discussion, we will use the 10 major headings of the research design checklist.

Example #1: Sex Differences in Leadership Style

This is a field study of the management approaches of male and female middle managers in mid-sized companies. The data will be gathered through interviews of selected middle managers and people who are both above and below them in authority. This project is considered a pilot project in that no strong hypotheses are planned, although there is enough in the research literature to guide a number of tentative hypotheses.

1. Initial Problem Definition

With a study of this type, the problem definition will not be as detailed or precise as it would be in a study in which much more is already known and we are interested in testing very specific hypotheses. Nevertheless, the more we learn about the topic before we begin the study, the more likely we will zero in on fruitful material. 

Because this is a pilot study, we will actually have several problem statements focusing on issues such as (1) how men and women differ in their management strategies, (2) how effective specific strategies are for male and female managers, (3) whether there are interactions between sex of the manager and the type of management strategy that is effective, and (4) how males and females conceptualize their management roles. Studying the available literature on management approaches and on differences between men and women in social interactions, we are able to formulate a number of areas that seem fruitful to explore.

2. Research Hypothesis

Even though this is essentially a pilot study, it is useful to formulate specific hypotheses based on the information available in the research literature. For a study of this sort, we would likely formulate a dozen of more primary hypotheses and perhaps twice as many speculative hypotheses. The primary hypotheses would likely focus on specific hypothesized differences in management style and their effectiveness. We might expect interactions between the sex of the manager and the effectiveness of specific styles, and there may even be higher order interactions of these variables with the sex of those who are being managed. Even though some of our hypotheses will be little more than educated guesses, just formulating them will help us to develop our data collection procedures and guide other critical decisions in the design phase.

3. Statistical Analysis

In a project of this sort, the data analysis will emphasize descriptive statistics, although our tentative hypotheses will be tested with inferential statistics. But by far the largest number of analyses will be data snooping. Nevertheless, we encourage students, even in pilot studies like this one, to specify as many hypotheses as possible and decide exactly how they will be tested statistically. This process forces us to consider exactly what data our research questions will produce and whether the data that we will get will be sufficient to address the kinds of questions we are interested in. The decisions on how to operationally define both the independent and dependent variables will shape the statistical analyses. In this case, it is a good idea to let the consideration about the statistical analyses shape how the dependent variables are measured. For example, we have speculated that there may be several critical interactions between sex of the manager and management style. Interactions are most effectively evaluated with factorial ANOVAs. Factorial ANOVAs need score data. Therefore, we would be wise to structure our dependent measures to produce the score data that would make the critical analyses easier.

4. Theoretical Basis

You might think that theory plays a limited role in early pilot work on a topic, but that should not be the case. Theory plays a different role than it would in a more mature field, in which we have a good idea what to expect. Instead of using a theory to make one or more specific predictions, we use a broad range of theories to identify promising areas for study. The theories we might draw on are the theories of leadership, theories about sexual differences in goals and interaction styles, theories about interaction styles when underlying values are not shared by participants, and perhaps even evolutionary theories about sexual differences that have been shaped by different sex roles. Each of these theories will point us toward specific issues that may or may not turn out to be informative.

5. Independent Variable Manipulation

This is essentially a survey study with separate groups (male and female middle managers). There is no independent variable manipulation.

6. Dependent Measures

The core of this study is the dependent measures. To give ourselves maximum flexibility, we would likely use a structured interview for the study. This is the most widely used approach for such pilot work. The structured interview provides a variety of hard data that can be used in tests of specific hypotheses, while also allowing us to pick up on, and pursue, promising leads that may generate hypotheses for later study. In our proposed study, we wanted to interview the managers, at least one person who each manager reports to, and at least two people under the manager. We have different purposes for each of these interviews, but one reason for interviewing at different levels is to see if the management style is reported the same way by the different people. It would be interesting to find that some managers reported that they tried to use explanations about why something was important, while those they managed thought they just made autocratic decrees about how to do things. Since we know that upper level managers often encourage their middle managers to use a specific management style, we want to know what the managers' managers are telling them and what management style those upper level managers prefer.

We will be constructing a structured interview based on our best guesses of where the interesting data might lie. Therefore, our dependent measure is unlikely to have validational data available. It will be our responsibility to collect as much reliability and validity data as possible. For example, we might want to interview our managers twice with about a month between interviews. We may ask different questions at the two interviews, but we would likely want to repeat some key questions to see how stable the manager's responses are (test-retest reliability). We would like to record the interviews and have multiple raters evaluate the information to determine interrater reliability. We would correlate the reports of the managers with the reports of those they manage and those that manage them to get some preliminary validity data. If the managers report one thing, but others report something quite different, we would have to question the validity of the managers' reports. Direct observations of the managers in later studies may show that the managers are accurately reporting their behavior, whereas those they manage have a distorted view of their manager's actions. But we would have no way of knowing that until that follow-up study was done. That is one of the principle uses of pilot studies like this one. Pilot studies suggest the most fruitful areas for more in-depth study.

7. Controls

In a survey study like this one, the most important control issue is to obtain a representative sample. Since we want to compare male and female managers, we want a representative sample of each group, but we also want those samples to be comparable to one another. We would want to be sensitive to the industries that from which we drew our samples, because management style may vary from one industry to another. We might want to note demographic variables, such as age and education, which may vary between the male and female managers and could be a confounding variable. We would not likely want to try to control these variables at this stage, because we know so little about management styles that we would have little basis for knowing what controls to implement. Nevertheless, we would want to carefully measure as many of these potential confounding variables as possible, so that we could evaluate their effects in our analyses.

One important set of controls will focus on trying to encourage the most accurate reports of behavior possible. Two things would threaten the accuracy of the self-reports. The first is any ambiguity in the questions. We want to make sure that our questions are clear, specific, and unambiguous. We also want to make sure that they do not lead our participants toward specific answers. Most of the participants will probably want to present themselves in the best possible light, and it is likely that they will be very sensitive to subtle cues about what is considered the optimal behavior. Those cues may be part of the wording of the items, but they could also be part of the reaction of the interviewers to the answers provided. The interviewers must be trained to accept anything that the managers say without giving any indication of disapproval. That will encourage honest responses that are not modified to gain the approval of the interviewers. Furthermore, managers are likely to modify their responses if they feel that the information they provide will be shared with others (e.g., their boss). In a study of this sort, it is critical that the participants responses be confidential if we want quality data, and participants have to believe that those responses will be confidential. Perhaps interviewing participants away from the worksite will increase this sense that the information provided will be confidential. Certainly, using a room that provides privacy for the interviews will enhance the feeling. The attitude of the interviewer will make a huge difference. Specific statements in the informed consent form can increase the participant's sense of security.

8. Participants

We have already discussed several issues associated with participant selection. In a pilot study of this type, we want to select as broadly as possible to give ourselves a chance to spot interesting behavior and attitudes that may only be found in a subset of the individuals. A large sample size is desirable, although the costs of such a pilot study and the benefits probably will limit the sample size to a more moderate level. A large sample size will give us better information about the topic under study, but pilot studies are never taken seriously as a way of testing hypotheses. Therefore, we will still have to conduct more focused research studies with specific hypotheses, so we would not want to use too much of our resources on this pilot study.

Every study must meet ethical standards and should be reviewed by the appropriate IRB. The IRB will be interested in knowing how the interests of the participants are protected. For example, since some information provided might be damaging to the participants if their supervisors were to have access to it, there must be procedures to prevent this from happening. The informed consent form must spell out what will be required, what risks might exist, and what safeguards are in place to reduce those risks. If there are potential benefits to the study, they should also be a part of the Informed Consent Form. The feedback that will be given to each participant must also be spelled out in the research proposal to the IRB. No data can be obtained, including pilot data, without approval from the appropriate IRB.

9. Preparation of the Setting

We have already mentioned how the setting might affect the responses of the participants. For an interview study like the one proposed, the setting should be comfortable and functional. It should be quiet and private. The interviewer is very much a part of the setting in a study of this sort. He or she should be professionally dressed and should behave professionally throughout the process. Every effort should be made to keep the interviewer blind to key hypotheses. It might be desirable to have some of the data collected with paper and pencil measures to avoid the possibility that the interviewer might affect those data. There is evidence in the research literature that sensitive questions tend to be answered more truthfully when the questions are asked on a computer screen and the answers are provided to the computer. This approach might be used for those elements of the interview that are particularly sensitive. Of course, if this approach is used, the software to do this portion of the interview has to be designed so that it responds appropriately no matter how the participant responds. If the participant is given a choice of four responses (numbered 1-4), for example, and his or her finger slips hitting the "w" key, the program has to be written to request the response again.

10. Adequacy of Participant Preparation, Instruction, and Procedures

Most of these issues have already been covered in our discussion above. The critical feature here is that the interviewers must be trained to handle any situation and to make the participant feel comfortable and accepted. The interview itself must be carefully constructed so that questions are unambiguous and are not leading. There should be procedures in place to handle any potential problem, and the interviewer must be aware of all of those procedures.

 

Example #2: Dietary Variety and Weight Management in Rats

Unlike the study in the first example, this study has a specific hypothesis and a relatively narrow goal. We want to see how variety in a rat's diet affects their eating habits and their weight. We suspect that greater variety will encourage more eating, since we have noticed that, even when stuffed after a meal, we are tempted by the dessert tray at a restaurant, probably in part because the desserts are so different from the main course.

1. Initial Problem Definition

This study may have been triggered by observations of our own eating behavior. We may have noticed that we tend to eat more when there is variety in our diet and eat less if we eat the same thing over and over. We wonder whether this is a phenomenon that affects many species, and therefore may have a common biological base. We want to manipulate taste and appearance to see how it affects eating and weight.

2. Research Hypothesis

Our tentative hypothesis is that, even when nutritional value is controlled, rats will eat more and gain weight with a diet that has more variety.

3. Statistical Analysis

This is an experimental study with several groups. We are interested in group differences on two variables (amount eaten and weight). Both of these are measured on a ratio scale and thus produce score data. Our Decision-Tree Flowchart would lead you to select a one-way ANOVA for you inferential statistics. Be sure to compute the appropriate descriptive statistics for each group.

4. Theoretical Basis

The theoretical basis for this study is that we expect that evolution favored those animals that ate a balanced diet. Therefore, animals who had access to several different kinds of food would be better off nutritionally if they ate each of the foods. Animals with access to only one food, would likely tire of it and eat less, saving room for other foods if they are found.

5. Independent Variable Manipulation

Rats in the laboratory are normally fed a diet of pellets that are nutritionally balanced and tasty to rats. We decide to make this our control condition. We will also have two other carefully prepared diets for the rats. The first produces the same pellet mixture, but the shapes and colors are varied. The taste, however, is identical for all pellets, and the taste is the same as with the standard rat diet. The second produces pellets that are all the same color and shape but vary in taste. The tastes are manipulated chemically with chemicals that provide no significant nutritional value, and the tastes are tastes that would occur in a rat's normal diet. We decide to add another condition just for our own interest. Our first three conditions are equated on nutritional value and the mix of nutritional substances. We wonder what a rat would do if given the same diet available to any human who shops at a convenience mart. Therefore, we give the rat chips, cookies, canned soup, bread, and a dozen other things taken directly off the store shelf. No effort at all is made to balance the nutritional value of this diet to the other three diets.

This is an experiment, so the rats will be randomly assigned to one of these four conditions. We will monitor their eating and weight for the eight weeks of the planned study.

In this study, we really do not need to pretest the manipulation with the animals. We already have checked to make sure that the three conditions are nutritionally identical, although they differ on either appearance or taste. The fourth condition is obviously different on many variables, and we threw this condition in just to see what would happen. We really do not need a manipulation check either. Our dependent variables will determine whether differential responding occurs.

6. Dependent Measures

We have decided to let all of our rats have free access to food for an eight week period. Every 6 hours we will give them fresh food, and weigh the amount of food not eaten. Therefore, we will know the exact weight of the food consumed by each animal. We will also weigh the animals periodically to determine their weight gain. The animals were purchased from a breeder to all be the same age with the same genetic background. At the beginning of the study, they are all approximately that same weight.

Both of our dependent measures are weights (amount of food eaten and animal weight). These are frequently used measures in animal research. These measures can be made precisely. We will keep daily records on both of these variables, which will give us the equivalent of reliability data. Even though these weight measures are well established, it would still be a good idea to have the researcher making the measurements be blind to the condition that each rat was assigned to.

7. Controls

For our three primary conditions, we were careful to hold the nutritional value of the food constant and vary only the appearance or taste of the food. In that way, any differences that we might find between the groups could not be attributed to differences in nutritional value (a potential confounding variable). We randomly assigned our rats to conditions, which controls many sources of confounding. We know that our fourth condition (the junk food diet) has lots of confounding variables, but we are not concerned with that. We have added this condition just to see how a rat would respond to the diet that we are tempted with on a daily basis.

In animal research, we rarely sample randomly from the population. Therefore, we have to be careful about generalizing our findings to a broader population. Rats are bred for laboratory research. Different breeds have certain characteristics. We have chosen a common breed for our study, but we know that other breeds of rats may perform differently. Furthermore, other animals (cats, dogs, birds, etc.) may also behave differently. Controlling for external validity will be done be repeating this study with different breeds of rats and different animals to see if we get consistent findings.

8. Participants

As mentioned, animal research typically uses carefully bred animals that are very similar in their genetic makeup. We generally need to do little more than report the breed of the animals and their age at the beginning of the study to adequately describe the sample. Sample size is an important characteristic. We would decide on the size by determining the typical amount of variability in weight among the animals and deciding how small an effect we want to be able to detect. The details of this process is beyond the scope of this text, but those details can be found in any graduate level text in psychological statistics. The animals are randomly assigned to conditions. They each get their own cage for identification, but to confirm identification, each animal is also given a small ear-tag with a unique ID number. With lab rats, we do not recruit participants, but rather buy them from the breeder.

9. Preparation of the Setting

The setting for this study are the cages that these animals will be raised in. It is important that this setting be as natural as possible and provide a comfortable living environment for the animals. If we do not do this, we will not be confident that we are seeing normal behavior from these animals. We elect to use larger cages and put five animals in each cage, because these are normally social creatures and prefer to be with other rats. All the animals in one cage will be in the same diet condition. The cages must be kept clean throughout the study to protect the health of the animals. Appropriate distractions are placed in the cage to simulate a real-world environment. The food trays can be removed and replenished from the outside of the cage without disturbing the animals. Animals are handled carefully when they are weighed each day to minimize stress from the handling. All appropriate precautions are taken to protect the health and well being of the animals, so that we can be sure that we are observing typical rat eating behavior. The personnel working with these animals are trained in how best to handle the animals and what they should be watching out for. The person who weighs and records the weight is kept blind to group membership.

10. Adequacy of Participant Preparation, Instruction, and Procedures

We do not have to worry about the instructions to the rats. They already know how to eat when hungry. The question is how much will they eat under each condition. Most of the procedures described above are standard lab procedures that any well-trained animal lab technician or animal researcher would know well. Nevertheless, it is a good idea to write out the exact procedures for every aspect of the study and verify frequently that those procedures are being followed precisely.

Exercises Using the Research Design Checklist

The best way to learn any complex skill is to practice it in real life situations. We have included some exercises to provide that practice. To access those exercises, click on the heading for this section.