Going from the Research Question to Research Designs
This document will discuss the different types of research questions and describe several basic research designs that are used with each type. Also the general strengths and weaknesses of each design will be explained.
The domain of research designs is divided into three categories of research questions: descriptive, differences, and relationship. A descriptive research question seeks to identify and describe some phenomenon. An example: What is the ethnic breakdown of patients seen in the emergency room for non- emergency conditions. A differences research question asks if there are differences between groups on some phenomenon. For example: Do patients who receive massage experience more relief from sore muscle pain than patients who take a hot bath? A relationship question asks if two or more phenomena are related in some systematic manner. For example: If one increases his level of physical exercise does muscle mass also increase?
If you are reading a research article, you can decide which type of research question a study used by determining the number of groups and some characteristics of the variables. Use the following chart to decide which type of research question was used:
----------------------------------------------------------------
If you have two or more groups ................. Differences
If you have one group and measure
the DV two or more times ..................... Differences
If you have one group, and measure
the DV once ................................. Relationship
If you have one group, no IV, and observe
the DV once .................................. Descriptive
----------------------------------------------------------------
Table 1 illustrates the research designs used with each type of research question. The numbers of the research designs correspond to the designs on the following pages. The designs are just examples. You can change them around at will.
Table 1 - Research Questions and Designs
--------------------------------------------------------------------------
TYPE OF RESEARCH RESEARCH DESIGN
QUESTION USUALLY USED
--------------------------------------------------------------------------
Descriptive 1. Observational w/ one observation
(Describe conditions) 2. Observational w/ multiple obs.
3. Ex Post Facto
Differences 3. Ex Post Facto *
(Is there a difference?) 4. Pre/Post (two obs. of DV)
5. Pre/Post w/Control Group
(two obs. of DV)
6. Two-Group (one after treat. obs. of DV)
7. Three-Group (one after treat. obs. of DV)
8. Repeated Measures (two or more obs.)
9. Factorial (two or more IVs)
10. Co-variance (pre-observation as control)
11. ABA Time Series (single subject)
12. AB Time Series (single subject)
Relationships
(How do the variables 13. Correlation/Regression (one group)
relate to each other) 14. Correlation/Reliability (one
group and two obs.)
-------------------------------------------------------------------------
* This design bridges both types
To measure the phenomena being studied the researcher defines variables. Two types of variables are used: dependent variables (DV) and independent variables(IV) . A dependent variable is the phenomenon we want to study. In the examples, ethnic group, sore muscle pain, and muscle mass would all be dependent variables. An independent variable is a phenomenon that when it changes makes another phenomenon change. In the examples above, type of therapy (hot bath or massage), and exercise level (none, a little, a lot) are independent variables. The effect of the independent variable on the dependent variable can be seen in the examples: Which treatment therapy (a hot bath or a massage) will reduce sore muscle pain the most, or if we increase exercise, by how much will muscle mass increase.
There are two categories for how the variables are measured: continuous and categorical. Continuous variables can be measured using either the interval or ordinal scale of measurement. Categorical variables can be measured using either the ordinal or nominal scale of measurement. It is usually best to call your variable categorical when you have an ordinal measurement scale.
Independent and dependent variables can be either continuous or categorical depending on the research design. The combinations of variables allowed to be used in the same design (i.e., a continuous and a categorical) are determined by the various statistical tests. Generally with a differences research question you will have categorical IVs and either categorical or continuous DVs. With a relationships research question all variables must be either categorical or continuous. In Chapter 6 we will deal with which combinations are allowed in more detail.
Interval measurement uses a scale where the distances between the points on the scale are equal across the scale, i.e., measuring with a ruler is an interval measurement because inches are carefully defined to be a uniform length. A ratio scale is almost the same as an interval scale and for our purposes no distinction is made.
An ordinal scale is a scale where phenomena are ordered or ranked, i.e., arrange a group of ten people by height and number the tallest person one and the shortest person ten.
Finally, a nominal scale consists of categories with no order. The variable, type of therapy, from the example above, is a nominal variable with two categories: hot bath and massage. Note that treatment therapy is the variable and that it is ONE variable (with two categories: hot bath and massage), NOT two variables. Every variable MUST have at least two categories.
You will diagram the research design using a matrix where events are the columns and groups are the rows. An event is the occurrence of one of three possibilities: a treatment, no treatment, or an observation. A treatment event is when the independent variable (IV) is manipulated, i.e., from the above example, when subjects are given a hot bath or a massage. A no treatment event is one where nothing is done to the IV. An observation event is when the dependent variable (DV) is measured. Group is simply the number of groups of subjects in the study. An example matrix for the pain study is below.
EVENT 1 2 3
-------------------------
GROUP 1 O1 X1 O2
GROUP 2 O1 X2 O2
-------------------------
The first event, O1, is an observation of the pain level. The second event is the application of the treatment (X1 for massage and X2 for the hot bath). The third event, O2, is the second observation of the pain level.
Finally, we need to introduce you to some terms and shorthand conventions that are used to diagram the various designs.
The nature of the sample (random or non-random) and how subjects were assigned to groups (random or non-random) is indicated by a letter in the upper left corner of the diagram.
Descriptive research questions give rise to observational designs. Observational designs use three general ways to gather data: observation, interview, or survey. These three methods are obvious in their application. In the first we observe behavior, in the second we ask people to describe their behavior orally, and in the third we seek written replies.
The simplest design is to ask a sample of people about their behavior. Suppose we want to know how many people use seat belts. We would have a descriptive hypothesis: What is the percentage of people who wear seat belts? Suppose we wait at a stop light and when it is red we ask drivers if they usually wear their seat belts. A weakness of this study is that we do not know if the subjects are being totally honest with us. This "dishonesty" factor lowers the reliability and validity of this type of design. This is a non-random sample, with no treatment (no defined independent variable), and one dependent variable (percent of the sample who wear seat belts). The dependent variable is continuous and uses interval measurement. We observe the dependent variable once by asking each subject questions about this behavior. Our design looks like (1):
EVENT 1 ------------------- (1) Group 1 O -------------------
There are variables that effect seat belt usage, but in design (1) we are not studying them, thus the term "no defined IV."
Suppose we want to know how preferences for presidential candidates change over time? Design (2) is a survey design with multiple observations, no defined IV, and random selection of subjects (Note the R above "Event"). We will ask the same question to the same people in January, May, October, and just before the election in November. Our dependent variable (continuous and interval) is the percent of people who support each candidate. Our descriptive hypothesis for each observation might be: What percent of the sample support each candidate?
R EVENT 1 2 3 4 ----------------------------------------- (2) Group 1 O1 O2 O3 O4 -----------------------------------------
A more sophisticated design for the seat belt study from design (1) would emerge if we observed whether or not they were wearing their seat belt and then asked them how often they wore their seat belt, using this scale: 1) all the time, 2) some of the time, or 3) never. This is based on a real study, see Geller, Casali & Johnson (1980). We would now have an independent variable (categorical and nominal): seat belt on or seat belt off. Our dependent variable (categorical and ordinal) would be their answer to the question: 1) always, 2) sometimes, or 3) never. This would give us two groups, still non-random. This type of design is called ex post facto since we do not manipulate the independent variable (IV), we only use it to classify people into groups. The design looks like this:
EVENT 1 ------------------------ Group 1 (On) O (3) Group 2 (Off) O ------------------------
Even though we interview many people the design indicates just one observation (O). Because the conditions under which we interview each person (subject) are the same, it is all one observation. Each subject is called a replication, and we add up all the replications to get an estimate of the dependent variable for that observation. For a more complicated design with several observations, the conditions are different for each observation. We expect the behavior of the subjects to be different for each observation when there are several (see designs 2, 4, and 10 for example).
A directional hypothesis for design (3) is: People who are not wearing their seat belt will give answer 2 (sometimes) more often than people who are wearing their seat belts. Also, from the data collected from design (3) we could calculate another DV in terms of the percentages of people who said they wore seat belts when they were wearing them, and who said they wore them when they were not wearing them.
An observation study can provide very reliable data provided the criteria of what to look for are clearly defined. It is best to have two or more people observe, then correlate their observations. If their observations correlate well (a correlation value of .8 or .9), then we know we have reliable observations. If the correlation is not good (less than .7) then we need to redefine our criteria or train the observers better. In the Geller et al (1980) seat belt study they had two observers who observed 1,827 vehicles and they agreed 86.4% of the time on the gender of the driver, and whether or not the driver was wearing a seat belt. Geller et al only used observations where complete agreement was obtained, and discarded the non-agreement data.
Descriptive studies are characterized by either not having an IV or by having an IV that is not manipulated. Also, if you have a descriptive hypothesis you have a descriptive design. Often there is confusion about whether or not a descriptive study has an IV. Remember a study with one group usually does NOT have an IV. A one group study will have an IV only when it is a time series, covariance, or repeated measures design. A study with two or more groups always has an IV. If we use the IV only to classify subjects into groups then it is an ex post facto study. When we manipulate the IV we have a differences study, and that is discussed in the next section.
Differences Research Questions give rise to three types of designs: simple experimental designs (pre/post, two-group and three-group), factorial designs, and time-series designs. These designs are characterized by 1) manipulated IVs, and 2) two or more groups. These are the best designs to prove that altering (manipulating) the IV causes the DV to change. These designs enable you to control for unwanted effects due to extraneous variables. Thus the only thing effecting the DV is your manipulation of the IV. Experimental designs always have categorical IVs and the DV can be categorical or continuous. We will examine the simple experimental designs first, beginning with weak designs and ending with strong designs.
Simple Experimental Designs Design (3), ex post facto, is often called a simple experimental design even though we do not manipulate the IV; the dividing lines between types of designs is not always clear and you need to be able to accept some ambiguity.
A second simple design is the one group, pre/post, dependent means design, with or without random assignment (design 4). The term dependent means refers to the way we measure the DV. If we measure the same group of people twice, then the means of those two observations are related or dependent. The pre- observation (Event 1) is made before any treatment is done. The pre-observation is used to control for differences between individuals at the beginning of the study. In this design we use difference scores as measures of the dependent variable, i.e., difference scores are obtained by subtracting O1 from O2. The difference is the change made by the independent variable. Suppose we wanted to see if blood pressure increased after exercise. Our independent variable is time, and it has two nominal categories: before exercise and after exercise. We indicate the exercise period as Event 2 with an X. The directional hypothesis is: There will be a significant gain in blood pressure after exercise, i.e., that O2 will be larger than O1.
R EVENT 1 2 3 --------------------------------------- (4) Group 1 O1 X O2 ---------------------------------------
With design (4) we attribute any change at O2 from O1 to the treatment (X). This is a poor design because it does not have a control group. This type of design is called quasi-experimental because it has some of the qualities of an experimental design but lacks adequate controls to insure internal validity. i.e., in this case it lacks a control group. If a study used non-random selection or non-random assignment it would also be quasi- experimental.
An improved, true experimental study is seen in design (5):
RA EVENT 1 2 3 --------------------------------------- (5) Group 1 O1 X O2 Group 2 O1 O2 ---------------------------------------
In design (5) above, group 2 is the control group that is 1) the same (in terms of the distribution of relevant variables, e.g., age, sex, etc.) as group 1 due to the random assignment (RA), and 2) is not exposed to the treatment. A blank is used to indicate the no treatment condition. One could use X2, but a blank seems a logical indicator of the no treatment condition.
The directional hypothesis is: The experimental group (group 1) will have a larger gain than the control group (group 2). If the group 1 measurement at O2 is different from the group 2 measurement at O2, that difference is due to the treatment. In this study the IV is treatment/no treatment. The average value of the DV for the two O1 measurements will be the same for each group due to the random assignment.
If random selection of our sample is not possible then the study will not generalize to the population we are studying. Even without random selection an effective control group can be obtained by using random assignment. If random assignment is not possible, the O1 measurement serves as a control feature. Without random assignment, instead of comparing the O2 observations, we compare the remainders after we subtract O1 from O2. Design (5) that looks at differences pre to post is called a covariance design, and allows us to correct for initial differences between groups.
Design (5) could have two treatment groups and no control group. The two treatments would be indicated as X1 for group 1, and X2 for group 2. This would also be a true experimental design. Sometimes control groups are not needed if you know the dependent variable is not effected by the passage of time or other exteraneous variable (See Chapter 3 regarding the various threats to the internal validity of a study that might require you to use a control group).
The O1 measurement in design (5) can be deleted to obtain design (6). Since random assignment makes the groups equal there may be no need for the pre-treatment observation. Some times a pre-treatment observation can cause problems with internal validity, so if the pre-treatment observation can be deleted the study is improved (See index under pre-test influence for more information). Design (6) is called a two-group, independent means design. The means are independent because we measure each group only once. The directional hypothesis is the same as the one for design (5), only the number of observations has changed.
RA EVENT 1 2 ----------------------------- Group 1 X O (6) Group 2 O -----------------------------
Design (6) is good to use if you have a large sample (20 or more subjects in each group). If the group size is small then, even with random assignment, the groups could be unequal due to chance factors. With small group sizes (10 or less subjects per group) design (5) is better than design (6) because the analysis of covariance will mathematically compensate for the unequal starting points. If you have a large sample size then either desigb (5 or 6) is good to use.
If we define a more complicated IV, we add more groups to our design. Say we want to compare two therapies: individual counseling and group counseling, and are concerned that people tend to get well without counseling. If we add a control group to design (6), a "no counseling" group, we get a three-group design(7). In design (7), we randomly assigned, randomly selected people with mild emotional problems to three groups. We do not need a pre-measure, but we could use one if desired. Groups 1 and 2 received different treatments (individual or group counseling) for three weeks. A measure of emotional adjustment was made after the three months of therapy. The control group receives no therapy but is given the test for emotional adjustment at the same time as the other two groups.
RA EVENT 1 2 ----------------------------- Group 1 X1 O (7) Group 2 X2 O Group 3 O -----------------------------
The study in design (7) has two directional hypotheses: 1) group 1 will be different from group 2, and 2) group 2 will be different from group 3 (we need not hypothesize that group 1 will be different from group 3 since the answers to the first two hypotheses will provide the answer to the third by a process of elimination).
Suppose we want to see if the effect of the therapy is maintained over time. We can add follow up observations every month after stopping therapy. See design (8), events 3, 4, and 5.
RA EVENT 1 2 3 4 5 ----------------------------------------------------------- Group 1 X1 O1 O2 O3 O4 (8) Group 2 X2 O1 O2 O3 O4 Group 3 O1 O2 O3 O4 -----------------------------------------------------------
Designs like (8) with more than one observation are called repeated measures designs. They can have one or more groups. Repeated measures is not the same as time series, (10) and (11), or a pre/post, (5). Times series is used for studies with a single subject. If you have a pre-treatment observation and two or more after treatment observations, use repeated measures rather than a pre/post design.
If we had a group of clinic patients and assigned a third of them to each group, as in (8), then we would have random assignment without random selection (there would just be an A above EVENT). Using only random assignment the groups are equal, but we cannot generalize our results to a larger population (The internal validity is not threatened, but the external validity is weak). The patients that come to a clinic are not a random population of patients, they are a self selected group of people
DESIGNS ARE CONTINUED IN THE NEXT SECTION