Populations, Samples, and Validity

In this section we deal with the fifth step in the research process, the design. Chapter Three explains some of the basics regarding populations, samples and validity. We explore how to draw samples from populations, how to assign samples to groups, the influence of the sample on the external validity of the study, and the effect of other events and actions on the internal validity of the study.

Populations and Samples

Population - Any set of people or events from which the sample is selected and to which the study results will generalize.

Sample - A group of people or events drawn from a population. A research study is carried out on a sample from a population. The goal is to be able to find out true facts about the sample that will also be true of the population. In order for the sample to truly reflect the population, you need to have a sample that is representative of the population. The best method to use to obtain a representative sample is to randomly select your sample from the population. A study that has a large, randomly selected sample or a carefully matched sample is said to have external validity.

A non-random sample reduces the external validity of the study. Much medical research is done on the patients one sees in the clinic, this is a non-random sample that is not representative of a larger population and will not generalize. Because it will not generalize is not a fatal flaw in the study. A study with a non-random sample still identifies true facts about the sample and perhaps those findings will be true for others as well. It is best to define your population first, and then obtain a random sample.

The sample size required depends on the requirements of the study and size of the population. As a rule the bigger the better. If the sample is too small then the performance of a few individuals can have a big effect on the data, and render the data less representative of the population.

Sample Selection Methods - There are several methods for drawing random samples. All methods produce good random samples.

1. Simple random sampling from the population using a Random Number Table (At end of Chapter) or some other random process (slips of paper in a hat).
2. Stratified random sampling from sub-groups in the population, i.e., to have a random sample of 100 people evenly divided by gender, you would divide population into male and female groups and randomly select 50 from each group.
3. Proportional sampling to insure maintenance of sub-group proportions, i.e., divide population of the School of Allied Health into male and female groups. Since there are 8 women to every man, in order to have a random sample of 100 people balanced on gender we need to randomly select 80 women and 20 men.
4. Systematic sampling - drawing every kth person, i.e., to get a random sample of voters you select every 10th person from the Voter Registration Roles at the courthouse.
5. Cluster sampling - a method to get random samples when the population is large, there are important control variables, and you can only study a small sample (i.e., to get a random sample of 60 administrators of hospitals in the United States, you could group hospitals into clusters based on private/public ownership, and big/medium/small hospitals and then randomly select ten subjects from each cluster. This method is a more elaborate version of stratified sampling.

Sample Assignment Methods - If you have more than one group then subjects need to be assigned to groups. It is important that groups be equal at the beginning of a study. If the groups are not equal then you cannot know if the independent variable caused changes in the subjects or if the inequality did. There are two good methods for assigning your sample to experimental groups and a third way, not so good, that often occurs:

1. Random - insures distribution of extraneous variables and is usually the best way to assign subjects. This method is best if the group size is twenty or more. Random assignment assures an equal distribution of all extraneous variables. For any extraneous variable (age, weight, distribution of blood types), you will find the average of the variables (or the distributions) to be the same for each group if you use random assignment
2. Matched - insures distribution of control variables by matching pairs of subjects in the different groups, i.e., you find pairs of subjects who are very similar to each other on control variables, such as age, sex, race, etc., and then you randomly assign one of the pair to one group and the other to the second. Good when the sample has to be small.
3. Pre-existing groups or non-random assignment - Often a study uses groups that are pre-existing, i.e., people who die or who live after a heart attack. Other times you use the first ten patients for treatment A, and the second ten for treatment B. These groups have non-random assignment and are also not necessarily representative of the population you want to study. Non-random assignment reduces the internal validity of a study, because the groups are different at the start of the study. The magnitude of the difference determines the degree of reduction in internal validity.

Summary of Sample Selection and Assignment. Figure 4 summarizes the preceding discussion in terms of sample selection and assignment. The diagram also points out the consequences of how the sample is selected and assigned on the external and internal validity of the study. The next section discusses these two aspects of validity in more detail.

A valid research study is one that finds the truth. We hope to discover facts and principles that explain or predict. There are two components of validity that have been identified. These components are external and internal validity. We can use these components of validity as criteria to see if a particular study is valid.

External validity is dependent on the adequacy of the sample. If the sample is representative of the desired population then our results will generalize. This is called generalizability. Thus, if we study patients in a free clinic can we generalize to patients of a private physician? The answer is no, to be able to generalize to both groups we must include subjects from both care sources.

To have a generalizable sample, first define your population, then randomly select a large sample. With a random sample of sufficient size research findings can generalize to the larger population. As a rule of thumb random sample sizes of twenty subjects per group are minimally sufficient.

Internal validity refers to the adequacy of our study design and the degree of control we have exercised in our data gathering. Good internal validity is insured by application of the concept of control. This concept is very important in research. By control we mean that all variables except the dependent variable are controlled by the experimenter. In this way if the dependent variable changes during the study then that change is due to the changes the experimenter made in the independent variable(s). The concept of control has six major parts:

1. Groups have equal scores on the dependent variable at the start of the study and are of large size. Random assignment will insure they are equal without testing if sample size is large
2. Extraneous variables are controlled so no group is effected by them during the study.
3. Each group receives identical treatment during the study except for the manipulations of the independent variable
4. Large numbers of subjects are not lost from the study and any losses are distributed evenly across the groups
5. The treatment (manipulation of the independent variable) was of sufficient magnitude and duration to expect it to change the dependent variable(s).
6. The dependent variable(s) are accurately measured.

In evaluating the internal validity of a study we ask this question: Was the experimental manipulation the only possible cause of a change in the dependent variable? In general, if a study adequately responds to the six factors above, then it will have controlled for many extraneous influences, will allow the researcher a good chance of detecting any change in the dependent variable, and will have internal validity. Note that a study can have good internal validity and NOT find any changes in the dependent variable due to the independent variable. Also, a study can have good internal validity, but without a generalizable sample, it may have no external validity. Finally, remember that a study with no external validity still found true relationships for the sample that was studied. If I study Mongolian fishermen in Cleveland, I cannot generalize to Vietnamese shrimpers in the Gulf, but I still know more about Mongolian fishermen.

Do not confuse internal validity with the validity of the method by which the dependent variable is measured, called test validity. Internal validity refers to the overall degree of control exercised. Test validity refers to the suitability of the measuring instrument used.

Over the years a number of terms have been introduced that describe various factors that can adversely influence the internal validity. We have already discussed most of these factors, but we have not necessarily used these common terms before. The list below will familiarize you with these names and their definitions.

1. History: Specific events unrelated to the study, occurring between the first and second measurements in addition to the experimental treatment.
2. Maturation: General events/experiences occurring to participants over an extended period of time, e.g., growing older, fatiguing, etc.
3. Pre-Test Influence (Test Practice): The effects (e.g., cueing and practice) of taking one test upon the results of taking a second test. Sometimes the subjects can learn about the dependent variable by just taking the pre-test. This additional learning can confound the effect of the independent variable on the dependent variable.
4. Statistical Regression: Changes in scores over time due to unreliability of measuring devices; especially troublesome when using subjects selected on the basis of extreme scores.
5. Experimental Mortality: Loss of more subjects from one group than from the other. This may make groups unequal.
6. Instrumentation Error: Error of measurement due to:
   1. Changes in the assessment instrument (e.g., shortening a test, adding different items, changing the scoring procedure),
   2. Changes in the observers (e.g., different observers at O1 and O2, some observers using different standards than others, or training of observers changes from one treatment to the next), and
   3. Changes in the equipment (e.g., a fault in the equipment, non-standardization of equipment prior to study, loss of calibration).
7. Bias in Group Composition (Differential Selection): Biases or conveniences in creating comparison groups that cannot be assumed to be equivalent (e.g., the groups are not equal because they were not randomly chosen. For example, if one hospital uses one treatment method and a second hospital uses a second method, then the groups are biased because it is unrealistic to assume the hospital populations are the same).
8. Selection-Maturation Interaction: Biases in the selection of groups to be included in the study may differentially be affected by the time between assessments. For example, if the subjects are children and the average age of one group is older than the others, then the maturation process will effect the older group differently than the younger groups. If changes due to maturation can be confounded with changes due to the independent variable then the internal validity of the study is reduced.
9. Hawthorne Effect: Being in an experiment sometimes changes the response of the subjects. New treatment methods may be exciting, and people improve due to the thrill of it all and the increased attention.

Summary of Internal and External Validity

Probability is the science of figuring how often something will happen. Probabilities are based on past events. If I toss a coin, there are two outcomes, heads or tails. If I toss a coin a lot of times, I will find that half of the time I get a head. Thus, the probability of getting a head on the next coin toss is 1 outcome divided by 2 possible outcomes or .5.

Why is it that the probability of getting a head is .5 and not .6 or .4? This has to do with the concept of "random events". When I toss a coin there are two outcomes, heads or tails. The result of my coin toss is a random event; each outcome has an equal chance of occurring. Since there are two possible outcomes, and the outcome is a random event, then the probability of a head is .5 (or 50 times out of 100 I will get a head). If I roll a die, which has six sides (six outcomes), the probability of my getting "three spots" is one divided by six or .17 (17 times out of 100 I will get "three spots" when I roll a die). The probability of any random event is the number of events (tosses or rolls) divided by the number of possible outcomes.

When we take a random sample from a population, each subject we select is a random event. Thus, if our population was 60% women and 40% men, the probability of our drawing a woman each time would be .6 and of drawing a man would be .4. If we draw a big sample, the final proportion of men and women in our sample would be the same as in the population. Each event has an equal chance of occurring and thus will occur in the proportion with which it exists. This is why a random sample is the easiest and best way to draw a sample.

Random samples are only the best way if the sample can be large. With a small sample, you may not get the proportions to "average out". If we only drew five people from a population that was 60% women and 40% men we could very well draw five women, just by chance. Only with a large sample (say 20 to 100) will the random events produce samples that are proportional to the population.

Use of the Random Number Table - A random number table is often used for subject assignment and sample selection. While subjects can be selected or assigned by drawing their name out of a hat, a more elegant method is to use a random number table. The steps below illustrate the use of a random number table.

To use the table:

1. List subjects in any convenient order, i.e., alphabetical
2. Close your eyes and point to any row of the table.
3. Close your eyes and point to any column.
4. Begin at the intersection of the row and column you picked. Assign the random numbers that appear in the column to your subjects, e.g., the first number is given to the first subject on the list.
5. When you get to the end of the column of numbers begin over with the next column.
6. Now order your subjects by their assigned random numbers.
7. You now have a list of randomly ordered subjects, you can assign them to groups by just taking the first k subjects for group 1, the next k subjects for group 2, and so on until all are assigned.