Other extraneous variables which are known or expected to have an effect on the independent variable, the dependent variable. or both, may be used as control variables and incorporated into the experimental design. For example, in a study of the effect of housing arrangement (integrated versus segregated) on racial prejudice, if the researcher expected that females tend to be less prejudiced than males, he could further analyze his findings for females and males separately in each housing arrangement. As most of the social activities and behaviors have multiple interlocking relationships. the more control variables brought into the design, the more chances the researcher has to demonstrate the effect of the independent variable on the dependent variable. For if in spite of the effects of the control variables on the dependent variable, either directly and independently or jointly with the independent variable, the independent variable still shows a significant effect on the dependent variable, the researcher then has greater confidence in the causal relation between the independent and the dependent variables.
Manipulation of the independent variable represents the most distinctive characteristic of the experiment. as compared to other methods of data collection. In the above mentioned example, the housing arrangement was the independent variable which was manipulated by the researcher-the residents were assigned to different housing developments. In reality, such manipulation may present many problems, but other variables are much easier to manipulate. For example, to assess the effect of source credibility on the believability of a message, the researcher can manipulate the identification of the sources who present the same message to two groups of subjects (as respondents are usually called in an experiment, because they are subjected to manipulation by the researcher). In delivering a message about how to feed young children, to one group of young mothers, the source person may identify herself as a nutritionist affiliated with a medical college and, to another, the same source may identify herself as a housewife with two young children. Many other variables can be manipulated in a similar fashion. The manipulation of the independent variable provides the researcher with an opportunity to vary the kind and the intensity of the values of the independent variable and to assess the effect of such variations on the dependent variable.
These three characteristics. randomization, control and manipulation, make the experiment a desirable method to use in testing specific hypotheses, because of its ability to single out the independent and dependent variables while eliminating and controlling the effects of other variables. The manipulation aspect becomes more attractive to a researcher for testing a causal relation between two variables.
The basic design of an experiment involves the following steps : 1) selection and random assignment of subjects to experimental groups, 2) pretest measurement of the dependent variable, 3) differential treatment (manipulation), and 4) posttest measurement of the dependent variable.
Strictly speaking, the selection of the subjects should also use a representative sampling plan. But in most experiments conducted by social scientists, voluntary subjects are recruited. The researcher then screens the volunteers for competence in receiving, understanding, and responding to the type of manipulation to be performed. Hopefully, any bias introduced by this voluntary recruitment procedure will be eliminated by randomly assigning the subjects to different experimental groups.
Then the subjects are interviewed or required to complete a
questionnaire in which measurement of the dependent variable is made. In
order to not sensitize the subjects to the purpose of the study, the actual
instrument used contains many more questions than those dealing with the
dependent variable. In fact, successful manipulation depends to a large
extent on the ability of the researcher to camouflage the measurement of the
dependent variable in the pretest measurement.
Usually, there is a time lag between the pretest measurement and the
treatment (manipulation of the independent variable); should there be any
lingering suspicion about the purpose of the study among some subjects, the
time lag hopefully will diminish it to a minimum. The variations of the
treatment can be either different values or categories of the independent
variable, such as the integrated housing versus the segregated housing in our
illustration. Or, they could represent the presence and the absence of the
independent variable (e.g., one group is exposed to a message while the
other is not).
Finally, after another time gap, the instrument administered in the pretest
measurement is again administered to the subjects.
To measure the effect of the independent variable on the dependent
variable, a difference score is computed between the pretest measurement
and the posttest measurement of the dependent variable for each subject in
each group. For example, in an experiment to test the differential effects of
source credibility on attitude change toward racial prejudice, if a subject in
the high-credibility source group registered a "2" response on a racial
prejudice scale of 7 points ("1," most prejudice; "7," least prejudice) in the
pretest measurement and a "5" response on the same scale in the posttest
measurement, he receives a difference score of positive 3 (5 - 2 = +3). We
may then compute the averaged difference score for each group, say, the
high-credibility source group and the low-credibility source group. From
now on, the term difference score is used to indicate the averaged difference
score for each group. Ideally, the difference score should be interpreted as
the change in the value of the dependent variable (e.g., attitude change
toward racial prejudice) resulting from the treatment (manipulation of the
values of the independent variable, e.g., high- and low-credibility sources).
But, in reality, such an interpretation is faulty, for the difference score may
actually represent the effects of a number of other factors involved in the
experiment, in addition to the effect of the independent variable. ln the
following sections, the various factors affecting the difference score, and
elaborations of the basic design, will be discussed.
There are at least seven factors which may contribute to the difference score:
1. The treatment effect T. The effect of the treatment (the independent
variable) on the difference score.
2 The pretest measurement effect X. The pretest measurement may sensitize
the subjects to the manipulation and the subsequent posttest measurement,
thus affecting the difference score.
3 The time effect U. Since the experiment is conducted over a period of time,
time-related events and maturation may influence a subject's exposure to the
treatment and his responses on the posttest measurement.Time-related
factors cannot be controlled by the researcher: thus, they are also known as
uncontrolled factors.
4 The interaction effect between the pretest measurement and the
uncontrolled factors Ixu. Sensitization to the pretest measurement and factors
related to time may jointly affect the posttest measurement scores of a respondent.
Here, the symbol I is used to indicate interaction effects, X the pretest measurement
effect, and U the time effect. For example, sensitization to the pretest
measurement may not by itself affect the posttest measurement. But should
some event occur between the pretest measurement and the posttest
measurement, it may cause the respondent to recall the pretest measurement,
resulting in certain pattern of response which otherwise would not have been
formulated. An illustration might be a study focusing on attitudes toward
blacks among whites. After the pretest measurement of such attitudes, along with a
number of other attitude measurement, and preceding the posttest
measurement, civil rights legislation has been passed into law by Congress.
This event triggers the respondents' recall of the pretest measurement items and
may result in more positive attitudes toward blacks in the posttest measurement
than would be the case if the legislation had not occurred.
5 The interaction effect between the pretest measurement and the treatment
IXT Sensitization to the pretest measurement and the exposure to the
manipulation may jointly affect a subject's posttest measurement score.
6 The interaction effect between the treatment and the uncontrolled factors
ITU. Exposure to the treatment and the time-related uncontrolled factors
may jointly affect the posttest score of a subject.
7 The interaction effect among the pretest measurement, the treatment, and
the uncontrolled factors IXTU. Finally, activities associated with the pretest
measurement, the treatment, and the time-related uncontrolled factors may
all jointly affect a subject's posttest score.
All the subjects in the basic experimental design discussed in the last
section are subject to the influence of these factors. Further, these seven
factors may independently affect the difference score. Thus, the difference
score may represent the added effect of all seven factors. Using the symbol d
to represent the difference score, a simple equation may be constructed to
represent the relationship between d and the seven factors: namely, d is
equal to the sum of the seven factors:
d = X + T + U + IXT + ITU + IXU + IXTU (14.1)
Equation 14.1 shows that, for the basic experimental design in Figure 14.1,
the averaged difference score for all the subjects in each experimental group
in fact represents the consequences of the effects of the seven factors on the
subjects over the period during which the experiment takes place.
The problem then becomes whether it is possible to identify each of the
effects so that the difference score d can be decomposed and interpreted
adequately and the effect of the treatment T on the difference score
identified. For the researcher, equation 14.1 contains only one known score-
the difference score computed from the change between the pretest
measurement and posttest measurement. In general, this type of equation
(called a nonhomogenous equation) can be solved for only one unknown.
But equation 14.1 has seven unknowns, and therefore we cannot determine
the effects individually.
The strategy, then, is to construct different experimental groups, so that
different equations can be formulated containing these effects. Theoretically,
if seven different equations, or seven different experimental groups, could be
constructed, then all seven effects could be identified. This is possible
because seven equations would provide solutions for seven unknowns.
Unfortunately, it is impossible to construct seven different experimental
groups. In other words, there is no perfect experimental design.
There are four different experimental groups a researcher can utilize in an
experiment. These groups are presented in Table 14.4. Group 1 is the basic
experimental group discussed previously. The subjects in this group are
exposed to the pretest measurement, the treatment, the uncontrolled factors,
and the posttest measurement. In group 2, the subjects are administered the
pretest measurement but do not participate in the treatment. They are,
however, subjected to the effects of the uncontrolled factors.
Group 3 subjects are not administered the pretest measurement, but are
exposed to the treatment ; therefore they are affected by T, U, and P. Finally,
group 4 subjects do not participate in the pretest measurement and do not
receive the treatment ; they are administered only the posttest measurement.
There are other possible but useless groups which can be constructed. For
example, groups could be constructed which participate in the pretest
measurement only, the treatment only, or the pretest measurement and the
treatment only. But lack of posttest measurement would prevent the
computation of the difference scores for the subjects in these groups. Thus,
they cannot be considered.
For each of the four available groups, we may discuss the computation of the
difference score. For clarity d1 will represent the difference score computed
for group 1, d2 for group 2 etc.
Thus,
D1 = X + T + U + IXT + IXU + ITU + IXTU
This equation has one known score (d,) and seven unknowns. Thus, there is
no way to identify each individual effect.
D2 = X + U + IXU
There is one equation but three unknowns.
D3 = T + U + ITU
Since this group does not have a pretset measurement, d3 is not known.
Therefore, equation 14.3 (group 3) is meaningful only if the pretest score can
be estimated from another group (either group 1 or group 2)). In that case,
and if it can be assumed that the groups are relatively large (therefore that the
estimated pretest score is stable) and randomized (therefore that there are no
prearranged differences between the groups), then the averaged pretest score
of another group can be used to find d3. This estimation method will be
demonstrated later in the chapter.
Equation 14.3 then, also has three unknowns (T,U, and ITU).
D4 = U
In summary, there can be up to four groups, and their equations are :
Since four equations can be solved only for four unknowns, no matter which
groups are used, no experiment design can identify all the effects.
One solution to this problem of lack of information from the data is for the
researcher to make necessary assumptions about certain unknowns. For
example, if a researcher chooses a one-group design (group 1), he has one
equation and seven unknowns. In order for him to solve for the effect of one
factor, say the effect of the treatment T, he would have to make assumptions
about the other six unknowns in equation 14.1. Three types of assumptions
can be made. An easy way out is simply to assume that all the other
unknowns are equal to zero. In other words, all other factors have no effect
on the difference score. Or, a researcher can assume that these factors have
effects on the difference score but that effects canceled each other out. Third,
he can assign numbers to the various factors to represent the extent of the
effects they have on the difference score.
These numbers can be based on evidence from past research. How realistic
these assumptions are depends on the experimental situation, the nature of
the study, the activities which have taken place during the experiment, and
past evidence reported in the literature. The fewer assumptions a researcher
has to make, the less likely it is that he will distort the data.
In general, two criteria help the researcher to select a particular design: (1)
to minimize the number of assumptions. and (2) to assume effects which
either have been determined m previous studies or are less consequential in
relation to other alternative effects.
Thus, a research design which requires two assumptions is in general
preferred to a design requiring three assumptions. A research design
assuming effects determined in the past is preferred to a design assuming
effects without such previously determined values. A research design
assuming effects which intuitively have no important consequences on the
crucial variables is preferred to one which has to assume values for effects
affecting the crucial variables.
In the following, we will discuss several popular experimental designs and
assess their relative merits in terms of the two criteria just mentioned.
D1=X+T+U+IXT+ITU+IXU+IXTU
This is the most commonly used experimental design, in which one
experimental group receives the treatment and the other does not. Since two
equations are involved, and there are seven unknowns, the researcher must
make assumptions about five unknowns in order to find solutions for the
other two unknowns. To illustrate the solutions, an example is provided in
Table 14.5. In this example, the pretest measurement of the dependent
variable shows an average of 20 points on the scale for both groups,
indicating that randomization was effective in the assignment of subjects to
the two groups. The posttest measurement scores are 100 and 60 points,
respectively, for the two groups. Thus, the averaged difference scores can be
computed for the two groups. The researcher decided that he would assume
that all the interaction effects and the uncontrolled factors did not
appreciably affect the posttest scores of the subjects. After eliminating the
terms assumed to vanish in the equations, the researcher obtained two
equations containing only two unknowns. The two unknowns were then
solved. The researcher concluded that the treatment induced a change of 40
points from the pretest measurement to the posttest measurement, and that
the pretest measurement effected a change of 40 points also.
Instead of assuming that the uncontrolled factors U had no effect on the
difference score, the researcher could alternatively make a similar
assumption about the pretest measurement effect X or the interaction effect
between the pretest measurement and the uncontrolled factors lxu The
decision would not affect the solution for the treatment T, but would
drastically change the effect of X, U, or lXu, depending on which one was
selected for solution.
Thus, it can be concluded that the experimental design utilizing groups 1
and 2 requires the researcher to make many assumptions and that the
solutions are extremely unreliable.
D3 = T + U + I
Thus, there are two equations and three unknowns. The researcher needs to
make only one assumption to solve for two unknowns. However, since
neither group received a pretest measurement, how can the difference score
be computed? If we denote the pretest measurement score for group 3 as b3,
and the posttest measurement score as a3, and likewise for group 4 as b4 and
a4, then the difference scores for the two groups are:
D3 = a3 - b3
Thus,
D3 - D4 = a3 - b3 - (a4 - b4)
However, if randomization was in effect in the assignment of subjects to the
two experimental groups, then the pretest measurement scores for the two
groups should be approximately the same. In order words, b3 and b4 should
be approximately the same and cancel each other out in the last equation.
Then,
D3 - D4 = a3 - a4
Thus, the difference between the two difference scores for the two groups
can be computed from the difference between the two posttest measurement
scores alone.
From equations 14.3 and 14.4, we know that :
D3 - D4 = a3 - a4 = T + ITU
Thus, we only need to make an assumption about ITU to solve for T. An
example of this design is presented in Table 14.6.
A comparison between this design and the two-group design 1 discussed
earlier (Table 14.5) shows that this design needs only one assumption as
contrasted to three assumptions for the other design to solve for the effect of
the treatment T. Further, no pretest measurement represents savings for the
researcher. Thus, we may conclude that in general the two-group design
utilizing groups 3 and 4 is superior to the two-group design utilizing groups 1
and 2 in finding the effect of the treatment. This conclusion is valid only if the
assignment of the subjects to the groups is completely random, so that the
pretest measurement scores can be assumed to have been similar for the two
groups if a pretest measurement was administered.
The only advantage of the two-group design 1 over the two-group design 2 is
the fact that the former permits determination of the effect of another factor
(for examplea in Table 14.5 the effect of the pretest measurement), whereas
the latter does not (it is impossible to find U, because the difference score for
each group is unknown). However, this advantage is based on the heavy
price a researcher has to pay for making assumptions about the effects of five
factors.
Thus, this design has three equations and seven unknowns. To solve for three
unknowns, assumptions must be made for the other four unknowns. In
general, the researcher is interested in finding T, so no assumption should be
made for it. Further, assumptions can be made for only two of the unknowns
X, U, and IXU, since the third value would automatically be derived from
equation 14.2. Thus, the researcher can make assumptions about the four
factors selected from the combination of two or three of the three interactions
IXT, IXU, and IXTU, and one or two from among X, U, and IXU. An
example of such a design is presented in Table 14.7. In this example, the
researcher assumed that the effects of IXT, ITU, IXU and IXTU were
minimal, so that solutions were found for T, X, and U.
Note that in this example the third group (TUP) did not take the pretest
measurement. In order to compute d3, an estimate of the pretest
measurement score was computed by taking the average of the pretest
measurement scores from the first two groups. This could be done because,
again, it was assumed that the random assignment of subjects to the three
groups ensured similar pretest scores for all groups. However, when taking
the average from the pretest scores from the first two groups, the researcher
should take into account the fact that the scores differed slightly from subject
to subject. This variation indicates that there might be some random error
involved in the estimate of the pretest score for group 3. Thus, the final
estimate must include a point estimate (250) and an interval estimate
(plus/minus 20). The two estimates must be carried in all subsequent
computations.
A comparison between the two three-group designs discussed shows that
the three-group design 2 needs fewer assumptions (two versus four) and
pretest measurements (one versus two) than the three-group design 1. Thus,
in general, the three-group design utilizing groups 2, 3, and 4 is superior to
the three-group design utilizing groups 1, 2, and 3. Again, this statement is
valid only if complete random assigmnent of the subjects to the groups is
assumed, so that the estimates of the pretest measurement scores for the
groups not taking the pretest measurement are reliable and within a tolerable
range.
Note several things in Table 14.9. First, the pretest and posttest
measurement scores are not given; only the difference scores and the
estimated change scores are presented. Second, the assumptions about the
effects of the three factors X, IXT. ITU are actual scores rather than zeroes.
This suggests that past research evidence has provided realistic estimates of
the effects of the factors. Finally, the effect of the interaction between the
pretest measurement and the uncontrolled factors is 15 + 25, an interval
including both positive and negative scores. This suggests that the effect of
this factor was small, thus making the estimate unreliable.
FACTORS AFFECTING THE DIFFERENCE SCORE
Available Experimental Groups and Alternative
Experimental Designs
Experimental Groups
Group 1
As represented in equation 14.1, the group 1 difference score d is
composed of: (1) the effect of the pretest measurement X, (2) the effect of the
treatment T, (3) the effect of uncontrolled events U, (4) the effect of the
interaction of the pretest measurement and the treatment IXT, (5) the effect
of the interaction of the pretest measurement and the uncontrolled
events IXU. (6) the effect of the interaction of the treatment
and the uncontrolled events ITU, and (7) the effect of the
interaction among the pretest measurement, the treatment, and
the uncontrolled events IXTU.Group2
ln the same way, the difference score d2 between the posttest and
the pretest measurements can be shown as follows:Group 3
The equation for the effects in group 3 is :Group 4
Group 4 does not have a pretest measurement. The discussion for
group 3 also applies here to obtain d4 for the following equation :Experimental Designs
Two-Group Design 1
In this design, groups I and 2 are used. Thus,
equations 14.1 and 14.2 are employed:
D2=X+U+IXUTable 14.5 Example Solving Two Unknowns in Two-Group Design 1
Two Group Design 2
An alternative two-group design utilizes groups 3 and
4. The equations involved are:
D4 = U
D4 = a4 - b4
= (a3 - a4) - (b3 - b4)Three-group design 1
The best known three-group design uses groups 1, 2,
and 3. The system of equations involved is :Table 14.7 Example of solving three unknowns in Three Group design 1
Four-Group Design
The design utilizing all four groups is known as the
Solomon four group design and involves all four equations and seven
unknowns. With assumptions made for the effects of three factors, solutions
for the effects of four factors can be found. An example of the design and the
solutions appears in Table 14.9.