EXPERIMENTAL RESEARCH AND DESIGN IN EDUCATION
(Only for M.Ed. Students, Teacher Educators and Educational researchers.This article is not meant for B.Ed. students.)
M.Sc., M.Ed., JRF & NET
Assistant Professor in Physical Science, Arafa Institute for Teacher Education
Experimental Research is an attempt by the researcher to maintain control over all factors that may affect the result of an experiment. In doing this, the researcher attempts to determine or predict what may occur.experimental Design – is a blueprint of the procedure that enables the researcher to test his hypothesis by reaching valid conclusions about relationships between independent and dependent variables. It refers to the conceptual framework within which the experiment is conducted.
Steps involved in conducting an experimental study
· Identify and define the problem.
· Formulate hypotheses and deduce their consequences.
· Construct an experimental design that represents all the elements, conditions, and relations of the consequences.
1 .Select sample of subject.
2. Group or pair subjects.
3. Identify and control non experimental factors.
4. Select or construct, and validate instruments to measure outcomes.
5. Conduct pilot study.
6. Determine place, time, and duration of the experiment.
§ Conduct the experiment.
§ Compile raw data and reduce to usable form.
§ Apply an appropriate test of significance.
Some terms and symbols
1. Experimenter(E):The person who manipulates the experimental conditions.
2. Subject(S):The living organism that is studied.
3. Independent /treatment/antecedent variable(IV,X):The manipulated variable.It may be a teaching method,a teaching aid etc
4. Dependent/criterion/predicted variable(Y):Iit is the condition or characteristic that changes as the experimenter changes the X.
5. Control group(C):The group that does not receive any experimental treatment.
6. Experimental group(E):The group that is given the independent variable treatment.
7. Pre-test(T1):Test before the application of X.
8. Post-test(T2):Test after the application of X.
The Three Important Criteria In Selecting an experimental design
The researcher should select the design that will do the job it is supposed to do and is able to arrange objectively the experimental conditions to meet the requirements of his study.
2.Adequacy of control
It must provide adequate control so that the effects of the independent variable on the dependent variable can be measured.
3.Validity of Experimental Design
Internal Validity :The extraneous variables that affect the control of a design.
External Validity:Concerned with the generalizability or representativeness of the experimental findings.
Threats to Internal Validity
1. History:Unplanned or extraneous events that occur during the research and affect the results.
Example: Researcher collects gross sales data before and after a 5 day 50% off sale. During the sale a hurricane occurs and results of the study may be affected because of the hurricane, not the sale.
2. Maturation–Occurs when an effect is due to maturational or other natural changes in the subjects.(being older,wiser,stronger,tired)
Example: Subjects become tired after completing a training session, and their responses on the Posttest are affected.
3. Pre-testing :Occurs when the act of taking a test or responding to a questionnaireaffects the subjects.
Example: Subjects take a Pretest and think about some of the items. On the Posttest they change to answers they feel are more acceptable. Experimental group learns from the pretest.
4. Measuring Instruments - Changes in instruments, calibration of instruments, observers, or scorers may cause changes in the measurements.
Example: Interviewers are very careful with their first two or three interviews but on the 4th, 5th, 6th become fatigued and are less careful and make errors.
5. Statistical Regression :Aresult may be due to respondent’s being identified on the basis of extreme high or low scores.
Example: Managers who are performing poorly are selected for training. Their average Posttest scores will be higher than their Pretest scores because of statistical regression, even if no training were given.
6. Differential Selection - Different individuals or groups would have different previous knowledge or ability which would affect the final measurement if not taken into account.
Example: A group of subjects who have viewed a TV program is compared with a group which has not. There is no way of knowing that the groups would have been equivalent since they were not randomly assigned to view the TV program.
7. Experimental Mortality - The loss of subjects from comparison groups could greatly affect the comparisons because of unique characteristics of those subjects. Groups to be compared need to be the same after as before the experiment.
Example: Over a 6 month experiment aimed to change accounting practices, 12 accountants drop out of the experimental group and none drop out of the control group. Not only is there differential loss in the two groups, but the 12 dropouts may be very different from those who remained in the experimental group.
8. Interaction of Factors, such as Selection Maturation, etc. - Combinations of these factors may interact especially in multiple group comparisons to produce erroneous measurements.
9. Experimenter bias: Attributes or expectations of researcher that influence results. Attributes: Age, sex, race, status, hostility, authoritarianism, physical appearance Expectations: Deliberate or unintentional effects of bias on part of researcher that results in treating subjects differently.
10. Diffusion of Treatment:
Treatment given to experimental group affects control group (contamination).
Example: Members of the two groups are aware of the research or share information about treatment with one another. Best if groups do not know about one another1
11 .Subject EffectsChanges in subjects’ behavior in response to the experimental situation. Alter behavior to respond more favorably.
Example:Hawthorne effect (subjects respond differently because aware being studied). John Henry effect (control group may compete or overcompensate). Demoralization effect (control group feels neglected, puts forth less effort or gives up ). Novelty effect (subjects more motivated, participatory because new and different)
Threats to External Validity or Generalizability
Ø POPULATION VALIDITY:
I. Selection of subjects-Generalization is limited to the subjects in the sample if subjects are not selected randomly from an identified population.
II. Characteristics of subjects-Generalization is limited to the characteristics of the sample or population (socioeconomic status,age,location,ability,race)
III. Subject-treatment interaction-Generalization may be limited because of the interaction between the subjects and treatment.
Ø ECOLOGICAL VALIDITY:
I. Description of variables-Generalization is limited to the operational definitions of the independent and dependent variable.
II. Multiple treatment interference-In experiments in which subjects receive more than one treatment, generalizability is limited to similar multiple treatment situations because of the effect of first treatment on subsequent treatments.
III. Setting-treatment interaction-Generalization is given to the setting in which the study is conducted.(room,time of day,others presence,other surroundings etc)
IV. Time of measurement treatment interaction-Results may be limited to the time frame in which they were obtained.Treatments causing immediate effects may not have lasting effects.
V. Pretest-post test sensitization-They may interact with the treatment so that similar results are obtained only when the testing conditions are tested.
VI. Novelty or disruption effect-Subjects may respond differently because of a change in routine and generalization may be limited to situations that involve similar novelty .
Types of experimental designs
A set of commonly used symbols are
X = The exposure of an independent variable to a group of test subjects for which the effects are to be determined
O = The process of observation or measurement of the dependent variable (effect outcome) on the test subjects
R = The random assignment of test subjects to separate treatment groups
Classification of experimental designs
a.The one shot case study
b.The one group pretest -post test study
c.The static group comparison design
2.True experimental designs
a. The post test only equivalent group design
b. The pretest posttest equivalent group design
c. The two group non random pre test post test design
d. The Solomon four group design
a) Simple factorial design of 2 by 2
a. The pretest posttest nonequivalent group design Counter balanced design
b. Counter balanced design
a.One –Group time-Series Design.
b.Control –Group Time- Series Design
Pre experimental design
Pre- experimental design represents the crudest form of experimentation and is undertaken only when nothing stronger is possible. The designs are characterized by and absence of randomization of test subjects. The pre-experimental designs are weak in their scientific measurement power because they fail to control adequately the various threats to the internal validity
1.The "One-Shot Case Study"(inadequate control of extraneous variables)
A single group of test subjects is exposed to the independent variable treatment X, and then a single measurement on the dependent variable is taken .One-Shot case study does not use pre test and control group. As a result this design is inadequate for establishing causality. For eg. A study on the employee education campaign about the automation of the office activities without a prior measurement of employee knowledge. Result would reveal only how much the employees know after the education campaign, but there is no way to judge the effectiveness of the campaign
No control group. This design has virtually no internal or external validity.
2.One group Pre-test, Post-test design
Minimal Control. There is somewhat more structure, there is a single selected group under observation, with a careful measurement being done before applying the experimental treatment and then measuring after. This design has minimal internal validity, controlling only for selection of subject and experimental mortality. It has no external validity. Continuing the example given above it measures the awareness of the employees before the campaign and after the campaign
Pre-test Treatment Post-test
O X O
3.The static group comparison design
Compares the status of a group that has received anexperimental treatment with one that has not.The limitation is that there is no provision for establishing the equivalence of the experimental and control group.
The main advantage of this design is randomization. The post-test comparison with randomized subjects controls for the main effects of history, maturation, and pre-testing; because no pre-test is used there can be no interaction effect of pre-test and X. Another advantage of this design is that it can be extended to include more than two groups if necessary. For eg in a field setting an experiment is designed to study the effect of a natural disaster (experimental treatment) on the psychological trauma (measured outcome). A pre-test before the natural disaster say tsunami is possible but not on a large scale. Moreover the timing of the pretest would be problematic. The control group, receiving the post-test would consist of subject whose property is safe.
True experimental design/lab experiment
The experiments performed in an artificial or contrived environment is known as lab experiments. In the lab experiments the researcher has complete control over all aspects. The researcher has a control over the experiment, who, what, when, where and how.
1.Two groups ,randomized subjects,post test only design
Test subjects are randomly assigned to either the experimental or control group. The experimental group is then exposed to the independent treatment, after which both group receive a post treatment measure of the dependent variable. In this design, the pretest measurements are omitted. The design is
R X O1
The experiment effect is measured by the difference between O1 and O2. The design is more simple and attractive. Internal validity threats from the history, maturation, selection and statistical regression are adequately controlled by random assignment. Since the subjects are measured only once, the threats of testing and implementation are handled. The different mortality rates between experimental and control groups continue to be a problem. The design reduces the external validity problem of testing interaction effect.
2.Two groups, Random Selection, Pre-test, Post-test
Group Pre-test Treatment Post-test
Expl groupE(R) O X O
Control gpC(R) O C O
The advantage here is the randomization, so that any differences that appear in the posttest should be the result of the experimental variable rather than possible difference between the two groups to start with. This is the classical type of experimental design and has good internal validity. The external validity or generalizability of the study is limited by the possible effect of pre-testing.
3.Two groups, Nonrandom Selection, Pre-test, Post-test
Group Pre-test Treatment Post-test
Expll gp E(R) O XO
Control gpC(R) O CO
The main weakness of this research design is the internal validity is questioned from the interaction between such variables as selection and maturation or selection and testing. In the absence of randomization, the possibility always exists that some critical difference, not reflected in the pretest, is operating to contaminate the posttest data. For example, if the experimental group consists of volunteers, they may be more highly motivated, or if they happen to have a different experience background that affects how they interact with the experimental treatment - such factors rather than X by itself, may account for the differences.
4.Solomon Four-Group Design
Group Pre-test Treatment Post-test
Pre-tested Eptll Gp E (R) O X O
Pre-tested Controlgp C (R) O O
Unpre-tested Eptll Gp UE (R) X O
Unpre-tested Control GpUC (R) O
This design overcomes the external validity weakness in the above design caused when pre-testing affect the subjects in such a way that they become sensitized to the experimental variable and they respond differently than the unpre-tested subjects.
Quasi experimental design
In a quasi-experimental design, the research substitutes statistical "controls" for the absence of physical control of the experimental situation.
1,Non equivalent pretest posttest design:
The most common quasi-experimental design is the non equivalent group Pre-test/Post-test Design. This design is the same as the classic controlled experimental design except that the subjects cannot be randomly assigned to either the experimental or the control group, or the researcher cannot control which group will get the treatment. In other words, participants do not all have the same chance of being in the control or the experimental groups, or of receiving or not receiving the treatment.
This design can be diagrammed as follows:
Pretest treatment posttest
O1 X O2
Comparison of the pre test result (O1 – O2) is one indicator of the degree of equivalence between test and control groups. If the pre test results are significantly different, there is a real question about comparability. On the other hand, if pretest observations are similar between groups, there is more reason to believe internal validity of the experiment is good.
The counterbalanced design
It may be used when the random assignment of subject to experimental group and control group is not possible. This design is also known as rotation group design. In counterbalanced design each group of subject is assigned to experimental treatment at different times during the
experiement.This design overcomes the weakness of non-equivalent design. When intact groups are used, rotation of groups provides an opportunity to eliminate any differences that might exist between the groups. Since all the groups are exposed to all the treatments, the results obtained cannot be attributed to the preexisting differences in the subjects. The limitation of this design is that there is carry-over effect of the groups from one treatment to the next. Therefore, this design should be used only when the experimental treatments are such that the administration of one treatment on a group will have no effect on the next treatment. There is possibility of boring students with repeated testing.
Replication o1X1 O2X2 O3X3 O4X4
1 Group A B C D
2 GroupB D A C
3 GroupC A D B
4 GroupD C B A
More than one independent variables can be manipulate simultaniously. Both indepentent and interaction effects of two or more than two factors can be studied. Experiments may be designed to study simultaneously the effects of two or more variables. Such an experiment are called factorial experiment.
Simple Factorial Design:
A factor is a major independent variable . A level is a subdivision of a factor. A simple factorial design is 2x2 factorial design. In this design there are two independent variables and each os the variables has two levels. One advantage is that information is obtained about the interaction of factors. Both independent and interaction efects of two or more than two factors can be studied with the help of this factorial design. With two factors each taking two levels, a factorial experiment would have four treatment combinations in total, and is usually called a 2x2 factorial design. The first independent variable, which is manipulated, has two values called the experimental variable. The second independent variable, which is divided into levels, may be called control variable. For example, there are two experimental treatments, that is, teaching through co-operative learning and teaching through lecture method. It is observed that there may be different levels of intelligence of the students. On the basis of the IQ score the experimenter divides the students into two groups:one high intelligent group and the other the low intellegent group. There are four groups of students within each of the two levels of intelligence.
High Intelligence Group
Low Intelligence Group
Teaching Through Co-operative Learning Method
Gain Score on the Dependent Variable
Gain Score on the Dependent Variable
Teaching Through Lecture Method
Gain Score on the Dependent Variable
Gain Score on the Dependent Variable
Since one of the objectives is to compare various combinations of these groups, the experimenter has to obtain the mean scores for each row and each column. The experimenter can first study the main effect of the two independent variables and the interaction effect between the intelligence level and teaching method
TIME SERIES DESIGN
A research design in which measurements of the same variables are taken at different points in time, often with a view to studying social trends. For this reason such designs are sometimes also known as trend designs and are distinguishable from ‘one case post test only designs in which measurements are taken only once. Time series designs can be used in conjunction with official data, for example by plotting crime rates for the same area but for different point in time (monthly, quarterly, annually). This acts as a basis for making statements about trends in levels of crime. It is possible to plot the trends for different variables at one and the same time with a view to making inferences about their relationship, for example, to map unemployment rates against crime ...
Time –Series Designs
There are two types of time –series designs.
1.One –Group time-Series Design.
A series of measurement on the dependent variable are taken before and after the group is exposed to experimental treatment. The experimenter takes a number of measurements on the independent variable Y, exposes the group to the experimental treatment X, and then again takes additional measurements (T) on the independent variable Y.
This design is useful in the school settings to study the effects of a major change in administrative policy upon various issues concerning discipline. It is also useful in the study of attitude change in the students as a result of the effect produced by the introduction of a documentary film designed to change attitudes.The multi-testing of students in this design provides a check on some sources of internal validity than in Design 1.
1.This design fails to control the effects due to history. For example, the factors such as climatic changes,examinations may contribute to the observed change in the dependent variable.
2. Because of the repeated tests, there may be a kind of interaction effect of testing that would restrict the findings the those populations which have been subjected to repeated testing.
3.The usual statistical tests of significance may not be appropriate with a time design.
2 Control –Group Time- Series Design
Control-group Time-Series design is an extension of One-group Time-Series Design. To overcome the limitations of One-group Time-series Design, this design utilizes a control group. The control group is also an intact class group. This group like the experimental group is tested on the dependent variable at the same intervals of time, but is not exposed to the experimental treatment.
1.This design overcomes the weakness of One-group Time-Series Design by controlling the effects due to history.
2.The inclusion of control group in this design is useful for necessary comparison.
There may be interaction effect due to repeated tests and this would restrict the findings to those populations which have been subjected to repeated testing. The usual statistical techniques may not be applicable with such designs.
Experimentation is a sophisticated technique for problem solving and may not be an appropriate activity for the beginning researcher . It has been suggested that teachers may make their most effective contribution to educational research by identifying important problems that they encounter in their classrooms and by working cooperatively with research specialists in the conduct and interpretation of classroom experiments.
1,Reserch in education: James H. McMillan & Sallly Schumacher.
2.Methodology of Educatin Research: Lokesh Koul.
3.Research in eduction: John W. Best and James V. Khan.
4Understanding Educational Research: An Introductionb: Deobold B Vandalen.
5.Introduction to educationl Research: C.M. Charles.