a comparison of the effects of self-versus teacher-assessment of on-task behavior

BEHAVIOR THERAPY 13, 715-723 0 9 8 2 )

A Comparison of the Effects of Self- Versus Teacher-Assessment of On-Task Behavior

DANIEL P. HALLAHAN

JOHN WILLS LLOYD

REBECCA DAILEY KNEEDLER

KATHLEEN J. MARSHALL

University of Virginia

The relative reactivity of self- versus teacher-assessment of a child's on-task behavior during self-monitoring was compared for a learning disabled boy exhib- Ring severe atleniional problems. Both procedures increased on-task behavior over baseline conditions, but self-assessment was significantly superior to teacher-assessment. Both procedures also increased the number of correct addition problems completed, but were not different statistically. Results are discussed in terms of the theoretical question "Is the 'self" important in self-monitoring?" and the practical question "Is self-assessment more cost-effective than teacher-assessment in terms of teacher time and student performance?"

G i v e n the g rowing in t e re s t in the use o f s e l f -mon i to r ing as a b e h a v i o r change t e chn ique , it is su rp r i s ing tha t a f u n d a m e n t a l t h e o r e t i c a l ques t i on a b o u t the p r o c e d u r e first p o s e d b y K a z d i n (1974b) has b e e n l a rge ly unex - amined . K a z d i n p o i n t e d out tha t " i t is i m p o r t a n t to d e t e r m i n e i f S M has an add i t i ona l e f fec t on b e h a v i o r b e y o n d the ef fec t o f r e a c t i v e a s s e s s m e n t , wh ich is p r e s e n t when a s s e s s m e n t is m a d e b y an e x t e r n a l a g e n t " (p. 236). On ly th ree s tud ies , all wi th adu l t s , h a v e a d d r e s s e d the i s sue o f w h e t h e r se l f -mon i to r ing is m o r e o r less e f fec t ive than m o n i t o r i n g b y an ex t e rna l o b s e r v e r ( C a v i o r & M a r a b o t t o , 1976; K a z d i n , 1974a; N e l s o n , L ip in sk i , & Black , 1976). F u r t h e r m o r e , n o n e o f the t h ree has dea l t wi th p r o b l e m - a t ic t a rge t b e h a v i o r s for wh ich p e o p l e a r e l ike ly to be r e f e r r e d for t rea t - ment .

Resu l t s o f the s tud ies c o m p a r i n g self- ve r sus e x t e r n a l - m o n i t o r i n g h a v e

This study was supported by a contract (300-80-0623) from the Office of Special Educa- tion, U.S. Department of Education, for the University of Virginia Learning Disabilities Research Institute. We are particularly indebted to Linden A. Moot for her help in con- ducting this study. Requests for reprints should be sent to Daniel P. Hallaban, Department of Special Education, 152 Ruffner Hall, University of Virginia, Charlottesville, VA, 22903.

7 1 5 ~o~-Tsg~s~07 ~s~72351.~ Copyright 1982 by Assoeiatioo for Advancemcr, t of Behavior Thora.py

All rights of repi'oduction in any form rescued.

716 H A L L A H A N E'I- At , .

been mixed. Kazdin (1974a), recording the use of pronouns during a sen- tence construction task as the dependent variable, and Cavior and Mar- abotto, using verbal behaviors that subjects wanted to increase or decrease during dyadic interactions (e.g., asking the other person questions about himself [increase]; using the word " a h " [decrease]) as dependent variables, found self-monitoring and monitoring by another person equal- ly reactive. Nelson et at., on the other hand, found self-monitoring more reactive than an external observer in decreasing the amount of face touch- ing of students in a college classroom.

Literature reviews of studies from both the laboratory and the classroom (Hallahan, 1975; Hallahan & Reeve, 1980) clearly indicate that off- task behavior is a prominent problem for many learning disabled children. A number of studies (Broden, Hall, & Mitts, 1971; Glynn & Thomas, 1974; Glynn, Thomas, & Shee, 1973; Hallahan, Lloyd, Kosiewicz, Kauff- man, & Graves, 1979: Hallahan, Marshall, & Lloyd, 1981; Heins, 1980; Lloyd, Hallahan, Kosiewicz, & Kneedler, in press; Young, Birnbrauer, & Sanson-Fisher, 1977) have demonstrated the efficacy of self-monitoring as a means of increasing the on-task behavior and, in some cases, the academic behavior of learning disabled children and other children with learning problems. The present study examined the relative reactivity of teacher- versus self-assessed observations of attentional behavior on the on-task behavior and academic productivity of learning disabled children.

METHOD Subject and Setting

Peter, an 8-year-old boy identified by school district criteria as learning disabled, served as the subject, He had a Wechsler Intelligence Scale for Children Full Scale IQ of 117, but standard scores of 77, 75, and 77 in the areas of reading, mathematics, and written language, respectively, on the Woodcock-Johnson Tests of Achievement. He had been assigned to a self-contained special education classroom along with nine other children (eight boys and one girl). All of the students enrolled in the class had been selected because of the severity of their learning and attention problems as observed in regular classrooms the previous year. The average age of the 10 children in the class was 9.5 years, the average Full Scale IQ was 96, and the average reading standard score was 76. Of the students in the room, Peter was identified by the teacher as the student with the most diffimflty attending to task.

The class was a highly structured setting with emphasis on academic learning and appropriate student behaviors which had made it unneces- sary to conduct specific interventions with other students. The teacher had an advanced degree in special education and 3 years previous teach- ing experience. Experimental sessions were conducted during a regularly scheduled period of individual academic work. Peter and most other students were seated at their desks working on individual academic materials while the teacher met with students individually at the back of the class-

SELF- VS, T E A C H E R - A S S E S S M E N T 717

room and the aide met with others at the side of the room. Consultants, supervisors, and observers visited the class frequently, and, consequent- ly, the students were accustomed to the presence of others in the room.

Dependent Measures Direct observations and permanent products provided data for mea-

suring on-task behavior and arithmetic productivity, respectively. Data were collected Monday through Thursday of each week.

On-task behavior. An independent observer collected data on Peter's attention-to-task behavior using a time-sampling technique. Every 4 sec, the observer looked at Peter and made a judgment about whether he was on- or off-task. He was judged to be attending to task when: (a) he was looking at the assigned materials and had his pencil in a writing or erasing position; or (b) he was counting on his fingers. Percent of time attending to task was derived by dividing the number of time samples judged on- task by the number of time samples observed and multiplying by 100. Observer training was provided prior to the onset of data collection for the present study. Training included defining target behaviors, discussing informal observations, and practicing observations; practice sessions were conducted in actual classroom settings until high levels of agreement with the trainer were obtained. Observer agreement was assessed by having a second observer conduct simultaneous observations using the same time samples on 13 of the observation days. The observers situated themselves so that each of them could not note what the other was recording.

Arithmetic productivity. The teacher gave Peter three arithmetic papers at the beginning of each 20-rain session. Each paper contained 100 addition problems, all of which were composed of two single-digit addends. At the end of each session, the papers were collected by the teacher and later given to the observer for scoring. Peter did not receive any feedback on these pages regarding problem accuracy.

Academic productivity was measured by counting the number of problems completed correctly in each session. Productivity, rather than percent correct, was used as the dependent variable because Peter had dif- ficulty with task completion rather than accuracy; he rarely made any errors.

A total of 13 agreement checks were obtained with at least one check during each phase. Papers were checked by the primary observer (who made no marks on the original) for number of problems completed correctly and then independently checked by a second scorer. The two re- suiting scores were compared directly for agreement.

Procedures

Self-assessed self-recording was compared to teacher-assessed self-recording by having two conditions that differed only in who made judgments about whether Peter was attending to his assignments.

Se l f assessment. In the self-assessment condition, the self-recording

718 H A L I _ A | | A N ET AL.

procedure reported in Hallahan et al. (1979) was followed. Briefly, an audiotape recorder emitted quiet tones on an average of once every 42 sec (range: 11 to 92 sec). When the tone sounded, the student was told to ask himself "Was I paying a t tent ion?" and then record his answer (yes or no) on a prepared recording sheet he kept at his desk. An un- marked sheet was used for each session. The teacher trained Peter to use the procedure in one 20-rain session; she explained it to him, modeled it, supervised his initial use of it, and faded her supervision when he was using it correctly.

Teacher-assessment. During the teacher-assessment condition, vir- tually identical procedures were followed except that when the tone sounded, the teacher judged whether Peter was paying attention and then indicated to him what to record on the prepared recording shee t - - she gave a " thumb up" for a " y e s " and a " thumb d o w n " for a " n o . " Train- ing in the use of this procedure was accomplished by simply informing Peter that the same procedure would be used, but that the teacher would make the judgments. Practice was provided in correct use of the teacher- assessment self-recording condition on the first day it was in effect.

On days following the initial training in the use of each assessment procedure, the teacher simply told Peter which condition was in effect. She reminded him at the beginning of the session that he was (in the self- assessment condition) to listen for the tones and mark his recording sheet, or he was (in the teacher-assessment condition) to listen for the tones, look at the teacher, and mark his recording sheet accordingly. On days following holidays, the teacher conducted brief training sessions with Peter. During neither condition was feedback about the accuracy of his recording or the quantity of his on-task behavior provided.

Design An alternating treatments design was used to compare the effects of

the two conditions. Prior baseline and treatment withdrawal phases were also scheduled. Additional phase changes were implemented to reduce Peter 's reliance on the treatment procedures and to assess maintenance of effects.

Baseline. During the first 8 days of data collection, baseline conditions were in effect. During this phase, Peter was given his regular assignment and directed to work on his own at his desk.

Self-recording. On the 9th day of observation and the first day following baseline, self-assessed self-recording was introduced, it was also in effect on the following days of the first self-recording condition: 10, 13, 15, 19, 20, 22, and 24. Teacher-assessed self-recording was first introduced on the I lth day of data collection and was in effect on days 12, 14, 16, 17, 18, 21, 23, and 25. Determination of which condition was in effect was made randomly within each week with the restriction that neither condition could be in effect more than 3 days in any one week. During the self-recording phase, other conditions (i.e., teacher directions to the stu-

SELF- VS. TEACHER-ASSESSMEN1 719

dent, contingencies, and tasks) were kept the same as they had been during baseline.

Reversal. Following the analysis of effects of the two assessment conditions, a reversal of conditions was instituted. For nine consecutive observations, no self-recording procedures were implemented and conditions remained as they had been during baseline.

Self-recording. In order to provide Peter with the beneficial effects of the self-recording condition and to establish more clearly the effects of the procedure with him, self-assessed self-recording was reintroduced on the 35th day of data collection. Self-assessed self-recording was in effect for all 6 days of the phase during which other conditions remained the same as they had been in the previous phases.

When treatment effects had been reestablished, parts of the self-recording procedure were systematically eliminated in order to reduce Pe- ter 's reliance on them. On the 41st through 45th days of data collection, the tape-recorded cues were no longer provided and Peter was directed to record whether he was paying attention whenever he thought about it. Observations were made twice per week during this time. For observations 46--49 (follow-up), the prepared sheet on which he was to record answers to his questions about whether he was paying attention was removed and he was directed to continue working on his assignments to ascertain whether the effects of the treatment persisted in the absence of alI requirements to self-record. Observations were made once per week for sessions 46-48 and 1 month eIapsed between observations 48 and 49.

RESULTS Observer Agreement

Occurrence agreement checks for on-task behavior (the number of samples which both observers scored as on-task divided by the number of agreements plus disagreements, multiplied by 100) had a mean of 96% with a range from 93% to 98%. Agreement for the academic productivity data was 100%.

On-Task Behavior As can be seen from the first three phases (baseline, self-recording,

and reversal) depicted in Fig. 1, both treatment conditions, teacher-assessment and self-assessment, led to immediate and dramatic increases in percent of on-task behavior. In addition, both treatments resulted in sharp slope changes compared with the baseline condition. Fig. 1 also reveals that the self-assessment condition was superior to the teacher- assessment condition. Not only was the median for self-assessment (97%) higher than that for teacher-assessment (88%) but the daily scores for the former were consistentIy higher than those for the latter. Across a total of 17 data points (8 for self-assessment and 9 for teacher-assessment) there was no overlap between the curves for the two treatments. The

72{) t t A L [ . A H A N F: I AL.

9 0

80

p-

:~ s o o

N so

i o ~ 3 0

u

~ 2 0

S E L F - RECORDING S E L F - R E C O R D I N G B A S E L I N E R E V E R S A L CUED , NON- t F O L L Q W - U P

, CUED

q l ~ , , = i i TEACHER SELF = i ', ASSESSMENT A$SESS MENT

lO I 5 20 25 30 55 40 4"J

CONSECUTIVE 06,~ E R",tATIO N S

FIG. I. Percent of time samples on-task for all phases or" the stud}'.

difference between the two conditions was significant (Z = 3.32, p < .01) according to a Mann-Whitney Test. Reintroduction after reversal of the self-recording t reatment using the self-assessment procedure again re- suited in immediate and clear improvements in on-task behavior, further establishing the reliability of this treatment. Data from observat ions conducted while the t reatment was faded and during follow-up indicate that elements of the self-recording procedure ( tape-recorded cues and recording sheet} were successfully withdrawn one at a t ime with no deleterious effect on on-task behavior.

Academic Productivity In general, the results for academic productivi ty were in the same di-

rection as those for on-task behavior but were not as dramatic (see Fig. 2). During the first self-recording phase, for example , both t rea tment conditions resulted in increased productivi ty over baseline and reversal conditions; but, while the median for self -assessment (t49) was higher than that for teacher -assessment (135), there was considerable overlap between data points for each of the two conditions and the difference between the two conditions was not significant (Z = 1.25, p .~- . I t) according to a Mann-Whitney Test. As was the case with on-task behavior , academic productivi ty remained high when the tape recorder and recording sheet were removed.

D I S C U S S I O N

Results suggest that, while both self-assessed and teacher-assessed sell'- recording resulted in increased on-task behaviors, the effects were greater when the child did both the assessment and the recording of his own

721

FOLLOW~UP

2 5 0 l

~- ~0.0 " L~

o

150

ia

o

100

z

BASELINE ~ SELF-RECORDING REVERSAL

.%-7.~, s~,'F- I ASS F-~ ~ M,~N T AZSESSMENT

.'T~. r, , , , i,.. ~i, . , .i. ~., rl. , i , = ,

'ISELF-RECORDING I FOLLOW~UP , CUED NON-

, ~ l\l ~ ..... l I ~ i

1

, , I " , " [ , , , , I i i ,

4 O ,~5

S E L F - V S . T E A C I - I E R - A S S E S S M E N T

6 I0 15 70 25 30 3~

OONSECUTIVE OB~ERVATtON S

F~G. 2. Number of problems completed correctly for aU phases of the study. Horizontal lines represent medians for each condition.

behavior. Thus, the reactive effects that have been obtained for self- monitoring of attentional behavior by learning disabled children (Hallahan et al., 1979; Hallahan et al., 1981; Heins, 1980; Lloyd et al., in press) appear to have been enhanced by the fact that the children themselves were assessing and recording their behavior. This conclusion may not hold, however, for the dependent variable of academic productivity. Al- though the median level of academic productivity was higher for the self- versus the teacher-assessment condition, the difference was not significant.

Results from the present study, lend some support to those theorists (e.g., Kanfer, 1970, 1975) who claim that self-monitoring, in itself, takes on motivational properties that result in behavioral changes. In other words, teacher-assessment may have been less successful than self-assessment because it may not have prompted self-evaluative and self-rein- forcing behaviors that would have been prompted during self-assessment. More operant-oriented theorists (e.g., Rachlin, 1974) would no doubt have predicted comparable reactivity between the two conditions because they each had comparable chances of cuing external reinforcement.

Based on this study, the theoretical question of whether the "self" is important in self-monitoring can only be answered with a tentative "Yes" at this time. Yet, the practical question of whether self-assessment is more efficient than teacher-assessment in terms of teacher time and student performance can be answered with an unqualified "Yes ." In the self-assessment condition, after a brief period of training the child to use the procedure, the teacher is free to engage in other activities, whereas in the teacher-assessment condition, the teacher is randomly interrupted

722 ttALLAHAN El" AL.

on the average of every 42 seconds to look at the child and make a judgment regarding whether the child is on- or off-task. Reports from the teacher also confirm that the self-assessment condition is preferable from the teacher ' s viewpoint. While she noted that the child's behavior im- proved under both conditions, she did observe that the self-assessment condition was much tess disruptive to her classroom routine. Thus, while both procedures--se l f - and teacher-assessment--resul t in increased on- task behavior and academic productivity, self-assessment appears to be the treatment of choice in that compared with teacher-assessment it: (a) leads to clear benefits in on-task behavior, (b) results in at least equal levels of academic productivity, and (c) requires less teacher time.

it is, also, of considerable practical interest that the highest levels of both on-task behavior and academic productivity were achieved during the final phase of the study when all t reatment procedures had been withdrawn. The maintenance of t reatment effects over an extended period of time after the removal of, first, the tape recorder and then the recording sheet, also, supports the findings of Hallahan el al. (1979, 1981) who followed similar withdrawal procedures. These studies, as a group, indicate that although these components appear to be necessary during the initial stages of t reatment (Heins, 1980; Lloyd et al., in press), when using self-monitoring procedures in the classroom, the teacher can wean the child from reliance on "ex t e rna l " components such as cues to record and the recording device. The relative ease and speed with which the treatment procedures can be withdrawn while maintaining treatment effects are additional practical features for classroom application.

Not only does the present study indicate that self-observation is more reactive than other-observation; but it should also caution one against assuming that the same factors influence the relative reactivity of each of these types of behavioral observation. A recent review by Haynes and Horn (in press) on the reactivity of behavioral observation, for example, lumped together studies of external-observation and self-observation. It may be, however , that different mechanisms operate to produce reactivity in self-monitoring versus observations by others.

REFERENCES Broden, M., Hall, V., & Mitts, B. The effect of self-recording on the classroom behavior

of" two eighth-grade students. Jnurnal of Applied Behavior Analysis, 197t, 4, 191-199. Cavior, N., & Marabotto, C. M. Monitoring verbal behaviors in a dyadic interaction.

J(~arnal of Consulting and Clinical Psychology, 1976, 44, 68-76. Glynn, E. L., & Thomas, J. D. Effect. of cuing on self-control of classroom behavior.

J~urnal of Applied Behavior Analysis, 1974, 7, 299-306. Glynn, E. L., Thomas, J. D., & Shoe, S .M. Behavioral self-control of on-task behavior

in an elementary classroom. Journal of Applied Behavior Analysis, 1973, 6, 105-113, Hallahan, D.P . Distractibility in the learning disabled child. In W. M. Cruickshaok & D.

P, Hatlahan (Eds.), Perceptual and learning disabilities in children (Vol. 2): Research and theory. Syracuse, NY: Syracuse University Press, 1975.

SELF- VS. TEACHER-ASSESSMENT 723

Haliahan, D. P., Lloyd, J., Kosiewicz, M. M., Kauffman, J. M., & Graves, A . W . Self- monitoring of attention as a treatment for a learning disabled boy's off-task behavior. Learning Disability Quarterly, 1979, 2, 24-32.

Hallahan, D. P., Marshall, K. J., & Lloyd, J .W. Self-recording daring group instruction: Effects on attention-to-task. Learning Disability Quarterly, 1981, 4, 407-413.

Hallahan, D. P., & Reeve, R . E . Selective attention and distractibility. In B. K. Keogh (Ed.), Advances in special education (VoL 1). Greenwich, CN: JAI Press, 1980.

Haynes, S. N., & Horn, W.F. Reactivity in behavioral observation: A review. Behavioral Assessment, in press.

Heins, E . D . Training learning disabled children's self-control: Cued and non-cued self- recording in the classroom ~Doctoral dissertation, University of Virginia, 1980). Dis- sertation Abstracts International, 1980, 41, 39~6A. (University Microfilms No. 8102575.)

Kanfer, F . H . Serf-regulation: Research, issues, and speculation. In C. Neuringer & J. L. Michael (Eds.), Behavior modification in clinical psychology. New York: Appleton- Century-Crofts, 1970.

Kanfer, F. H. Self-management methods. In F. H. Kanfer & A. P. Goldstein (Eds.), Helping people change. New York: Pergamon Press, 1975.

Kazdin, A . E . Reactive self-monitoring: The effects of response desirability, goal setting, and feedback. Journal of Consulting and Clinical Psychology, 1974, 42, 704-716. (a)

Kazdin, A .E . Serf-monitoring and behavior change. In M. J. Mahoney & C. E. Thoresen (Eds.), Se~f-control: Power to the person. Monterey, CA.: Brooks/Cole, 1974. (b)

Lloyd, J. W., Hallahan, D. P., Kosiewicz, M. M., & Kneedler, R .D. Reactive effects of self-assessment and self-recording on attention to task and academic productivity. Learning Disability Quarterly, in press.

Nelson, R. O., Lipinski, D. P., & Black, J . L . The relative reactivity of external observations and self-monitoring. Behavior Therapy, 1976, 7, 314-321.

Rachlin, H. Self-controf. Behaviorism, 1974, 2, 94-107. Young, P., Birnbrauer, J. S., & Sanson-Fisher, R .W. The effects of self-recording on the

study behavior of female juvenile delinquents. In B. C. Etzel, J. M. LeBlane, & D. M. Baer (Eds.), New developments in behavioral research: Theory, method, and application. Hil[sdale, N J: Erlbaum, 1977.

RECEJVEn: August 21, 1981 FINAL ACCEVrANCE; June 10, 1982

a comparison of the effects of self-versus teacher-assessment of on-task behavior

Documents