+1443 776-2705 panelessays@gmail.com
  

1 pg summary on article attached below

335

JOURNAL OF APPLIED BEHAVIOR ANALYSIS 2000, 33, 335–338 NUMBER 3 (FALL 2000)

FURTHER EVALUATION OF THE ACCURACY OF
REINFORCER SURVEYS: A SYSTEMATIC REPLICATION

JOHN NORTHUP
LOUISIANA STATE UNIVERSITY

The present report evaluates the accuracy of a reinforcer survey by comparing the survey
results to the results of subsequent reinforcer assessments for 20 children using a con-
current-operants arrangement to assess relative reinforcer preference. Total accuracy for
the survey was determined to be approximately 57%. The results provide a systematic
replication of Northup et al. (1996) with a much larger sample of children. A need for
the development of more accurate and comprehensive reinforcer assessment methods for
verbal children is discussed.

DESCRIPTORS: reinforcer assessment, functional assessment, surveys, attention def-
icit hyperactivity disorder

Self-reports of potential reinforcers con-
tinue to be widely used to develop treat-
ments for the most common and pervasive
childhood behavior problems. The use of re-
inforcer surveys to identify potential rein-
forcers has a long history in child behavior
therapy. The practice appears to be well es-
tablished, and continued use may be further
supported by the ease and apparent efficien-
cy of administration. Nevertheless, poor cor-
respondence between verbal self-reports and
subsequent behavior has been long noted
and often demonstrated (e.g., Bernstein &
Michael, 1990; Risley & Hart, 1968). Chil-
dren’s ability to accurately name events that
will reinforce future behavior may be partic-
ularly questionable. Northup, George, Jones,
Broussard, and Vollmer (1996) demonstrat-
ed that correspondence between the results
of a reinforcer survey and a subsequent eval-
uation of reinforcement effects was no better
than chance for 4 children with a diagnosis
of attention deficit hyperactivity disorder
(ADHD). However, the evaluation of only
4 subjects for a comparison of this type is a
major limitation of the Northup et al. study.

Correspondence and requests for reprints should be
sent to John Northup, Department of Psychology, 236
Audubon Hall, Louisiana State University, Baton
Rouge, Louisiana 70803.

The Northup et al. (1996) study has ap-
parently not been replicated, and no other
similar evaluations of the accuracy and treat-
ment utility of reinforcer surveys could be
found. The present report provides a system-
atic replication of the procedures of Northup
et al. by (a) conducting a comparison of the
results of a reinforcer survey and subsequent
reinforcer assessments for a much larger
sample of 20 children, and (b) presenting
potential reinforcers (token coupons) simul-
taneously in a more efficient concurrent-op-
erants arrangement rather than singly as in
Northup et al. (1996). Presenting the token
coupons simultaneously is both more effi-
cient and provides a more direct assessment
of relative reinforcement effects.

METHOD

Participants

The files of all children who attended a
summer program for children with a diag-
nosis of ADHD during the past 5 years were
reviewed. Children were included in this
evaluation on the basis of the following cri-
teria: (a) The results of a reinforcer survey
as described below were available; (b) the re-
sults of a reinforcer assessment that included
at least three baseline sessions, three assess-

336 JOHN NORTHUP

ment sessions, and three return to baseline
sessions were also available; and (c) the re-
sults had not been published elsewhere.

General Procedure

The survey and reinforcer assessment pro-
cedures were similar to those of Northup et
al. (1996) with the following exceptions: (a)
Token coupons were presented simulta-
neously in a concurrent-operants arrange-
ment, (b) a separate category for peer atten-
tion was included, and (c) easy math prob-
lems, rather than a coding task, were used
during the reinforcer assessment. Token cou-
pons were used to represent a wide range of
potential reinforcers that are common to this
population but that are difficult or impos-
sible to tangibly or immediately provide dur-
ing an assessment (e.g., to get out of a school
activity).

Reinforcer survey. The reinforcer survey in-
cluded a total of 42 items, with seven items
representing each of the following six cate-
gories; edible items, peer attention, activities,
tangible items, teacher attention, and escape.
The survey was administered verbally with
the instruction that ‘‘I’m going to name
some things that kids sometimes get in
school. I want to know how much you like
each of these things. After I name each thing
you tell me if you like it ‘not at all,’ ‘a little,’
or ‘a lot.’ For example . . .’’ Rankings of each
item were given the value of 1, 2, and 3, for
not at all, a little, and a lot, respectively, and
a percentage score was calculated for each
category by dividing the summed score of
the item rankings by the total possible score.
Categories with a score of 75% or greater
were considered to be high preference. The
complete survey is available from the author.

Reinforcer assessment. During baseline, the
child was seated across from an examiner,
given a worksheet containing easy math
problems (i.e., problems previously complet-
ed with greater than 90% accuracy), and giv-
en the instruction that ‘‘you can do as much

as you want, as little as you want, or none
at all.’’ The session ended if the child stated
that he or she did not want to do any, said
that he or she was done, or worked for 5
min. During the reinforcer assessment, to-
ken coupons representing each of the six re-
inforcer categories and the associated stimuli
on the survey (and a control coupon) were
placed directly above the worksheet. The
stimuli associated with each coupon were re-
viewed and sampled by each child prior to
the assessment. The child was given the
same instructions as during baseline but was
also told that if he or she completed a cri-
terion number of problems he or she could
stop and take a coupon of his or her choice
and that he or she could earn as many cou-
pons of any kind as he or she wanted. The
criterion number of problems required to
choose a coupon was determined individu-
ally based on the average number of prob-
lems completed during baseline and was
marked on the worksheet prior to the ses-
sion. Problems did not have to be completed
correctly; however, the correct number of
digits for each answer had to be provided
(accuracy was typically 100%, in that prob-
lems were predetermined to be easy). Ses-
sions ended as in baseline except that no
time limit was imposed and the child was
allowed to work (and choose coupons) until
he or she stated that he or she was done. A
return to baseline was conducted to dem-
onstrate any reinforcement effects associated
with the token coupons. The chosen cou-
pons could be exchanged for any of the as-
sociated stimuli at any time upon request, as
in Northup et al. (1996).

RESULTS AND DISCUSSION

The overall accuracy of the survey was de-
termined as follows. For the survey, each cat-
egory of stimuli (i.e., edible items, peer at-
tention, etc.) was first determined to be ei-
ther high (.75%) or low (,75%) prefer-

337REINFORCER SURVEYS

Table 1
Accuracy of the Reinforcer Survey

True
positives

True
negatives

False
positives

False
negatives

Total
possibilities

Total
accuracy

41
(34%)

28
(23%)

35
(29%)

16
(13%)

120 57%

ence for each participant. For the reinforcer
assessment, the total number of problems as-
sociated with each coupon was first deter-
mined for each session (i.e., all problems
completed prior to each coupon selection
were attributed to the coupon selected).
Subsequently, all token coupons associated
with clear reinforcement effects were iden-
tified for each participant. The criteria for
clear reinforcement effects were the same as
in Northup et al. (1996) and were as fol-
lows: (a) The number of completed math
problems associated with the coupon was
greater than baseline, and (b) the number
completed was greater than those for the
control coupon for each session. The accu-
racy of the reinforcer survey then was cal-
culated by determining the number of (a)
true positives (categories identified as high
preference on the survey that functioned as
reinforcers), (b) false positives (categories
that were identified as high preference that
did not function as reinforcers), (c) true neg-
atives (categories that were identified as low
preference on the survey that did not func-
tion as reinforcers), and (d) false negatives
(categories that were identified as low pref-
erence on the survey that did function as
reinforcers). Total accuracy was calculated as
the sum of true positives and true negatives
divided by the total of all possibilities.

Table 1 shows the overall accuracy of the
reinforcer survey compared to the results of
the reinforcer assessments. The total accu-
racy of 57% was nearly identical to the 55%
accuracy reported by Northup et al. (1996).
The percentage of true and false positives
and negatives was also quite similar and nev-

er varied by more than 7%. Overall, the re-
sults of this comparison replicate those of
Northup et al. (1996) with a much larger
sample of children. The results further sug-
gest that reinforcer surveys may add little to
an assessment beyond chance. The relatively
high number of false positives and low num-
ber of false negatives again suggest that sur-
veys may more accurately identify stimuli
that are not reinforcers than those that are.
False positive results would typically be ex-
pected to be of much greater concern than
false negative ones in applied settings. That
is, selecting stimuli for use in a behavioral
treatment that do not in fact function as re-
inforcers (a false positive) will presumably
result in treatment failure, whereas a failure
to identify a potential reinforcer (false neg-
ative) may be of little or no concern (as long
as at least one true reinforcer is identified).
Because the results were very similar to those
of Northup et al. (1996), they further sug-
gest that presenting the token coupons si-
multaneously, rather than singly, had little or
no effect on overall results.

A limitation of the procedures is that the
potential effect of a single highly preferred
item on the survey could have been masked
by its inclusion in a category with other low-
preference items (a false negative for only
that item). However, an inspection of indi-
vidual survey responses did not indicate this
outcome, and a comparison of overall cate-
gories was of primary interest in this study.
Similarly, it is possible that the absolute val-
ue of any one coupon may be obscured by
the simultaneous presentation of all cou-
pons. However, reinforcer assessments are

338 JOHN NORTHUP

most likely to be conducted in applied set-
tings when there is a particular interest in
identifying the most potent or valued rein-
forcers for use in a behavioral treatment. It
may also be considered a design limitation
of this study that survey categories received
a score independent of any comparison to
any other category.

Some stimuli, such as access to widely
popular edible items and toys, may typically
function as reinforcers for so many children
that no formal assessment is needed. How-
ever, the current challenge for behavior an-
alysts is to accurately identify naturally oc-
curring contingencies that maintain problem
behaviors such as the relative value of teach-
er, parent, and peer attention, or escape from
aversive events. What is needed is the further
development of practical and efficient as-
sessment methods for verbal children that
can more accurately identify such contingen-
cies. For example, the further development
of verbal choice procedures and the evalua-
tion of other procedures (e.g., instructions)
or conditions that may mediate reinforcer

assessment accuracy for verbal children may
be fruitful directions for future research. The
use of more experimentally based reinforcer
assessments, such as those described in this
report, may offer both a valid criterion mea-
sure for future research with verbal children
and an interim alternative method of assess-
ment if a more rigorous reinforcer assess-
ment is considered necessary or desirable.

REFERENCES

Bernstein, D. J., & Michael, R. L. (1990). The utility
of verbal and behavioral assessment of value. Jour-
nal of the Experimental Analysis of Behavior, 54,
173–184.

Northup, J., George, T., Jones, K., Broussard, & Voll-
mer, T. (1996). A comparison of reinforcer as-
sessment methods: The utility of verbal and pic-
torial choice procedures. Journal of Applied Behav-
ior Analysis, 29, 201–212.

Risley, T. R., & Hart, B. (1968). Developmental cor-
respondence between the nonverbal and verbal be-
havior of preschool children. Journal of Applied
Behavior Analysis, 1, 267–281.

Received April 20, 1999
Final acceptance May 14, 2000
Action Editor, Cathleen Piazza