Dependability of Two Scaling Approaches to Direct Behavior Rating Multi-Item Scales Assessing Disruptive Classroom Behavior

This study examines the dependability of two scaling approaches for using a five-item Direct Behavior Rating multi-item scale to assess student disruptive behavior. A series of generalizability theory studies were used to compare a traditional frequency-based scaling approach with an approach wherein the informant compares a target student's behavior with that of classroom peers. A total of seven novice raters (i.e., graduate students) used both types of scales to rate 10-min video clips of the classroom behavior of nine middle school students across three occasions. Generalizability of composite scores derived from each type of scale was examined across raters and occasions. Subsequent decision studies were conducted to determine the number of measurement occasions that would be required to obtain an acceptable level of dependability. Results of these studies indicated that the type of scale accounted for a substantial proportion of variance (29%) and that the traditional frequency approach required far fewer assessment occasions to reach the criterion for absolute and relative decisions (4 and 8 occasions, respectively) compared with the comparative scaling approach (>30 occasions). Implications for future research and current practice are discussed.