The Impact of Probe Difficulty Variation on Brief Experimental Analysis of Reading Skills

Abstract

The impact of variation in probe difficulty on the ability to replicate results in brief experimental analysis (BEA) of reading was examined. In the first phase of the study, 41 first- and second- grade students completed 16 reading probes. Calculations of probe difficulty were used to identify Low and High Variability probe sets. In the second phase of the study, 40 second- through fifth-grade students’ performance on two reading interventions was compared in a BEA-like task. The best-performing intervention was unlikely to be replicated on either probe set (i.e., for only 43% of students); rather, the best determinant of intervention replication was each students’ average difference in performance across the two interventions. The best-performing intervention was more likely to be replicated (i.e., 60% of students) when averages of two trials per intervention were compared. These results are discussed in the context of developing rules for determining the best-performing intervention in academic BEA.