In each of the co-teaching classes, there has been a marked improvement in student outcomes, as revealed by the California Dashboard. For example, in 2017–2018 in the sixth grade co-teaching classroom at Armstrong Elementary, 64 percent of their students scored a 3 or 4 in math, although 84 percent of the cohort they taught entered as 1s or 2s from the previous year. At Cortez Elementary, similarly significant improvements were also recorded. Disaggregated data also showed important improvements in addressing gaps between targeted groups of students. A comparison between the 2017–2018 Math and English Language Arts (ELA) Smarter Balanced scores of students in the co-teaching classes and the rest of PUSD classrooms showed clear gains and improvements for African American students, Native American students, special education students, and English Learners. As an example, 50 percent of sixth grade African American students in the co-taught classrooms scored a 3 or 4 in Mathematics in comparison to 20.98 percent in PUSD. For Latinx students in the same sixth grade co-taught classrooms, 69.23 percent scored a 3 or 4 in ELA in comparison to 41.78 percent in the rest of PUSD classrooms.
While talking about her school data with other district PBIS leaders, Goens, principal at Simons Middle School, shared a brief story that illustrates some early indicators of change at her site:
The fact that we were at 1,136 (2012–2013) referrals in a year and now we are at 223 (2016–2017). That’s a lot less of instructional time missed. We’re also seeing our honor roll at over 56 percent of our students, which is bigger than ever. There is clear evidence of academic growth since we’ve become a PBIS school. Another one of our successes is our school climate report card. We finished 99 percentile for similar schools and 99 percentile for the state. I would say that’s kids’ perspective about the success that’s happening at the site. And I think when you look at those data points and you see how kids are viewing the school, and you hear them say things like “best middle school!” you really, you see it.
Leading Evidence of Change
The story of Simons Middle School is not unique. Four years after the pilot schools began PBIS implementation, the number of Office Discipline Referrals across PBIS schools has decreased by 48 percent and suspensions have decreased by 61 percent. Office Discipline Referrals decreased from 1,278 in 2014–2015 to 607 in 2017–2018. If we translate this number into its instructional impact, equating every referral to 45 minutes of missed class time, in 2014–2015, students missed 460,000 minutes of learning time (1,278 days). In 2017–2018, referrals were reduced to 5,369, equalling 240,000 minutes (671 instructional days), resulting in an overall increase of 607 days of instructional engagement time (see Figure 2).
Looking at the overall number of suspensions across the district, the district has seen an overall decrease over the past several years (see Figure 3).
Looking at the overall two-year comparison for ELA and Math PUSD data, the overwhelming majority of schools that increased both their ELA (82 percent) and Mathematics (86.7 percent) scores in PUSD were PBIS schools (see Figure 4).
PBIS at the School Site
Across PBIS sites, indicators show clear and significant progress from 2015–2016 (the first year of implementation) to 2017–2018:
- From the SCS, a School Climate Survey designed to gauge PBIS implementation as perceived by students: 78 percent of students at the PBIS feel accepted at school and 66 percent like attending school.
- From the TFI: Of the 27 sites where the PBIS team has completed the Tiered Fidelity Inventory, there has been a 35 percent increase in the overall average from the school year 2015–2016 to the 2017–2018 school year.
- From the SAS: At the 27 PBIS sites that have completed the Self-Assessment Survey, all teachers and staff provided input on PBIS implementation and rated: Expectations 97 percent, Expectations Taught 87 percent, Rewards System 80 percent, Violations 67 percent, Monitoring System 69 percent, Management 68 percent, District Support 83 percent, Overall 78 percent.
- DigiCOACH: 23/23 sites completed the DigiCOACH walkthroughs with four sites rated as Exemplary, 17 sites rated as Evident, and 2 sites rated as Emerging in Implementation.
School-Site Based PBIS Assessment Tools and Indicators
Once the leadership teams are established, school teams use several assessment tools and surveys for various purposes, which include informing action planning, appraising the status of drivers or elements related to the implementation of Schoolwide Positive Behavioral Interventions and Supports (SWPBIS), and tracking data that speaks to both fidelity of implementation and impact. These self-assessment tools and surveys are consistent across all states, meet the needs not currently duplicated by another survey are reliable, and are evidence based. Additionally, these tools are intended to build systemic capacity for sustainable, culturally and contextually relevant, high-fidelity implementation of multitiered practices and systems of support.
- School Climate Surveys (SCS) – a set of multidimensional surveys to measure the perceptions of students around school climate. These surveys are brief, reliable, and valid for assessing perceived school climate among students in grades 3-12.
- Tiered Fidelity Inventory (TFI) – a self-assessment tool used to measure the fidelity with which school personnel are applying the core features of SWPBIS at all three tiers.
- Self-Assessment Survey (SAS) – an annual assessment designed to gauge staff perception of the implementation status and improvement priorities for school-wide, classroom, nonclassroom and individual student systems. These results are especially important for informing future plans and implementation.
- DigiCOACH Walkthrogh Data – digiCOACH is a digital mobile system designed for instructional leaders to conduct classroom walkthroughs and to support the growth of educators through five key areas.
- PBIS Apps at Educational and Community Supports (n.d.). Positive Behavioral Interventions and Supports Assessment Surveys. Eugene, OR: University of Oregon. Retrieved from https://www.pbisapps.org/Applications/Pages/PBIS-Assessment-Surveys.aspx#tfi.
- OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports (May 2017). Positive Behavioral Interventions and Supports (PBIS) Implementation Blueprint: Part 2 – Self-Assessment and Action Planning. Eugene, OR: University of Oregon. Retrieved from www.pbis.org.
- DigiCOACH (n.d.). Redlands, CA. Retrieved from http://www.digicoach.com/editions/EducationalResearch.pdf.
Beyond the Data
While impact data has been promising across the district, most of the stories uncovered by this study highlighted the importance of acknowledging two key lessons: the first, that change takes time and the second, that sometimes the most profound and important changes are not fully captured by the traditional measures of impact data.
When presenting and sharing their data (even when it showed clear positive outcomes), many stakeholders and school leaders pointed out that they believed these were “leading indicators,” and that the real change is happening in ways not yet captured by current numbers.
Additionally, an ongoing challenge for PUSD heading into the future is capturing impact and being clear and intentional about the ways in which they gather, interpret, and communicate data. Morillo-Shone, while talking about the new California Dashboard and the way it tracks data, explained: “It is interesting what’s reported in the dashboard is suspensions and expulsions. We measure that too, and while we have had some progress, we now need to backwards map and think about broader strategies that capture data that is useful to us and that is able to speak to the drivers we need.” PBIS has not only emphasized the importance of collecting data but has also created lots of data. In turn, a new challenge is how to use it strategically.