Science education has experienced a significant transition over the last decade, catalyzed by a re-envisioning of what students should know and be able to do in science. That re-envisioning culminated in the release of the Next Generation Science Standards (NGSS) in 2013. The new standards set off a chain reaction of standards adoption and implementation across states, districts, and schools, including steps taken toward transforming science professional learning, instruction, curriculum, and assessment.
It was in this dynamic context that Empirical Education conducted an impact evaluation, as part of an Investing in Innovation (i3) grant, of WestEd’s Making Sense of SCIENCE project, a teacher professional learning model aimed at raising students’ science achievement through improving science instruction. Under this grant, WestEd and Empirical also partnered with Heller Research Associates (HRA) to conduct an implementation study and a scale up study of Making Sense of SCIENCE. The impact evaluation was a two-year cluster-randomized control trial (RCT) that took place in California and Wisconsin across seven school districts and 66 elementary schools in the 2016–17 and 2017–18 school years. We randomized schools to either receive the Making Sense of SCIENCE professional learning or to the business-as-usual (“control”) group, which received the professional learning (delayed-treatment) after the study ended.
The study yielded a positive and significant impact on teacher content knowledge (standardized effect size = 0.56, p = .006) and teacher pedagogical content knowledge on a holistic scale (standardized effect size = 0.41, p = .026).
The impact on student science achievement using an evaluator-developed assessment was 0.064 standard deviations, but was not statistically significant (p = .494). Similarly, the impacts on student science achievement among students in the lowest third of incoming achievement in math and English Language Art were also positive but not statistically significant. The figure below displays the impacts on other exploratory student academic measures.
An exploratory analysis of intermediate outcomes revealed positive and significant impacts on the amount of time teachers spend on science instruction and on the emphasis teachers placed on NGSS-aligned instructional practices. Suggestive evidence that Making Sense of SCIENCE changes classroom science learning experiences in ways that align with expectations in NGSS, which is a hypothesized precursor to measuring impacts on student achievement, deserves notice.
We identified three potential contributors to the findings of limited impact on student science achievement in this study. We elaborate on each of these points in the report.
Our research dissemination for this study spanned the entire 5 years, starting with registering the study in the REES registry and ending with writing a story about our lessons learned from our two i3 evaluations: the impact of CREATE and MSS.
This study was presented at the 2020 virtual conference for the Society for Research on Educational Effectiveness (SREE) in September 2020. Download the slide deck or watch the video.
Following the study, Empirical Education participated in an interactive video series and online dialogue—Making the Change to Next Generation Science Teaching and Learning—organized by WestEd (in September 2020) to share practical insights and research findings from national experts, district and state leaders, and teachers who participated in the study. The three videos below present the impact study’s key findings, voices from the field on science education, and lessons learned.
