blog posts and news stories

Report Released on the Effectiveness of SRI/CAST's Enhanced Units

Summary of Findings

Empirical Education has released the results of a semester-long randomized experiment on the effectiveness of SRI/CAST’s Enhanced Units (EU). This study was conducted in cooperation with one district in California, and with two districts in Virginia, and was funded through a competitive Investing in Innovation (i3) grant from the U.S. Department of Education. EU combines research-based content enhancement routines, collaboration strategies and technology components for secondary history and biology classes. The goal of the grant is to improve student content learning and higher order reasoning, especially for students with disabilities. EU was developed during a two-year design-based implementation process with teachers and administrators co-designing the units with developers.

The evaluation employed a group randomized control trial in which classes were randomly assigned within teachers to receive the EU curriculum, or continue with business-as-usual. All teachers were trained in Enhanced Units. Overall, the study involved three districts, five schools, 13 teachers, 14 randomized blocks, and 30 classes (15 in each condition, with 18 in biology and 12 in U.S. History). This was an intent-to-treat design, with impact estimates generated by comparing average student outcomes for classes randomly assigned to the EU group with average student outcomes for classes assigned to control group status, regardless of the level of participation in or teacher implementation of EU instructional approaches after random assignment.

Overall, we found a positive impact of EU on student learning in history, but not on biology or across the two domains combined. Within biology, we found that students experienced greater impact on the Evolution unit than the Ecology unit. These findings supports a theory developed by the program developers that EU works especially well with content that progresses in a sequential and linear way. We also found a positive differential effect favoring students with disabilities, which is an encouraging result given the goal of the grant.

Final Report of CAST Enhanced Units Findings

The full report for this study can be downloaded using the link below.

Enhanced Units final report

Dissemination of Findings

2023 Dissemination

In April 2023, The U.S. Department of Education’s Office of Innovation and Early Learning Programs (IELP) within the Office of Elementary and Secondary Education (OESE) compiled cross-project summaries of completed Investing in Innovation (i3) and Education Innovation and Research (EIR) projects. Our CAST Enhanced Units study is included in one of the cross-project summaries. Read the 16-page summary using the link below.

Findings from Projects with a Focus on Serving Students with Disabilities

2020 Dissemination

Hannah D’ Apice presented these findings at the 2020 virtual conference for the Society for Research on Educational Effectiveness (SREE) in September 2020. Watch the recorded presentation using the link below.

Symposium Session 9A. Unpacking the Logic Model: A Discussion of Mediators and Antecedents of Educational Outcomes from the Investing in Innovation (i3) Program

Empirical Releases Final Report on HMH Fuse™ iPad App

Today Empirical and Houghton Mifflin Harcourt made the following announcement. You can download the report and research summary from the study using the links below.
Fuse research report
Fuse research summary

Study Shows HMH Fuse™ iPad® App Can Dramatically Improve Student Achievement

Strong implementation in Riverside Unified School District associated with nine-point increase in percentile standing

BOSTON – April 10, 2012 – A study of HMH Fuse: Algebra 1 app released today by research firm Empirical Education Inc. identifies implementation as a key factor in the success of mobile technology. The 2010–2011 study was a pilot of a new educational app from global education leader Houghton Mifflin Harcourt (HMH) that re-imagines the conventional textbook to fully deploy interactive features of the mobile device. The HMH Fuse platform encourages the use of personalized lesson plans by combining direct instruction, ongoing support, assessment and intervention in one easy-to-use suite of tools.

Empirical found that the iPad-using students in the four participating districts: Long Beach, Fresno, San Francisco and Riverside Unified School District (Riverside Unified), performed on average as well as their peers using the traditional textbook. However, after examining its own results, Riverside Unified found an increase in test scores among students taught with HMH Fuse compared to their peers. Empirical corroborated these results, finding a statistically significant impact equivalent to a nine-point percentile increase. The Riverside Unified teachers also reported substantially greater usage of the HMH Fuse app both in teaching and by the students in class.

“Education technology does not operate in a vacuum, and the research findings reinforce that with a supportive school culture and strategic implementation, technology can have a significant impact on student achievement,” said Linda Zecher, President and CEO of HMH. “We’re encouraged by the results of the study and the potential of mobile learning to accelerate student achievement and deepen understanding in difficult to teach subjects like algebra.”

Across all districts, the study found a positive effect on student attitudes toward math, and those students with positive attitudes toward math achieved higher scores on the California Standards Test.

The research design was a “gold standard” randomized control trial that examined the performance of eighth-grade students during the 2010-2011 school year. Each teacher’s classes were randomly assigned to either the treatment group that used the HMH Fuse app or the control group that used the conventional print format of the same content.

“The rapid pace of mobile technology’s introduction into K-12 education leaves many educators with important questions about its efficacy especially given their own resources and experience,” said Denis Newman, CEO of Empirical Education. “The results from Riverside highlight the importance of future research on mobile technologies that account for differences in teacher experience and implementation.”

To access the full research report, go to www.empiricaleducation.com. A white paper detailing the implementation and impact of HMH Fuse in Riverside is available on the HMH website.

New Education Pilot Brings Apple’s iPad Into the Classroom

Above: Empirical Education President Denis Newman converses with Secretary Bonnie Reiss and author, Dr. Edward Burger

They’re not contest winners, but today, dozens of lucky 8th grade Algebra 1 students enthusiastically received new iPad devices, as part of a pilot of the new technology.

California Secretary of Education Bonnie Reiss joined local officials, publishers, and researchers at Washington Middle School in Long Beach for the kick-off. Built around this pilot is a scientific study designed to test the effectiveness of a new iPad-delivered Algebra textbook. Over the course of the new school year, Empirical Education researchers will compare the effect of the interactive iPad-delivered textbook to that of its conventional paper counterpart.

The new Algebra I iPad Application is published by Houghton Mifflin Harcourt and features interactive lessons, videos, quizzes, problem solving, and more. While students have to flip pages in a traditional textbook to reveal answers and explanations, students using the iPad version will be able to view interactive explanations and study guides instantly by tapping on the screen. Researchers will be able to study data collected from usage logs to enhance their understanding of usage patterns.

Empirical Education is charged with conducting the study, which will incorporate the performance of over twelve hundred students from four school districts throughout California, including Long Beach, San Francisco, Riverside, and Fresno. Researchers will combine measures of math achievement and program implementation to estimate the new program’s advantage while accounting for the effects of teacher differences and other influences on implementation and student achievement. Each participating teacher has one randomly selected class using the iPads while the other classes continue with the text version of the same material.

Though the researchers haven’t come up with a way of dealing with jealousy from students who will not receive an iPad, they did come up with a fair way to choose the groups who would use the new high tech program. Classes who received iPads were determined by a random number generator.

REL West Releases Report of RCT on Problem-Based Economics Conducted with Empirical Ed Help

Three years ago, Empirical Education began assisting the Regional Educational Laboratory West (REL West) housed at WestEd in conducting a large-scale randomized experiment on the effectiveness of the Problem-Based Economics (PBE) curriculum.

Today, the Institute of Education Sciences released the final report indicating a significant impact of the program for students in 12th grade as measured by the Test of Economic Literacy. In addition to the primary focus on student achievement outcomes, the study examined changes in teachers’ content knowledge in economics, their pedagogical practices, and satisfaction with the curriculum. The report, Effects of Problem Based Economics on High School Economics Instruction is found on the IES website.

Eighty Arizona and California school districts participated in the study, which encompassed 84 teachers and over 8,000 students. Empirical Education was responsible for major aspects of research operations, which involved collecting, tracking, scoring, and warehousing all data including rosters and student records from the districts, as well as the distribution of the PBE curricular materials, assessments, and student and teacher surveys. To handle the high volume and multiple administrations of surveys and assessments, we created a detail-oriented operation including schedules for following up with survey responses where we achieved response rates of over 95% for both teacher and student surveys. The experienced team of research managers, RAs and data warehouse engineers maintained a rigorous 3-day turnaround for gathering end-of-unit exams and sending score reports to each teacher. The complete, documented dataset was delivered to the researchers at WestEd as our contribution to this REL West achievement.

Two-Year Study on Effectiveness of Graphing Calculators Released

Results are in from a two-year randomized control trial on the effectiveness of graphing calculators on Algebra and Geometry achievement. Two reports are now available for this project, which was sponsored by Texas Instruments. In year 1 we contrasted business as usual in the math classes of two California school districts with classes equipped with sets of graphing calculators and led by teachers who received training in their use. In the second year we contrasted calculator-only classrooms with those also equipped with a calculator-based wireless networking system.

The project tracked achievement through state and other standardized test scores and implementation through surveys and observations. For the most part, the experiment could not discern an impact as a result of providing the equipment and training for the teachers. Data from surveys and observations make clear that the technology was not used extensively (and by some, not at all) suggesting that training, usability, and alignment issues must be addressed in adoption of this kind of program. There were modest effects, especially for Geometry, but these were often not found consistently for the two measurement scales. In one case contradictory results for the two school districts suggests that researchers should use caution in combining data from different settings.

REL West Calls on Empirical for Assistance with Experiment on New Economics Program

WestEd, which holds the contract for the Regional Education Laboratory contract in the western region (REL West) has contracted with Empirical Education for the operations of a large randomized experiment involving over 75 high school economics teachers throughout California and Arizona. With a September 2007 start, the project has called for very rapid start up by Empirical Education staff including delivery and processing approximately 40,000 student tests and surveys, acquiring data from over 50 school districts and conducting web-based surveys with 84 teachers. The Problem-Based Economics program, developed by Buck Institute for Education is being tested in the context of a single-semester course. During the 2007-2008 school year, two cohorts of students will take the Test of Economic Literacy (TEL) as well as performance assessments and attitudinal surveys to test the impact of the new program compared to the materials in use in the classrooms of control teachers.