blog posts and news stories

Quasi-experimental Design Used to Build Evidence for Adolescent Reading Intervention

A study of Jamestown Reading Navigator (JRN) from McGraw-Hill (now posted on our reports page), conducted in Miami-Dade County Public Schools, found positive results on the Florida state reading test (FCAT) for high school students in their intensive reading classes. JRN is an online application, with internal record keeping making it possible to identify the treatment group for a comparison design. While the full student, teacher and roster data for 9th and 10th grade intensive reading classes were provided by the district, JRN—as an online application—provided the identification of the student and teacher users through the computer logs. The quasi-experimental design was strengthened by using schools with both JRN and non-JRN students. Of the 70 schools that had JRN logs, 23 had JRN and non-JRN intensive reading classes and sufficient data for analysis.

Download the 2010 report here.

Empirical Education Partners with Carnegie Learning on New Student Performance Guarantee

Schools looking to improve student Algebra and Geometry achievement have signed up for a guarantee from Carnegie Learning® that states that students using the company’s Cognitive Tutor programs will pass their math courses. Empirical Education is tasked to monitor student performance in participating schools. Starting this school year, Carnegie Learning guarantees that students who take three complete and consecutive years of Carnegie Learning’s math courses will pass their math class in the third year. The guarantee applies to middle and high school students taking the Carnegie Learning Bridge to Algebra, Algebra, Algebra II, and Geometry courses.

In the coming weeks/months, Empirical will collect roster data, course grades, and assessment scores from schools as well as usage data from Carnegie Learning’s math teaching software. These data will be combined to generate biannual reports that will provide schools with evidence they can use to effectively improve implementation of the courses and raise student achievement.

Carnegie Learning’s guarantee is part of their School Improvement Grant support efforts. “Partnering with Empirical Education will allow us to get mid- and end-of-year research reports into the hands of our school partners,” says Steve Ritter, Co-Founder and Chief Scientist at Carnegie Learning. “It’s part of our continuous improvement cycle; we’re excited to see the progress districts committed to the turnaround and transformation process can make with these new, powerful tools.”

Empirical Education is Part of Winning i3 Team

Of the almost 1700 grant applications submitted to the federal Investing in Innovation (i3) fund, the U.S. Department of Education chose only 49 proposals for this round of funding. A proposal submitted by our colleagues at WestEd was the third highest rated. Empirical Education assisted in developing the evaluation plan for the project. The project (officially named “Scaling Up Content-Area Academic Literacy in High School English Language Arts, Science and History Classes for High Needs Students”) is based on the Reading Apprenticeship model of academic literacy instruction. The grant will span five years and total $22.6 million, including 20 percent in matching funds from the private sector. This collaborative effort is expected to include 2,800 teachers and more than 400,000 students in 300 schools across four states. The evaluation component, on which we will collaborate with researchers from Academy for Educational Development, will combine a large scale randomized control trial with extensive formative research for continuous improvement of the innovation as it scales up.

REL West Releases Report of RCT on Problem-Based Economics Conducted with Empirical Ed Help

Three years ago, Empirical Education began assisting the Regional Educational Laboratory West (REL West) housed at WestEd in conducting a large-scale randomized experiment on the effectiveness of the Problem-Based Economics (PBE) curriculum.

Today, the Institute of Education Sciences released the final report indicating a significant impact of the program for students in 12th grade as measured by the Test of Economic Literacy. In addition to the primary focus on student achievement outcomes, the study examined changes in teachers’ content knowledge in economics, their pedagogical practices, and satisfaction with the curriculum. The report, Effects of Problem Based Economics on High School Economics Instruction is found on the IES website.

Eighty Arizona and California school districts participated in the study, which encompassed 84 teachers and over 8,000 students. Empirical Education was responsible for major aspects of research operations, which involved collecting, tracking, scoring, and warehousing all data including rosters and student records from the districts, as well as the distribution of the PBE curricular materials, assessments, and student and teacher surveys. To handle the high volume and multiple administrations of surveys and assessments, we created a detail-oriented operation including schedules for following up with survey responses where we achieved response rates of over 95% for both teacher and student surveys. The experienced team of research managers, RAs and data warehouse engineers maintained a rigorous 3-day turnaround for gathering end-of-unit exams and sending score reports to each teacher. The complete, documented dataset was delivered to the researchers at WestEd as our contribution to this REL West achievement.

Report completed on the effectiveness of MathForward

Empirical Education assisted Richardson Independent School District (RISD) in conducting an evaluation of the MathForward algebra readiness program published by Texas Instruments. RISD implemented MathForward in their 7th and 8th grade general mathematics and 9th grade Algebra I classes. The research employed an interrupted time series design comparing existing student achievement scores with MathForward to student achievement scores from the three years prior to the introduction of MathForward.

The results of this four-year study suggest that 7th grade students scored, on average, 11 percentile points higher with MathForward than the 7th grade students from the three previous years without MathForward. A similar result for the 8th grade suggests that students participating in MathForward scored, on average, 9 percentile points higher. While the trend did not hold for 9th grade, further exploration suggests that 9th grade students whose teachers had 3 years experience using MathForward scored higher than 9th grade students whose teachers did not use MathForward.

The report further illustrates how an interrupted time series design can be used to study a program as it is rolled out over several years. This research will be presented at the 2010 AERA conference in Denver, Colorado (Friday, April 30 — Tuesday, May 4).

New MeasureResults® Webpage

Empirical Education clients and partners can now keep track of the latest MeasureResults developments on our new webpage. The webpage features video clips that explain how MeasureResults’ online interface can be used to set up and conduct school level studies, as well as details about the design and development of program features. Updates and announcements will be posted on a regular basis. In addition to serving our existing MeasureResults subscribers, Empirical Education is currently developing a custom-designed MeasureResults tool for use in more than ten school districts for a high-school math effectiveness study.

REL West Calls on Empirical for Assistance with Experiment on New Economics Program

WestEd, which holds the contract for the Regional Education Laboratory contract in the western region (REL West) has contracted with Empirical Education for the operations of a large randomized experiment involving over 75 high school economics teachers throughout California and Arizona. With a September 2007 start, the project has called for very rapid start up by Empirical Education staff including delivery and processing approximately 40,000 student tests and surveys, acquiring data from over 50 school districts and conducting web-based surveys with 84 teachers. The Problem-Based Economics program, developed by Buck Institute for Education is being tested in the context of a single-semester course. During the 2007-2008 school year, two cohorts of students will take the Test of Economic Literacy (TEL) as well as performance assessments and attitudinal surveys to test the impact of the new program compared to the materials in use in the classrooms of control teachers.