blog posts and news stories

New Reports Show Positive Results for Elementary Reading Program

Two studies of the Treasures reading program from McGraw-Hill are now posted on our reports page. Treasures is a basal reading program for students in grades K–6. Although the first study was a multi-site study while the second was conducted in the Osceola school district, both found positive impacts on reading achievement in grades 3–5.

The primary data for the first study were scores supplied with district permission by Northwest Evaluation Association from their MAP reading test. The study uses a quasi-experimental comparison group design based on 35 Treasures and 48 comparison schools primarily in the midwest. The study found that Treasures had a positive impact on overall elementary student reading scores, the strongest effect being observed for grade 5.

The second study’s data were provided by the Osceola school district and consist of demographic information, FCAT test scores, and information on student transfers during the year (between schools within the districts and from other districts). The dataset for this time series design covered five consecutive school years from 2005–06 to 2009–10, including two years prior to introduction of the intervention and three years after the introduction. The study included exploration of moderators that demonstrated a stronger positive effect for students with disabilities and English learners than the rest of the student population. We also found a stronger positive impact on girls than on boys.

Check back for results from follow-up studies, which are currently underway in other states and districts.

Poway Completes Study from MeasureResults Pilot

The results are in for Poway Unified School District’s first research study using our MeasureResults online tool. PUSD was interested in measuring the impact of CompassLearning’s Odyssey Reading program in the middle grades. Using an “interrupted time series” design with a comparison group, they found that both 7th and 8th grade students averaged 1 to 2 points higher than expected on the NWEA MAP Literacy assessment. PUSD plans to continue their evaluation of CompassLearning Odyssey in different subject areas and grade levels. Join us in D.C. at this year’s CoSN conference on March 1, 2010 as Eric Lehew, Executive Director of Learning Support Services at PUSD, presents findings and reflections on the process of using MeasureResults to conduct research at the local district level.

Click here to download a copy of the PUSD achievement report.

Empirical Education Partners with NWEA to Research Virtual Control Groups

Northwest Evaluation Association, the leading provider of computer adaptive testing for schools, is partnering with Empirical Education to analyze the properties of its virtual control group (VCG) technologies. Empirical has already conducted a large number of randomized experiments in which NWEA’s “Measures of Academic Progress” (MAP) served both as pretest and posttest. The characteristics of a randomly assigned control group provide a yardstick in evaluating the characteristics of the VCG. The proposed research builds on extensive theoretical work on approaches to forming comparison groups for obtaining unbiased impact estimates from quasi-experiments.

In parallel to this theoretical analysis, NWEA and Empirical Education are cooperating in a nationwide comparison group (“quasi-”) experiment to estimate the impact of a basal reading program in wide use nationally. Taking advantage of the fact that MAP is in use in thousands of schools, Empirical will identify a group of schools currently using this reading program and testing their students’ reading using MAP and then select a well matched comparison group from non-users who also test with MAP. Characteristics of the schools such as SES, percent English learner, urbanicity, ethnicity, and geographic region, as well as prior reading achievement, will be used in identifying the comparison group.

Blue Valley Schools and Empirical Education Explain Local Program Evaluations

Dr. Bo Yan, Program Evaluator for the Blue Valley Schools in Kansas, and Dr. Denis Newman, president of Empirical Education, co-presented at the Northwest Evaluation Association’s annual Members Seminar in Portland OR. The topic was how school districts can use their own testing such as that administered by the NWEA member districts to conduct their own local program evaluations. Dr. Yan, who is expecting to conduct seven such evaluations in his district this coming year, used an evaluation of READ 180 as an illustration of a comparison group design using primarily statistical controls. Dr. Newman presented the randomized control work Empirical Education has done with the Maui school system to evaluate math software and curriculum from Carnegie Learning (Cognitive Tutor: Year 1 and Year 2). Both emphasized the importance of local evaluations to estimate the impact of the programs for the specific populations and resources available to the district. They also made clear the need for a comparison group from the local district since the improvement a district can expect is anchored in its own students’ current level of achievement. While the presentation focused mainly on the use of NWEA’s Measures of Academic Progress (MAP) in the quantitative estimation of the program’s impact, the presenters also emphasized the importance of gathering information on implementation and of the conversations that must go on to integrate evaluation findings into the district’s decision-making. NWEA will be providing this presentation, along with the PowerPoint slides, as a podcast.

Learning Point and Empirical Partner for Research on Formative Assessment

Learning Point Associates, which holds the contract for the Midwest Regional Education Lab, has contracted with Empirical Education for the operations of a large randomized experiment expected to include more than 100 elementary school teachers when in full swing in the fall of 2008. A pilot experiment, beginning this fall, involves a small number of schools in Illinois. The experiment will test the effectiveness of Northwest Evaluation Association’s formative assessment and professional development to be used in fourth- and fifth-grade classrooms. Working with the principal investigators Matt Dawson of LPA and David Cordray of Vanderbilt, Empirical will be responsible for recruiting schools, acquiring and warehousing student data, and conducting observations and surveys.