blog posts and news stories

Maui Schools Sign Subscription Agreement for Empirical Education Research Services

Empirical Education will be providing research services to the Maui School District through an innovative subscription arrangement for MeasureResults™, an Internet-based research and consulting offering. The initial application of this service will be investigations of the longer term impact of the Cognitive Tutor program that was implemented under a Math and Science Partnership program grant.

The company’s MeasureResults service is a response to the ever increasing demands on school systems to validate their program and spending decisions based on the analysis of solid data. Most districts do not have the staff and facilities to set up data and run complex statistical analyses. In Maui, the service will take advantage of the sophisticated data warehousing capabilities being put in place statewide. MeasureResults is designed to simplify the technical and logistical steps of conducting experiments by building powerful and verified analytical techniques into an uncomplicated framework. The offering includes consultative services on research design and web-based interfaces to gather data, automate analysis, and generate reports. MeasureResults is bundled with technical support that includes review of all analyses and reports by trained statisticians.

Empirical at 2008 AERA Conference

Acceptance notifications are in! Empirical Education will have a strong showing at the 2008 American Educational Research Association conference, which will be held in New York on March 24–28, 2008. Empirical staff will be presenting their research and findings in several divisions and special interest groups, including School Evaluation and Program Development, Learning and Instruction, Survey Research in Education, and Measurement and Research Methodology: Quantitative Methods and Statistical Theory. Our six presentations will include a variety of topics:

• Evaluating Alabama’s Math, Science, and Technology Initiative: Initial Results of a State-Wide Randomized Experiment
• Matched-Pairs Designs and Standard Errors of Impact Estimates: Lessons from Ten Experiments
• Effects of Pairing on the Precision of Moderator Effects: Lessons from Nine Cluster Randomized Trials
• Comparative Effectiveness of the TI-84 Graphing Calculator on Geometry Outcomes
• Multi-Site Randomized Field Trial of an Elementary Science Curriculum
• Measuring Time and Implementation in the Alabama Mathematics, Science, and Technology Initiative

View pictures of Empirical Education’s reception at The Chocolate Bar in NYC.

Empirical Education Pilots Workshop at REL-NEI Regional Meeting

Providence, RI was the site of REL Northeast and Islands’ 2007 Regional Meeting on Teacher Quality. Empirical Education staff headed out to the east coast to pilot the “Becoming Good Consumers of Research” workshop to an audience of about 30 education researchers, school administrators, university professors, and other education professionals interested in using research to inform school decisions. Gloria Miller, the company’s director of evaluation design, facilitated group discussions that touched on identifying different types of research and potential sources of bias. Gloria also provided participants with a Critical Reader’s toolkit designed to help readers evaluate the trustworthiness and relevance of various pieces of research. “Consumers of Research” is the first in a planned series of workshops focused on increasing the understanding and use of research in schools. The next round of workshops, scheduled for March 2008, is currently being developed.

Report Released on the Effectiveness of Carnegie Learning’s Bridge to Algebra

Empirical Education released the results of a year-long randomized experiment on the comparative effectiveness of Carnegie Learning’s Bridge to Algebra program. This study, like that on Carnegie Learning’s Cognitive Tutor for Algebra (see news for May 24, 2007), was conducted in cooperation with the Maui School District in Hawai’i and funded through a grant to Empirical Education from the U.S. Department of Education. The experiment could not discern an overall difference between the program and control group measured by either NWEA’s test for general math or the test’s algebraic operations subscale. However, for the algebraic operations outcomes (but not for general math), we found that students scoring low before participating in Bridge to Algebra benefited significantly more from the program’s algebraic operations instruction than did students with high initial scores. The district was specifically interested in looking at how the different ethnic groups, particularly the Hawaiian/Part-Hawaiian and Filipino students, performed in the new program. Controlling for pretest, we did not find that the program had a different effect for different ethnicities. The district was also interested in learning whether the program was differentially effective for students taught by certified teachers versus those with non–certified teachers. For the overall score (but not the algebraic operations sub-strand), we found that the program gave the non–certified teachers an advantage. Finally, despite some implementation challenges, teachers reported a generally positive attitude toward the new program.

REL West Calls on Empirical for Assistance with Experiment on New Economics Program

WestEd, which holds the contract for the Regional Education Laboratory contract in the western region (REL West) has contracted with Empirical Education for the operations of a large randomized experiment involving over 75 high school economics teachers throughout California and Arizona. With a September 2007 start, the project has called for very rapid start up by Empirical Education staff including delivery and processing approximately 40,000 student tests and surveys, acquiring data from over 50 school districts and conducting web-based surveys with 84 teachers. The Problem-Based Economics program, developed by Buck Institute for Education is being tested in the context of a single-semester course. During the 2007-2008 school year, two cohorts of students will take the Test of Economic Literacy (TEL) as well as performance assessments and attitudinal surveys to test the impact of the new program compared to the materials in use in the classrooms of control teachers.

Learning Point and Empirical Partner for Research on Formative Assessment

Learning Point Associates, which holds the contract for the Midwest Regional Education Lab, has contracted with Empirical Education for the operations of a large randomized experiment expected to include more than 100 elementary school teachers when in full swing in the fall of 2008. A pilot experiment, beginning this fall, involves a small number of schools in Illinois. The experiment will test the effectiveness of Northwest Evaluation Association’s formative assessment and professional development to be used in fourth- and fifth-grade classrooms. Working with the principal investigators Matt Dawson of LPA and David Cordray of Vanderbilt, Empirical will be responsible for recruiting schools, acquiring and warehousing student data, and conducting observations and surveys.

Major Study of Elementary Science Reveals Reading Improvement

Empirical Education released the report of a randomized control trial of Scott Foresman Science. The report concludes that the program shows promise as an enhancement of the school’s reading program. The study encompassed five school districts in five different states and included more than 80 third- through fifth-grade teachers divided randomly between those who were trained and provided with the science text and materials and those who continued with their existing science materials. The Scott Foresman Science materials were geared to reading, providing leveled readers corresponding to each chapter of the text. The study was sponsored by Pearson Education.

Empirical Education Presents at Annual IES Conference

Empirical Education staff members were part of two presentations at the Institute of Education Sciences (IES) Research Conference in Washington D.C. Xin Wei and Denis Newman presented a study on paired randomization, a technique that the company uses extensively to improve the design of local school district experiments. The research, led by Andrew Jaciw, the company’s director of research design and analysis, drew on data from 12 experiments and provided an analysis of the statistical advantages of the approach. A full report is being prepared for presentation at AERA. The work is supported by a grant from the IES. A second presentation provided a preview of the state-wide experiment evaluating the effectiveness of the Alabama Math, Science and Technology Initiative. Empirical Education will begin analyzing the outcomes for the first year in fall 2007. Denis Newman and his co-PI, Richard Sawyer of Academy for Educational Development, led the presentation. The work is part of the Regional Education Lab in the Southeast.

Experiment Findings about Cognitive Tutor Not What Was Expected

Empirical Education released the results of its year-long randomized experiment on the comparative effectiveness of the Cognitive Tutor program, conducted in cooperation with the Maui School District in Hawai’i. Funded through a grant from the U.S. Department of Education to Empirical, the research evaluated the local implementation of Carnegie Learning’s Algebra I product. Company president Denis Newman explained the results, saying, “We were surprised by the results and consider this study something we can all learn from. Nobody expected a negative impact on the experienced teachers but this may show that it takes some time to get back up to their high level of performance when learning a new way of teaching. Following these teachers’ progress into a second year would help to verify this explanation. This effect was not found in previous research, which again points to the importance of conducting many experiments on any intervention before coming to a conclusion about its effectiveness.” A second experiment in the same school system, focused on the new Bridge to Algebra product, is nearing completion.

Empirical Education Reports Research at AERA

Staff and collaborators in Empirical Education research presented their findings this month at the American Educational Research Association’s annual conference—the primary meeting of the education research community. The company’s research and expertise was highlighted in six places on the program including contributions in the divisions for Learning & Instruction, Policy & Politics, Measurement & Research Methodology, and School Evaluation & Program Development.