Empirical Education helps school systems at the local, intermediate, and state levels generate and interpret evidence to help inform their decisions. We work directly with school systems to evaluate programs such as Invest in Innovation (i3) Grants and School Improvement Grants (SIG).

We also offer schools systems the opportunity to pilot new programs in the context of vendor-sponsored and federally-funded studies: see our Participation page. What follows are answers to frequently asked questions about conducting studies in school districts.

FAQ by School Systems

Please click on a question to learn the answer.

• My district already has data and evaluation experts. Why should I use an outside research group?

  Empirical Education’s professionals are experts in a very specialized area of research design and statistics that can help you predict how effective an intervention will be for your teachers and students. Using this level of scientific data is now generally required for many U.S. Department of Education grant programs.

  Since the time needed to develop and implement this kind of research design internally may be more than can be expected from a school district, it can be done efficiently by outside specialists.

• How is Empirical Education’s research different?

  Unlike academic researchers, we are not pursuing our own separate research agenda. Unlike many other evaluators, Empirical Education is solely focused on effectiveness research for instructional programs and is able to utilize the most advanced design and statistical analysis methods for this task. Because we understand the unique requirements of both school communities and the publishing world, our approach helps facilitate communication between schools and publishers.

• I’ve heard of the RCT. What exactly is the research activity?

  Basically, we design what you might consider a pilot of the new program. By working with the district to set up two equivalent groups—one using the new intervention and the other not—we conduct a scientific experiment in the classic “clinical trials” format.

  Our service includes design of the experiment, assistance with training and implementation, recruiting school and classroom sites, management of the research operations, data collection and analysis, and reporting results. A professional research report is the final product.

• Is the RCT evaluation based on scientifically based research methods as defined by the government, specifically NCLB guidelines?

  Yes, it is. Our rigorous design utilizes random assignment of teachers or classrooms to assure a fair test, a key requirement of scientifically based research as defined by the U.S. Department of Education. The department’s What Works Clearinghouse (WWC) is a reviewer of our research methods and findings. In many ways we go beyond the minimum requirements of the WWC in using multilevel modeling and other advanced statistical methods.

• Schools are complicated. What if things go wrong in the implementation?

  We know that many factors are involved when implementing a new program in a school. In addition to the quantitative data, we work with the district to collect surveys and interviews with the teachers and to record observations throughout the study. Unlike with data-mining studies, we’re able to keep a detailed record of activity during the course of the research that will shed additional light on the results.

• How long does a pilot study need to last to collect enough data?

  Pilots typically run for a school year and involve around 30 teachers divided into two groups. More intensive interventions can be evaluated in a semester. In instances where the RCT is not feasible, we work with you to identify an appropriate comparison group for participating teachers. A research program may extend beyond one year, but we always view the program as modular, with each year comprising a study that can be reported independently. We always aim at providing information in a timely manner so that it is useful for your decision-making.

• Who pays for the research?

  Many federal and state grants now require that schools conduct a scientific evaluation of programs they wish to implement. We can assist schools in preparing their proposal so that the evaluation is built right into the funding. In another model, some school districts will want to see the results of a pilot study prior to a major curriculum or program purchase. A “pilot-to-purchase” often is considered, in which the publisher would sponsor all or a portion of the study. Visit our participation page for more information.

  In addition, through a U.S. Department of Education grant, Empirical Education has had funds for some districts to conduct their own studies with help from the company at no cost. We anticipate the availability of additional federal funding in 2012.

• Can you conduct an experiment on interventions other than curriculum materials?

  Yes. The basic research design we use applies to any grade level and content area for which there is an accepted assessment. As an example, we have conducted studies on a district-developed staff development program aimed at making technology use more effective and thereby contributing to student achievement.

• What data do you use to compare student achievement or program effectiveness?

  For most school districts, the most useful kind of evidence comes from improvements in achievement as measured by state-mandated tests. In many cases, these can be used for both pre- and posttest measures. Other assessments regularly conducted by the district, even formative assessments or progress monitoring measures, can often be used. We also collect class roster and demographic information for each student in the participating classrooms. All student data must include a student ID number.

• Do we need to pass the data to Empirical Education? If so, how is it accomplished and how secure is your data server?

  Yes, Empirical Education reformats the data set within our data warehouse so that we can analyze it in a standard way. We receive student data in many forms but prefer Excel files. If the files contain student names, the original files are stored separately and only the data with ID numbers are included in our warehouse. The warehouse server itself is behind a professional firewall and includes additional layers of password protection so that only our staff can access it.

• When you conduct a RCT, how are teachers or classrooms chosen for the study?

  Teachers volunteer for participation after a meeting in which they are fully informed of the research procedures, time requirements, any compensation, and the materials they may be using. They volunteer prior to knowing whether they will be in the group piloting the new intervention or in the control group. These assignments are made in that initial meeting and, once determined, cannot be changed.

• Who works with the administration and trains the teachers?

  Empirical Education manages all data gathering, including surveys of the teachers. The training and support of the intervention is done by the vendor or developer. In this way, we are testing the actual product as it would ordinarily be delivered and supported.

• How much time per week is required for a teacher to participate in the research activity?

  Teachers who are assigned to pilot an intervention will devote time to training and to preparing to use the new materials or methods. The district or other research sponsor may provide substitutes or after-hours pay for this work. The research itself is not time-consuming. After the initial meeting of about 90 minutes, all will be asked to spend about 30 minutes a month on Web-based surveys and interviews.

• Do we need to obtain parental permission?

  Parental permission is not required by the Family Educational Rights and Privacy Act (FERPA). You are providing the data to Empirical Education for the purpose of improving instruction and with the assurance that we will not release personally identifiable information. In addition, Empirical Education does not directly test or interact with individual students and so the intervention being tested is the same as any other new instructional program the district might be piloting.

• What kind of information is in the reports?

  Our reports are a full description of the research and contain all the information required for a scientific peer review of the work. This includes the size of the difference the intervention made and the level of statistical confidence the reader can have in the conclusions.

  Our analysis, however, seldom stops with the simple yes or no answer about effectiveness because our statistical techniques allow us to consider the kinds of classrooms or the kinds of students for which the intervention worked well. In addition, our observations, interviews, and surveys of the teachers will often provide rich, formative data to aid in improving the implementation of the program. We write our reports in language that is accessible to school district decision-makers.

• Who owns the data? Who owns the report?

  The school district owns the data set and provides it to Empirical Education under a standard agreement. To maintain strict independence, we retain the copyright to our report. The school district receives copies of the report for internal use. If a vendor is participating as a sponsor, we provide them with a license to reproduce the report and use all or parts of it. Empirical Education is responsible for submitting the report for scientific review, including to the What Works Clearinghouse.