Visit the Observation Engine site at www.observationengine.com
Frequently Asked Questions about the Observation Engine
What is the Observation Engine?
The Observation Engine is a web-based tool that uses videos of classrooms to support systematic classroom observations as a part of teacher evaluation. It meets the needs of school systems that have selected an observation protocol or rubric and are ready to train, calibration and/or certify their teacher evaluators.
Is the Observation Engine a teacher evaluation training system?
While there are several ways in which the Engine can support teacher evaluation training, it is not itself a full service training system. Because the Engine supports the use of any teacher evaluation system and observation rubric, it allows school systems to flexibly employ its testing modules and videos of classroom instruction as a part of their own evaluator training. Empirical Education is happy to work with training providers to seamlessly incorporate the Engine into training programs.
How do most users generate master scores for tests? What if we do not have the capacity to supply our own master scores?
Since the system is protocol-agnostic, the Engine requires input of master scores (the “answer key”) for chosen videos during test set up. There are many scenarios in which it is most appropriate for school systems to do their own master scoring. For example, some local education agencies (LEAs) have designed their own protocols or have made significant modifications to existing protocols. In these cases, only staff from that particular LEA can function as expert raters. Alternatively, large education agencies may have established expert staff on hand that can master score videos using off-the-shelf protocols. However, not all sites have this capacity. There are a number of ways in which Empirical Education can assist customers in the master scoring process. We have partnerships with technical assistance providers that have the capacity to aid in the master scoring process. Alternatively, if you would like to use your own staff but need help calibrating your expert raters, our analytical software and expert staff we can help with that as well.
I understand the basics of what the Observation Engine does, but how exactly do school systems use it?
Because of The Observation Engine’s flexibility, it can be difficult to imagine all of its practical uses. Here are some example scenarios of how the observation can be used.
A school district implements an initial training for a group of teacher evaluators on a new observation protocol. The district can use the Observation Engine to run tests assessing the evaluators’ understanding of the protocol. The engine can help trainers pinpoint specific areas of difficulty as well as assess which observers may need further training.
After an initial pilot year, a state department of education is implementing a high-stakes certification process for principals using an established teacher evaluation system. The state can use the Observation Engine to administer a state-wide certification test to all teacher evaluators. They can use the system to notify evaluators that they have passed the certification and assign a retest to those that did not yet pass.
All principals in a school district have successfully passed a certification. The system can be used to conduct periodic calibration checks throughout the school year to ensure proficiency using the observation protocol.
We are moving towards high-stakes teacher evaluation, but we are still piloting our teacher evaluation system and orienting our evaluators to the process. Can we still use the Observation Engine?
Yes. In fact, there many advantages to using the Engine early in the process. Evaluators may find it stressful to jump right into a certification test online as their first experience with the Observation Engine. Allowing them to participate in practice testing throughout the first school year will help them get oriented to video observations and the online testing environment. By the time you are ready to administer a highstakes certification test, they will be familiar with the process. In addition, the data that results from practice testing can help you get a better general sense of how your observation protocol is understood and applied by your evaluators. You can then use this information to modify trainings or even tweak your protocol.
We are very excited about the access to high-quality classroom videos and can see several uses for them in our school system. Are there limits around access to these videos?
A principal goal of the Observation Engine is to provide classroom videos to school systems to improve teacher evaluation. However, there are some limits around how the videos contained in the Engine can be used. Due to permission issues, the system is designed to only allow licensed users to access the videos. The videos themselves are organized into a set of discrete test models designed to show a range of instruction quality. Depending on the level of service you decide to purchase, we may be able to put custom test models together for you that will fit your specific purposes. For more details about how the videos can be used, you can submit your questions here.
Is there a description of Observation Engine services?
The Observation Engine is offered to school systems on an annual subscription basis. In addition to online tool access, the following services are included as part of the basic service level.
Technical Support and Maintenance
Basic hosting: Access to the system is web-based and browser driven, requiring no remote clients.
Bug fixes: All routine bug fixes are provided through system updates.
Upgrades: System upgrades such as added functionality and feature enhancements are provided.
Telephone support: Basic telephone technical support for Authors, Managers, and Observers is available Monday through Friday from 7:00am - 5:00pm Pacific Time.
Email support: Support for Authors, Managers, and Observers is also available through email. A response is guaranteed by mid-morning the next business day.
Product training: Online user manuals, video tutorials, and FAQs are available for all users to assist with system setup and general use. Manager and Author level product trainings are also available via live webinar.
Specialized assistance: Additional product assistance is provided by Teacher Evaluation Specialists at Empirical Education. Teacher Evaluation Specialist assistance includes the following.
Personalized customer setup (including assistance with selecting appropriate test models)
Assistance with implementation timelines and templates
Custom and refined test models
Custom data preparation and aggregation
Administration of observer feedback surveys
Technical assistance summaries
Continued R&D: Constant needs analysis is conducted through regular communication with Teacher Evaluation Specialists. This informs the research and development efforts leading to future product functionality and feature enhancements that are offered through system upgrades.
Shared “best practices”: Methods such as surveys, community online forums, and bulletin postings are used to share user experiences across school site, and help cultivate community support.
“Office hours”: Popular topics and areas of interest across user sites are highlighted for community discussion through periodic informational webinars.
Additional custom services such as in-depth protocol analyses, in-person trainings, custom integration, and other types of consulting are also available for additional fees.