The Teacher/Principal Evaluation Program

Rater Agreement

Background

The four framework authors would like to underscore the importance of the shared experience between evaluator and evaluatee. To that end, WAC 392-191A-050 supports this intent:

  1. To acknowledge the critical importance of teacher and leadership quality and support professional learning as the underpinning of the new evaluation system.
  2. To identify, in consultation with classroom teachers, principals, and assistant principals, particular areas in which the professional performance is distinguished, proficient, basic or unsatisfactory, and particular areas in which the classroom teacher, principal, or assistant principal needs to improve his or her performance.
  3. To assist classroom teachers and certificated principals and assistant principals, who have identified areas needing improvement, in making those improvements.

The law requires that “principals and administrators who have evaluation responsibilities, must engage in professional development designed to...maximize rater agreement.” RCW 28A.405.120

Overarching Role and Responsibilities: District Office, Principals, and Teachers

Rater Agreement Implementation Responsibilities

  • District Office
    • Ensures fidelity of implementation of the chosen framework
    • Leads the oversight of ongoing evaluator training and capacity-building across a district
    • Moves evaluators toward maximizing rater agreement
    • Provides the structures (e.g., principal PLC time) and resources for teachers and principals to learn, implement, and sustain growth- oriented evaluation
  • Principal
    • Leads the implementation of the growth-oriented teacher evaluation processes.
    • Takes a collaborative learning stance with teachers in the evaluation process.
    • Takes responsibility for learning and sustaining their own rater agreement.
    • Provides the structures and resources for building level staff to learn, implement, and sustain the growth-oriented evaluation system.
  • Teacher
    • Engages in the implementation of the growth-oriented evaluation processes.
    • Takes a collaborative learning stance with the principal and colleagues in the evaluation process.
    • Takes responsibility for learning and applying the frameworks and rubrics to continually improve practice.

Communication

  • District Office
    • Provides communication and sponsorship for immediate and long-term implementation.
  • Principal Office
    • Provides communication and feedback to support immediate and long-term implementation.
  • Teacher
    • Provides communication and feedback to support immediate and long-term implementation.

What is Rater Agreement?
Rater Agreement is the extent to which the scores between the raters have consistency and accuracy against predetermined standards. The predetermined standards are the instructional and leadership frameworks and rubrics that define the basis for summative criterion level scores.

Stage 1
ESDs, through the services of the Framework Specialists, will provide a two-day Stage 1 training for all who evaluate classroom teachers. This two-day overview provides an understanding of the “Big Ideas” of the Instructional or Leadership Frameworks and the inter-dependency of the frameworks, rubrics, and state criteria.

Evaluators should know and be able to:
Understand the “Big Ideas” of the Instructional or Leadership Frameworks and the inter-dependency of the frameworks, rubrics, and state criteria. Accuracy in scoring depends on this foundational level. This foundational level understanding is the key to future calibration and application. It is recommended that this training occur prior to commencing the evaluation cycle.

Success Indicators
Understanding of:

  • The appropriate use of the instructional language and framework vocabulary
  • The alignment of framework to state criteria
  • The interdependence of Dimensions/Indicators (CEL), Domains/Components (Danielson & Marzano)
  • The definition of essential aspects of the framework
  • The possible evidence for aspects of the framework

Districts should be creating plans to:
Provide the Stage 1 training to all evaluators before evaluating staff.

Stage 2
ESDs, through the services of the Framework Specialists, will provide 30 hours* of ongoing training on the frameworks for all who evaluate classroom teachers, principals, and assistant principals.

Evaluators should know and be able to:
Apply the framework and rubric in a formative process. This includes the capacity to give feedback, provide immediate support for implementing the feedback, and provide long-term professional development. Real life observation must be included in the application of this stage. It is recommended that this training occur during the year-long evaluation cycle.

Success Indicators:
Formative Application of:

  • The appropriate use of language of instruction and framework vocabulary
  • The interdependence of dimensions/domains and indicators/components
  • The key ideas in each criteria indicators/components and what evidence would look like/sound like
  • The essential aspects of the framework
  • The evidence for all aspects of the framework
  • Develop expertise to provide formative feedback through professional conversations/coaching on the framework and rubrics to foster teacher growth development and implementation of short-term professional development plans

Summative Application of:

  • Analysis of evidence over time for the purpose of summative scoring
  • Utilizing the criterion scoring document, guiding principles, and scoring matrix
  • Utilize summative scoring methodology from WAC 392-191A. Develop and sustain rater agreement, including summative scoring against pre-determined standard identified in the framework and rubric.

Districts should be creating plans to:
Provide Stage 2 training to all evaluators prior to summative scoring in the evaluation cycle. Provide for teachers to be supported in the formative evaluation process through framework training, self-assessment/goal setting, observation cycles, professional conversations, instructional coaching, billing and district professional development, and collaborative groups like professional learning communities or grade level/content teams.

Stage 3
Districts will continuously move evaluators to accuracy and consistency through ongoing calibration conversations involving real-life or video-based observation. Framework Specialists do not provide training for this stage although portions of Stage 2 training do have applicability toward rater agreement. Stage 3 should be integrated into the evaluation of principals (Criterion 5) and district administrators.

Evaluators should know and be able to:
' Accurately align and assess observation data to the standards in the rubric within the selected framework.

Success Indicators: Summative Application of:

  • Appropriately use language of instruction and framework vocabulary
  • The evidence for all aspects of the framework and ability to determine summative indicator or component level scores using that evidence.
  • The evidence for all aspects of the rubric and ability to determine summative criterion level scores for each state criteria.

Districts should be creating plans to:
Provide for the ongoing professional development for all evaluators to demonstrate rater assurance.

 

   Updated 7/26/2016

Old Capitol Building, PO Box 47200, 600 Washington St. S.E., Olympia, WA  98504-7200  360-725-6000  TTY 360-664-3631
Contact Us    |    A-Z Index    |    Site Info    |    Staff Only    |    Education Data System (EDS)