What Makes a Good Accountability System?

As we enter the new era of Every Student Succeeds Act (ESSA) states have greater flexibility in designing new accountability systems.  This blog entry considers the characteristics of a quality accountability system.  We should keep these characteristics in mind as we develop new systems and ensure that our state education departments are disciplined in their development of new, more useful accountability systems. 

In an article in Educational Researcher (2013) Morgan Polikoff (and others) stated there are two broad theories that support accountability in education: (1) that incentives in the accountability system will direct schools towards behaviors that will improve student outcomes (principal agent theory) and (2) accountability information helps consumers (parents and children) make better choices (experiential goods).  Polikoff and colleagues argued that any accountability system that is going to achieve these goals needs to meet four criteria: (1) construct validity, (2) reliability, (3) fairness, and (4) transparency. 

Construct validity means that performance measures used (mostly test scores in the NCLB age) adequately cover the desired student outcomes and the inferences made on the basis of those measures are appropriate.  Of course, under NCLB there were numerous problems with the construct validity because proficiency levels (the primary measure used) did not estimate the contribution of the school and teacher and growth was difficult to measure (even though considered important).  In our new accountability systems we need to stay disciplined in our selection of measures to ensure that they align with the our desire student outcomes and we can make appropriate inferences (e.g. the school contributed to the outcome).  Our desired outcomes may be beyond reading and math (as it was under NCLB) and include other skills or competencies.  The accountability system should be designed to have construct validity relative to those desire outcomes. 

Reliability refers to the consistency of classification.  Accountability systems should be reasonably reliable in how they classify schools.

Fairness refers to whether the classifications are primarily due to factors beyond the control of the school.  In short, this means that system should not unfairly identify schools based on demographics.  The accountability system for New Hampshire's waiver unfairly penalizes schools and districts that are large or have high proportions of special education students.  What ends up happening is that some schools have special education counted multiple times.  In fact, in the calculation of Focus Schools (those with the largest gaps) most had special education counted twice.  On the other hand, most of the schools with the lowest gaps (deemed high performing) did not have special education counted at all.   

Lastly, transparency is whether the goal setting is clear and the performance measures are easily understood.  States must make clear how performance measures are being used to classify schools and make the data available to the public for inspection. 

As states begin developing new accountability systems, we need to make sure these systems meet these minimum characteristics.  Stakeholders, including parents, educators, and the public, need to hold state departments of education accountable for developing new systems that meet these characteristics and result in improved student outcomes and better information for consumers.