It May Be Ready, But is it Valid?

In today’s edition of Commissioner Candice McQueen’s Educator Update, she talks about pending legislation addressing teacher evaluation and TNReady.

Here’s what McQueen has to say about the issue:

As we continue to support students and educators in the transition to TNReady, the department has proposed legislation (HB 309) that lessens the impact of state test results on students’ grades and teachers’ evaluations this year.

In 2015, the Tennessee Teaching Evaluation Enhancement Act created a phase-in of TNReady in evaluation to acknowledge the state’s move to a new assessment that is fully aligned to Tennessee state standards with new types of test questions. Under the current law, TNReady data would be weighted at 20 percent for the 2016-17 year.

However, in the spirit of the original bill, the department’s new legislation resets the phase-in of growth scores from TNReady assessments as was originally proposed in the Tennessee Teaching Evaluation Enhancement Act. Additionally, moving forward, the most recent year’s growth score will be used for a teacher’s entire growth component if such use results in a higher evaluation score for the teacher.

We will update you as this bill moves through the legislative process, and if signed into law, we will share detailed guidance that includes the specific options available for educators this year. As we announced last year, if a teacher’s 2015-16 individual growth data ever negatively impacts his or her overall evaluation, it will be excluded. Additionally, as noted above, teachers will be able to use 2016-17 growth data as 35 percent of their evaluation if it results in a higher overall level of effectiveness.

And here’s a handy graphic that describes the change:

TNReady Graphic

 

 

Of course, there’s a problem with all of this: There’s not going to be valid data to use for TVAAS. Not this year. It’s bad enough that the state is transitioning from one type of test to another. That alone would call into question the validity of any comparison used to generate a value-added score. Now, there’s a gap in the data. As you might recall, there wasn’t a complete TNReady test last year. So, to generate a TVAAS score, the state will have to compare 2014-15 data from the old TCAP tests to 2016-17 data from what we hope is a sound administration of TNReady.

We really need at least three years of data from the new test to make anything approaching a valid comparison. Or, we should start over building a data-set with this year as the baseline. Better yet, we could go the way of Hawaii and Oklahoma and just scrap the use of value-added scores altogether.

Even in the best of scenarios — a smooth transition from TCAP to TNReady — data validity was going to be challenge.

As I noted when the issue of testing transition first came up:

Here’s what Lockwood and McCaffrey (2007) had to say in the Journal of Educational Measurement:

We find that the variation in estimated effects resulting from the different mathematics achievement measures is large relative to variation resulting from choices about model specification, and that the variation within teachers across achievement measures is larger than the variation across teachers. These results suggest that conclusions about individual teachers’ performance based on value-added models can be sensitive to the ways in which student achievement is measured.
These findings align with similar findings by Martineau (2006) and Schmidt et al (2005)
You get different results depending on the type of question you’re measuring.

The researchers tested various VAM models (including the type used in TVAAS) and found that teacher effect estimates changed significantly based on both what was being measured AND how it was measured.

And they concluded:

Our results provide a clear example that caution is needed when interpreting estimated teacher effects because there is the potential for teacher performance to depend on the skills that are measured by the achievement tests.

If you measure different skills, you get different results. That decreases (or eliminates) the reliability of those results. TNReady is measuring different skills in a different format than TCAP. It’s BOTH a different type of test AND a test on different standards. Any value-added comparison between the two tests is statistically suspect, at best. In the first year, such a comparison is invalid and unreliable.

So, we’re transitioning from TCAP to TNReady AND we have a gap in years of data. That’s especially problematic — but, not problematic enough to keep the Department of Education from plowing ahead (and patting themselves on the back) with a scheme that validates a result sure to be invalid.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Knox County Takes a Stand

Last night, the Knox County School Board voted 6-3 in favor of a resolution calling on the General Assembly and State Board of Education to waive the use of TCAP/TNReady data in student grades and teacher evaluations this year.

The move comes as the state prepares to administer the tests this year with a new vendor following last year’s TNReady disaster. The lack of a complete testing cycle last year plus the addition of a new vendor means this year is the first year of the new test.

The Board passed the resolution in spite of Governor Haslam warning against taking such a step.

In his warning, Haslam said:

“The results we’ve seen are not by accident in Tennessee, and I think you have to be really careful about doing anything that could cause that to back up,” Haslam said.

He added:

Haslam attributed that progress to three things, including tying standardized tests to teacher evaluations.

“It’s about raising our standards and expectations, it’s about having year-end assessments that match those standards and then I think it’s about having assessments that are part of teachers’ evaluations,” Haslam said. “I think that you have to have all of those for a recipe for success.”

Haslam can present no evidence for his claim about the use of student assessment in teacher evaluation. In fact, it’s worth noting that prior to 2008, Tennessee students achieved at a high level according to what were then the state standards. While the standards themselves were determined to need improvement, the point is teachers were helping students hit the designated mark.

Teachers were moving students forward at this time without evaluations tied to student test results. Policymakers set a mark for student performance, teachers worked to hit that mark and succeeded. Standards were raised in 2008, and since then, Tennessee has seen detectable growth in overall results, including some exciting news when NAEP results are released.

To suggest that a year without the use of TVAAS scores in teacher evaluations will cause a setback is to insult Tennessee’s teachers. As if they’ll just relax and not teach as hard.

Another argument raised against the resolution is that it will somehow absolve teachers and students of accountability.

Joe Sullivan reports in the Knoxville Mercury:

In an email to board members, [Interim Director of Schools Buzz] Thomas asserted that, “We need a good standardized test each year to tell us how we are doing compared to others across the state and the nation. We will achieve greatness not by shying away from this accountability but by embracing it.” And he fretted that, “This resolution puts that at risk. In short, it will divide us. Once again we could find ourselves in two disputing camps. The pro-achievement folks on the one side and the pro-teacher folks on the other.”

Right now, we don’t know if we have a good standardized test. Taking a year to get it right is important, especially in light of the frustrations of last year’s TNReady experience.

Of course, there’s no need for pro-achievement and pro-teacher folks to be divided into two camps, either. Tennessee can have a good, solid test that is an accurate measure of student achievement and also treat teachers fairly in the evaluation process.

To be clear, teachers aren’t asking for a waiver from all evaluation. They are asking for a fair, transparent evaluation system. TVAAS has long been criticized as neither. Even under the best of circumstances, TVAAS provides a minimal level of useful information about teacher performance.

Now, we’re shifting to a new test. That shift alone makes it impossible to achieve a valid value-added score. In fact, researchers in the Journal of Educational Measurement have said:

We find that the variation in estimated effects resulting from the different mathematics achievement measures is large relative to variation resulting from choices about model specification, and that the variation within teachers across achievement measures is larger than the variation across teachers. These results suggest that conclusions about individual teachers’ performance based on value-added models can be sensitive to the ways in which student achievement is measured.
These findings align with similar findings by Martineau (2006) and Schmidt et al (2005)
You get different results depending on the type of question you’re measuring.

The researchers tested various VAM models (including the type used in TVAAS) and found that teacher effect estimates changed significantly based on both what was being measured AND how it was measured.

Changing to a new type of test creates value-added uncertainty. That means results attributed to teachers based on a comparison of this year’s tests and the old tests will not yield valid results.

While insisting that districts use TVAAS in teacher evaluations this year, the state is also admitting it’s not quite sure how that will work.

From Sullivan’s story:

When asked how these determinations will be made, a spokesperson for the state Department of Education acknowledges that a different methodology will have to be employed and says that, “we are still working with various statisticians and experts to determine the exact methodology we will use this year.”

Why not at take at least a year, be sure there’s a test that works, and then build a model based on that? What harm would come from giving teachers and students a year with a test that’s just a test? Moreover, the best education researchers have already warned that testing transitions create value-added bumps. Why not avoid the bumps and work to create an evaluation system that is fair and transparent?

Knox County has taken a stand. We’ll soon see if others follow suit. And if the state is listening.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

 

Ready for a Fight

Yesterday, Williamson County Director of Schools Mike Looney issued a statement saying his district would not be administering the high school end of course tests in addition to the suspension of the grades 3-8 TNReady tests.

Commissioner McQueen is not very happy about that. She served notice to Looney and all other directors that refusing to administer the EOC would be considered a violation of state law.

Here’s the email she sent to Directors of Schools:

First, I want to thank you for your partnership and support as we have worked together to implement and administer the first year of a new assessment. I know you share my disappointment and frustration with the inability of our vendor to deliver on this higher quality assessment in grades 3-8, and I truly appreciate your patience and leadership.

 

I want to reiterate that the state’s termination of its contract with the testing vendor Measurement Incorporated (MI) and the related suspension of grades 3-8 testing does not apply to high school and End of Course (EOC) exams, and, therefore, all school districts are required to administer these assessments.

 

The state of Tennessee and local districts are under an obligation under both federal and state law, as well as state board of education rules and regulations, to administer annual assessments to our students. My decision to suspend grade 3-8 testing was based on the impossibility of testing and made in close consultation with the U.S. Department of Education (USDOE). Based on the fact that testing in grades 3-8 was not feasible due to the failure of MI to meet its contractual obligations, the USDOE has acknowledged that the department made a good faith effort to administer the assessments to all students in grades 3-8. Unlike grades 3-8, districts are in receipt of EOC exams and the challenges associated with the delivery of grades 3-8 do not exist.

 

Because EOC exams have been delivered, students should have the opportunity to show what they know to measure their progress toward postsecondary and the workforce. Failure to administer the high school assessments will adversely impact students who will not only lose the experience of an improved, high quality test aligned to our higher standards but also the information we plan to provide to students, parents and educators relative to student performance. In addition, districts will eliminate the option for their teachers to use this year’s student achievement data as part of their teacher evaluation if the data results in a higher score.

 

Because of these factors and because state or district action to cancel high school testing would willfully violate the laws that have been set forth relative to state assessment, neither the state nor districts have the authority to cancel EOC exams. Districts that have taken action to cancel EOC exams or communicated such action are in violation of the law and should rescind this action or communication.

What Does This Mean?

In response to the Murfreesboro City School Board considering refusing to administer Phase II of TNReady, the Department of Education issued a statement noting that doing so would be considered a major violation of state law and that withholding state funds was a possible penalty.

McQueen doesn’t say what the penalty would be if districts like Williamson proceed with their refusal to administer the EOCs, but she may well attempt to impose a financial penalty.

In her email, McQueen says:

Failure to administer the high school assessments will adversely impact students who will not only lose the experience of an improved, high quality test aligned to our higher standards but also the information we plan to provide to students, parents and educators relative to student performance.

Just what students want and need: Another test. Some have proposed using the ACT battery of tests as the high school testing measure rather than the current EOC structure.

McQueen also says:

In addition, districts will eliminate the option for their teachers to use this year’s student achievement data as part of their teacher evaluation if the data results in a higher score. 

While the idea of flexibility seems nice, I want to reiterate that any data gleaned from this year’s test is invalid as a value-added indicator of teacher performance. As such, there’s no useful information to be gained relative to teacher performance from this year’s EOCs. Put another way, McQueen’s argument about depriving teachers of an opportunity is invalid.

While the use of value-added data to assess teacher performance is of limited usefulness under optimum conditions, under this year’s transition, it is clearly and plainly invalid. If the goal of using such data is to improve teacher performance, why use data that yields essentially no information?

I have not yet seen a response from Dr. Looney or any other directors. But a fight could be brewing.

For more on education politics and policy in Tennessee, follow @TNEdReport

 

Ready to Refuse

As Tennessee schools prepare for Phase II of TNReady, the Department of Education has sent districts a memo outlining how they should handle students who refuse or attempt to “opt-out” of the test.

The general gist, according to reporting by Grace Tatter, is that you can’t opt-out or refuse. She reports:

District leaders received a memo last week instructing schools to “address student absences on testing days in the same manner as they would address a student’s failure to participate in any other mandatory activity at school (e.g. final exams) by applying the district’s or school’s attendance policies.”

The memo specifically notes:

 “State and federal law also requires student participation in state assessments. In fact, these statutes specifically reference the expectation that all students enrolled in public schools in Tennessee will complete annual assessments.”

That’s not entirely true.

Federal law, even with the newly passed Every Student Succeeds Act (ESSA), requires states to administer annual assessments in grades 3-8 and at least once in high school.

But there’s a difference in requiring a state to administer and requiring a student to complete an assessment. Federal law requires administration of the test, but does not compel students to complete the exams.

Then, there is state law. The memo lacks specific references to Tennessee statute, but there are a few sections that relate to testing.

TCA 49-1-6 includes references to performance assessment and the Tennessee Value-Added Assessment System (TVAAS). This portion of state law says that annual assessments will be administered in grades 3-8 and then outlines the secondary school testing schedule. Here again, the law notes tests will be administered, but contains no compulsory language for students.

Then there’s TCA 49-6-60 dealing with proficiency testing. This section specifically details testing to be administered in grades 8, 10, and 11 as a strategy to promote college readiness. As these three tests are required for graduation, they are essentially mandated. Students who don’t take them won’t complete the graduation requirements.

What’s missing? Language that compels a student to take the test or requires a district to compel students to take the test. The memo says that “state and federal” statutes specifically reference the expectation that students will complete the assessment. True, TVAAS and other accountability measures are made valid by significant student participation in state tests. But, that alone doesn’t make them compulsory. Unless it’s one of the three proficiency tests specifically referenced in the graduation requirements section, there’s no language directly compelling students to participate in annual assessments.

It’s worth noting that while the Department of Education has said there would be penalties if districts refused to administer the TNReady tests, the memo says districts are not authorized to allow “opting-out” or test refusal. What it doesn’t say is what impact allowing opt-out would have on the district. If a district offers the test, and students refuse, then what?

Stay tuned as Phase II starts later this month.

For more on education politics and policy in Tennessee, follow @TNEdReport

 

 

Ready to Waive

Governor Bill Haslam and Commissioner of Education Candice McQueen announced today that in light of difficulties with the administration of the TNReady test, they are proposing that TNReady data NOT be included in this year’s round of teacher evaluations.

The statement comes after the Knox County Board of Education made a similar request by way of resolution in December. That resolution was followed by a statewide call for a waiver by a coalition of education advocacy groups. More recently, principals in Hamilton County weighed in on the issue.

Here’s Governor Haslam’s press release on the waiver:
Tennessee Gov. Bill Haslam today announced he would seek additional flexibility for teachers as the state continues its transition to the TNReady student assessment.

Under the proposal, teachers would have the choice to include or not to include student results from the 2015-2016 TNReady assessment in his or her evaluation score, which typically consists of multiple years of data. The proposal keeps student learning and accountability as factors in an educator’s evaluation while giving teachers the option to include this year’s results if the results benefit them. The governor will work with the General Assembly on specific language and a plan to move the proposal through the legislative process.

“Tennessee students are showing historic progress. The state made adjustments to teacher evaluation and accountability last year to account for the transition to an improved assessment fully aligned with Tennessee standards, which we know has involved a tremendous amount of work on the part of our educators,” Haslam said. “Given recent, unexpected changes in the administration of the new assessment, we want to provide teachers with additional flexibility for this first year’s data.”

Tennessee has led the nation with a teacher evaluation model that has played a vital role in the state’s unprecedented progress in education. Tennessee students are the fastest improving students in the country since 2011. The state’s graduation rate has increased three years in a row, standing at 88 percent. Since 2011, 131,000 more students are on grade-level in math and nearly 60,000 more on grade-level in science.  The plan builds upon the Teaching Evaluation Enhancement Act proposed by the governor and approved by the General Assembly last year. This year is the first administration of TNReady, which is fully aligned with the state’s college and career readiness benchmarks.

“Providing teachers with the flexibility to exclude first-year TNReady data from their growth score over the course of this transition will both directly address many concerns we have heard and strengthen our partnership with educators while we move forward with a new assessment,” Department of Education Commissioner Candice McQueen said. “Regardless of the test medium, TNReady will measure skills that the real world will require of our students.”

Most educator evaluations have three main components: qualitative data, which includes principal observations and always counts for at least half of an educator’s evaluation; a student achievement measure that the educator chooses; and a student growth score, which usually comprises 35 percent of the overall evaluation

 

While the release mentions last year’s changes to teacher evaluation to account for TNReady, it fails to note the validity problems created by an evaluation system moving from a multiple choice (TCAP) to a constructed-response test (TNReady).

Here’s the Tennessee Education Association on the announcement:

“TEA applauds Gov. Haslam on his proposal to give teachers the flexibility to not use TNReady test data in their 2015-16 evaluations. It is encouraging to see the governor listen to the widespread calls from educators, parents and local school boards for a one-year moratorium for TNReady data in teacher evaluations.”

 

“It is important that schools are given the same leniency as students and teachers during the transition to TNReady. These test scores that Gov. Haslam is acknowledging are too unreliable for use in teacher evaluations, are the same scores that can place a school on the priority list and make it eligible for state takeover. All high-stakes decisions tied to TNReady test data need to be waived for the 2015-16 school year.”

 

“While the governor’s proposal is a step in the right direction toward decoupling standardized test scores with high-stakes decisions, these measurements have proven to be unreliable statistical estimates that are inappropriate for use in teacher evaluations at all. TEA will continue its push to eliminate all standardized test scores from annual teacher evaluations.”

For more on education politics and policy in Tennessee, follow @TNEdReport

As Flexible as a Brick Wall

Grace Tatter reports that officials at the Tennessee Department of Education are “perplexed” by concerns over using TNReady data in this year’s teacher evaluations.

While a number of districts have passed resolutions asking for a waiver from including TVAAS scores in this year’s teacher evaluations due to the transition to TNReady, a department spokesperson said:

“Districts have complete discretion to choose how they want to factor that data,” Ball said Thursday. “They don’t have to use TNReady or growth data in hiring, firing, retention or promotion.”

As Tatter’s story notes, however, data from TNReady will still be a part of a teacher’s TVAAS score — 10%. And that score becomes a part of a teacher’s overall evaluation score — a ranking from 1-5 that purports to measure a teacher’s relative effectiveness.

10% is enough to move a ranking up or down a number, and that can have significant impacts on a teacher’s career, even if they are not fired and their pay is not impacted. Of course, some districts may use this year’s data for those purposes, since it is not prohibited under the evaluation changes passed last year.

Dan Lawson outlines some of the of impact faced by teachers based on that final number:

The statutorily revised “new tenure” requires five years of service (probationary period) as well as an overall score of “4” or “5” for two consecutive years preceding the recommendation to the Board of Education. Last year, no social studies assessment score was provided since it was a field tested and the teacher was compelled to select a school wide measure of growth.  He chose POORLY and his observation score of a “4.38” paired with a school wide growth score in the selected area of a “2” producing a sum teacher score of “3” thereby making him ineligible for tenure nomination.

According to TCA 49-5-503, a teacher may not be awarded tenure unless she achieves a TEAM score of 4 or 5 in two consecutive years immediately prior to being tenure eligible. That means a TVAAS score that takes a teacher from a 4 to a 3 would render her ineligible.

Further, a tenured teacher who receives a TEAM score of a 1 or 2 in two consecutive years is returned to probationary status (TCA 49-5-504). So, that tenured teacher who was a 2 last year could be impacted by a TNReady-based TVAAS score that moves a TEAM score of a 3 down to a 2.

Districts don’t have “complete discretion” to waive state law as TNDOE spokesperson Ashley Ball seems to imply.

Further, basing any part of a teacher’s evaluation on TVAAS scores based on TNReady creates problems with validity. Why include a number in a teacher’s evaluation that is fundamentally invalid?

Teachers want an evaluation process that is fair and transparent. There’s nothing perplexing about that.

For more on education politics and policy in Tennessee, follow @TNEdReport

Still Not Ready

The MNPS Board of Education last night passed a resolution calling on the State of Tennessee to delay the use of TVAAS scores in teacher evaluations during the first year of the new TNReady test. The resolution is similar to one passed in Knox County last month.

Here’s the MNPS version:

A RESOLUTION OF THE METROPOLITAN NASHVILLE PUBLIC SCHOOLS BOARD OF EDUCATION IN OPPOSITION TO THE USE OF TNREADY DATA FOR TEACHER EVALUATIONS FOR THE SCHOOL YEAR 2015-2016

PROPOSED BY ANNA SHEPHERD

WHEREAS, Metropolitan Nashville Public Schools (MNPS) is responsible for providing a local system of public education, and
WHEREAS, The State of Tennessee through the work of the Tennessee General Assembly, the Tennessee Department of Education, the Tennessee Board of Education, and local boards of education, has established nationally recognized standards and measures for accountability in public education, and
WHEREAS, all public school systems in Tennessee have been granted a one-time pass in the 2015-2016 school year to not integrate TNReady scores into each student’s final grades due to an anticipated delay in assessment results, and
WHEREAS, teachers with at least five years of experience are eligible for tenure only if they receive an overall evaluation score above expectations or significantly above expectations for the prior two years, and
WHEREAS, this school year is the first year that the TNReady assessment will be administered, and
WHEREAS, the TNReady assessment is not a compatible assessment with the TCAP assessment, and
WHEREAS, the TNReady assessment requires the extensive use of technology and the State of Tennessee BEP funding formula, already inadequate, does not meet these technology needs or the needs of MNPS schools as a whole, and
WHEREAS, the Tennessee General Assembly and Tennessee Board of Education have already adopted the “Tennessee Teaching Evaluation Act” to lessen the evaluation score impact of TNReady in English/language arts and math, and
WHEREAS, over 70% of MNPS teachers, counselors, librarians, instructional coaches, and others do not produce individual TVAAS data, and
WHEREAS, MNPS seeks to recruit and retain excellent teachers to serve our students.
NOW, THEREFORE, BE IT RESOLVED BY METROPOLITAN NASHVILLE PUBLIC SCHOOLS BOARD OF EDUCATION AS FOLLOWS:
MNPS Board of Education strongly urges the Tennessee General Assembly and the Tennessee Board of Education to provide a waiver from utilizing the TNReady data for the use of teacher evaluations for the school year 2015-2016 or allow districts to only use observation data from evaluations to make decisions on hiring, placement, and compensation based strictly on the 2015-2016 TNReady data, and
BE IT FURTHER RESOLVED, that the Tennessee General Assembly and the Tennessee Board of Education consider the impact of the 2015-2016 TNReady data upon future years of teacher evaluations, and
BE IT FURTHER RESOLVED, that the Tennessee General Assembly and the Tennessee Board of Education consider allowing teachers to be eligible for tenure when they have received a composite score of four (4) or five (5) for two of any of the last five years, as opposed to the prior two years only.
ADOPTED BY THE MNPS BOARD OF EDUCATION AT ITS MEETING ON TUESDAY, JANUARY 12, 2016.

 

The resolution includes a few interesting notes:

  • 70% of MNPS teachers don’t have individual TVAAS data
  • There’s mention of the inadequacy of the BEP formula
  • There’s a call for further review of TVAAS after this year

According to prepared remarks by MNPS teacher Amanda Kail prior to the vote, four other counties have passed similar resolutions.

For more on education politics and policy in Tennessee, follow @TNEdReport

CAPE Flies into 2016

At the first MNPS Board meeting of 2016, advocacy group CAPE will again be encouraging teachers to raise their voices and speak out. CAPE member Amanda Kail previews the remarks she plans to make this evening:

Ladies and gentlemen of the school board — My name is Amanda Kail. I am an EL teacher at Margaret Allen Middle School.
First and foremost, I would like to wish all of you a happy new year. And in that vein, I would like all of us as a district to take a moment to reflect on what we have gotten right, and how we can improve in 2016.
First of all, you are to be commended in recognizing that over-testing has become a serious problem for our schools. Countless studies from leading experts in education, as well as the groundswell of parents around the country who are opting their children out of the tests, and even demands from students, such as the White Station High School students organizing in Shelby County point to the same conclusion — high-stakes testing has been a colossal mistake, regardless of the intentions. Many of you have made statements recognizing the need to reign in the testing as a priority. Thank you. Now let’s make 2016 the year that happens.
How can we do that? First, let’s end testing where we can. DISTRICT benchmarks take up SIGNIFICANT instructional time, and are often given so close to other tests as to be redundant. Getting rid of them would mean 3 less weeks of testing (and 3 weeks more of instruction).
Second, make instructional time THE FOCUS of school days again so teachers can teach and students can learn. Cap building-level testing to no more than once per semester. Remember that assessments are now given on-line, and that most schools at MNPS do not have enough computers to give these assessments in one day, meaning that a single whole-school assessment can drag on for one or two weeks in order to accommodate all students and grade levels.
Third, join Knox County, Blount County, Washington County and Anderson County schools by supporting Board Member Shepherd’s proposal to postpone using TN Ready scores on teacher evaluations this year. Tell Nashville teachers you respect our profession enough to not evaluate us on something that is so much beyond our control. Then tell the Tennessee legislature that it is time to reexamine the trust we have placed in high-stakes testing to tell us anything besides which schools are rich and which are poor.
Finally, lets find a director of schools who truly has ALL of our schools at heart. MNPS needs someone who will ask our legislature to end high-stakes testing and who will demand full funding for our district. Someone who will spend their time getting struggling schools more resources, like the wrap-around services from the Community Achieves program, and who will implement a fair and fully-supported discipline policy grounded in restorative justice. Someone who recognizes that threatening and punishing schools that are serving students with the highest needs is not nearly as useful as finding those schools the resources they need.
We have much work to do, but if we work together, this can be the year our system truly shines. Thank you.
For more on education politics and policy in Tennessee, follow @TNEdReport

A Matter of Fairness

A coalition of education advocacy groups released an online petition today calling for a one year waiver from using student test scores in teacher evaluations in Tennessee.

Here’s the press release:

A coalition of groups supporting public education today launched an online petition asking the Tennessee General Assembly and Governor Bill Haslam to grant teachers a grace period from the use of student test scores in their evaluations in the first year of new TNReady tests. The petition tracks language adopted unanimously by the Knox County School Board, which passed a resolution last week opposing the use of student test scores in teacher evaluation for this academic year.

“The state has granted waivers so that TNReady scores aren’t required to be counted in student grades for this year,” said Lyn Hoyt, president of Tennesseans Reclaiming Educational Excellence (TREE). “If TNReady won’t count in student grades, it’s only fair that it shouldn’t count for teacher evaluation.” Hoyt noted that the transition to the new test means entering uncharted territory in terms of student scores and impact on teacher evaluation scores. As such, she said, there should be a one year or more grace period to allow for adjustment to the new testing regime.

“TNReady is different than the standardized tests we’ve had in the past,” Hoyt said. “Our students and teachers both deserve a reasonable transition period. We support the Knox County resolution and we are calling on the General Assembly to take notice and take action. Taking a thoughtful path transitioning to the new test can also build confidence and trust in the process.”

Hoyt also cited a recent policy statement by the American Educational Research Association that cautions against using value-added data in teacher evaluations and for high-stakes purposes. “Researchers who study value-added data are urging states to be cautious in how it is used to evaluate teachers,” Hoyt said. “The transition to TNReady is the perfect time to take a closer look at how test scores are used in teacher evaluations. Let’s take a year off, and give our students and teachers time to adjust. It’s a matter of fundamental fairness.”

Groups supporting the petition include:

Strong Schools (Sumner County)
Williamson Strong (Williamson County)
SPEAK (Students, Parents, Educators Across Knox County)
SOCM (Statewide Organizing for Community eMpowerment)

Middle TN CAPE (Coalition Advocating for Public Education)
Momma Bears Blog
Advocates for Change in Education (Hamilton County)
Concerned Parents of Franklin County (Franklin County)
Parents of Wilson County, TN, Schools
Friends of Oak Ridge Schools (City of Oak Ridge Schools)
TNBATs (State branch of National BATs)
TREE (Tennesseans Reclaiming Educational Excellence)
TEA (Tennessee Education Association)

For more on education politics and policy in Tennessee, follow @TNEdReport

It All Comes Down to a Number

Dan Lawson is the Director of Schools for Tullahoma City Schools. He sent this message and the American Educational Research Association press release to a group of Tennessee lawmakers.

I am the superintendent of Tullahoma City Schools and in light of the media coverage associated with Representative Holt and a dialogue with teachers in west Tennessee I wanted to share a few thoughts with each of who represent teachers in other districts in Tennessee. I am thankful that each of you have a commitment to service and work to cultivate a great relationship with teachers and communities that you represent.

While it is certainly troubling that the standards taught are disconcerting in that developmental appropriateness is in question by many, and that the actual test administration may be a considerable challenge due to hardware, software and capacity concerns, I think one of the major issues has been overlooked and is one that could easily address many concerns and restore a sense of confidence in many of our teachers.

Earlier this week the American Educational Research Association released a statement (see below) cautioning states “against the use of VAM for high-stakes decisions regarding educators.” It seems to me that no matter what counsel I provide, what resources I bring to assist and how much I share our corporate school district priorities, we boil our work and worth as a teacher down to a number. And for many that number is a product of how well they guess on what a school-wide number could be since they don’t have a tested area.

Our teachers are tasked with a tremendous responsibility and our principals who provide direct supervision assign teachers to areas where they are most needed. The excessive reliance on production of a “teacher number” produces stress, a lack of confidence and a drive to first protect oneself rather than best educate the child. As an example, one of my principals joined me in meeting with an exceptional middle school math teacher, Trent Stout. Trent expressed great concerns about the order in which the standards were presented (grade level) and advised that our math department was confident that a different order would better serve our students developmentally and better prepare them for higher level math courses offered in our community. He went on to opine that while he thought we (and he) would take a “hit” on our eighth grade assessment it would serve our students better to adopt the proposed timeline. I agreed. It is important to note that I was able to dialogue with this professional out of a sense of joint respect and trust and with knowledge that his status with our district was solely controlled by local decision makers. He is a recipient of “old tenure.” However, don’t mishear me, I am not requesting the restoration of “old tenure,” simply a modification of the newly enacted statute. I propose that a great deal of confidence in “listening and valuing” teachers could be restored by amending the tenure statute to allow local control rather than state eligibility.

I have teachers in my employ with no test data who guess well and are eligible for the tenure status, while I have others who guess poorly and are not eligible. Certainly, the final decision to award tenure is a local one, but local based on state produced data that may be flawed or based on teachers other than the potential nominee. Furthermore, if we opine that tenure does indeed have value, I am absolutely lost when I attempt to explain to new teachers that if they are not eligible for tenure I may employ them for an unlimited number of added contracts but if they are eligible based on their number and our BOE decides that they will not award tenure to anyone I am compelled to non-renew those who may be highly effective teachers. The thought that statue allows me to reemploy a level 1 teacher while compelling me to non-renew a level 5 teacher seems more than a bit ironic and ridiculous.

I greatly appreciate your service to our state and our future and would love to see an extensive dialogue associated to the adoption of Common Sense.

The American Educational Research Association Statement on Value-Added Modeling:

In a statement released today, the American Educational Research Association (AERA) advises those using or considering use of value-added models (VAM) about the scientific and technical limitations of these measures for evaluating educators and programs that prepare teachers. The statement, approved by AERA Council, cautions against the use of VAM for high-stakes decisions regarding educators.

In recent years, many states and districts have attempted to use VAM to determine the contributions of educators, or the programs in which they were trained, to student learning outcomes, as captured by standardized student tests. The AERA statement speaks to the formidable statistical and methodological issues involved in isolating either the effects of educators or teacher preparation programs from a complex set of factors that shape student performance.

“This statement draws on the leading testing, statistical, and methodological expertise in the field of education research and related sciences, and on the highest standards that guide education research and its applications in policy and practice,” said AERA Executive Director Felice J. Levine.

The statement addresses the challenges facing the validity of inferences from VAM, as well as specifies eight technical requirements that must be met for the use of VAM to be accurate, reliable, and valid. It cautions that these requirements cannot be met in most evaluative contexts.

The statement notes that, while VAM may be superior to some other models of measuring teacher impacts on student learning outcomes, “it does not mean that they are ready for use in educator or program evaluation. There are potentially serious negative consequences in the context of evaluation that can result from the use of VAM based on incomplete or flawed data, as well as from the misinterpretation or misuse of the VAM results.”

The statement also notes that there are promising alternatives to VAM currently in use in the United States that merit attention, including the use of teacher observation data and peer assistance and review models that provide formative and summative assessments of teaching and honor teachers’ due process rights.

The statement concludes: “The value of high-quality, research-based evidence cannot be over-emphasized. Ultimately, only rigorously supported inferences about the quality and effectiveness of teachers, educational leaders, and preparation programs can contribute to improved student learning.” Thus, the statement also calls for substantial investment in research on VAM and on alternative methods and models of educator and educator preparation program evaluation.

The AERA Statement includes 8 technical requirements for the use of VAM:

  1. “VAM scores must only be derived from students’ scores on assessments that meet professional standards of reliability and validity for the purpose to be served…Relevant evidence should be reported in the documentation supporting the claims and proposed uses of VAM results, including evidence that the tests used are a valid measure of growth [emphasis added] by measuring the actual subject matter being taught and the full range of student achievement represented in teachers’ classrooms” (p. 3).
  2. “VAM scores must be accompanied by separate lines of evidence of reliability and validity that support each [and every] claim and interpretative argument” (p. 3).
  3. “VAM scores must be based on multiple years of data from sufficient numbers of students…[Related,] VAM scores should always be accompanied by estimates of uncertainty to guard against [simplistic] overinterpretation[s] of [simple] differences” (p. 3).
  4. “VAM scores must only be calculated from scores on tests that are comparable over time…[In addition,] VAM scores should generally not be employed across transitions [to new, albeit different tests over time]” (AERA Council, 2015, p. 3).
  5. “VAM scores must not be calculated in grades or for subjects where there are not standardized assessments that are accompanied by evidence of their reliability and validity…When standardized assessment data are not available across all grades (K–12) and subjects (e.g., health, social studies) in a state or district, alternative measures (e.g., locally developed assessments, proxy measures, observational ratings) are often employed in those grades and subjects to implement VAM. Such alternative assessments should not be used unless they are accompanied by evidence of reliability and validity as required by the AERA, APA, and NCME Standards for Educational and Psychological Testing” (p. 3).
  6. “VAM scores must never be used alone or in isolation in educator or program evaluation systems…Other measures of practice and student outcomes should always be integrated into judgments about overall teacher effectiveness” (p. 3).
  7. “Evaluation systems using VAM must include ongoing monitoring for technical quality and validity of use…Ongoing monitoring is essential to any educator evaluation program and especially important for those incorporating indicators based on VAM that have only recently been employed widely. If authorizing bodies mandate the use of VAM, they, together with the organizations that implement and report results, are responsible for conducting the ongoing evaluation of both intended and unintended consequences. The monitoring should be of sufficient scope and extent to provide evidence to document the technical quality of the VAM application and the validity of its use within a given evaluation system” (AERA Council, 2015, p. 3).
  8. “Evaluation reports and determinations based on VAM must include statistical estimates of error associated with student growth measures and any ratings or measures derived from them…There should be transparency with respect to VAM uses and the overall evaluation systems in which they are embedded. Reporting should include the rationale and methods used to estimate error and the precision associated with different VAM scores. Also, their reliability from year to year and course to course should be reported. Additionally, when cut scores or performance levels are established for the purpose of evaluative decisions, the methods used, as well as estimates of classification accuracy, should be documented and reported. Justification should [also] be provided for the inclusion of each indicator and the weight accorded to it in the evaluation process…Dissemination should [also] include accessible formats that are widely available to the public, as well as to professionals” ( p. 3-4).

The bottom line:  Tennessee’s use of TVAAS in teacher evaluations is highly problematic.

More on TVAAS:

Not Yet TNReady

The Worst Teachers

Validating the Invalid

More on Peer Assistance and Review:

Is PAR a Worthy Investment?

For more on education politics and policy in Tennessee, follow @TNEdReport