A Little Less Bad

From a story in Chalkbeat:

Tennessee’s teacher evaluation system is more accurate than ever in measuring teacher quality…

That’s the conclusion drawn from a report on the state’s teacher evaluation system conducted by the State Department of Education.

The idea is that the system is improving.

Here’s the evidence the report uses to justify the claim of an improving evaluation system:

1) Teacher observation scores now more closely align with teacher TVAAS scores — TVAAS is the value-added modeling system used to determine a teacher’s impact on student growth

2) More teachers in untested subjects are now being evaluated using the portfolio system rather than TVAAS data from students they never taught

On the second item, I’d note that previously, 3 districts were using the a portfolio model and now 11 districts use it. This model allows related-arts teachers and those in other untested subjects to present a portfolio of student work to demonstrate that teacher’s impact on growth. The model is generally applauded by teachers who have a chance to use it.

However, there are 141 districts in Tennessee and 11 use this model. Part of the reason is the time it takes to assess portfolios well and another reason is the cost associated with having trained evaluators assess the portfolios. Since the state has not (yet) provided funding for the use of portfolios, it’s no surprise more districts haven’t adopted the model. If the state wants the evaluation model to really improve (and thereby improve teaching practice), they should support districts in their efforts to provide meaningful evaluation to teachers.

A portfolio system could work well for all teachers, by the way. The state could move to a system of project-based learning and thus provide a rich source of material for both evaluating student mastery of concepts AND teacher ability to impact student learning.

On to the issue of TVAAS and observation alignment. Here’s what the report noted:

Among the findings, state education leaders are touting the higher correlation between a teacher’s value-added score (TVAAS), which estimates how much teachers contribute to students’ growth on statewide assessments, and observation scores conducted primarily by administrators.

First, the purpose of using multiple measures of teacher performance is not to find perfect alignment, or even strong correlation, but to utilize multiple inputs to assess performance. Pushing for alignment suggests that the department is actually looking for a way to make TVAAS the central input driving teacher evaluation.

Advocates of this approach will tell suggest that student growth can be determined accurately by TVAAS and that TVAAS is a reliable predictor of teacher performance.

I would suggest that TVAAS, like most value-added models, is not a significant differentiator of teacher performance. I’ve written before about the need for caution when using value-added data to evaluate teachers.

More recently, I wrote about the problems inherent in attempting to assign growth scores when shifting to a new testing regime, as Tennessee will do next year when it moves from TCAP to TNReady. In short, it’s not possible to assign valid growth scores when comparing two entirely different tests.  Researchers at RAND noted:

We find that the variation in estimated effects resulting from the different mathematics achievement measures is large relative to variation resulting from choices about model specification, and that the variation within teachers across achievement measures is larger than the variation across teachers. These results suggest that conclusions about individual teachers’ performance based on value-added models can be sensitive to the ways in which student achievement is measured.
These findings align with similar findings by Martineau (2006) and Schmidt et al (2005)
You get different results depending on the type of question you’re measuring.

The researchers tested various VAM models (including the type used in TVAAS) and found that teacher effect estimates changed significantly based on both what was being measured AND how it was measured. 

And they concluded:

Our results provide a clear example that caution is needed when interpreting estimated teacher effects because there is the potential for teacher performance to depend on the skills that are measured by the achievement tests.

So, even if you buy the idea that TVAAS is a significant differentiator of teacher performance, drawing meaningful conclusions from next year’s TNReady simply is not reliable.

The state is touting improvement in a flawed system that may now be a little less bad.  And because they insist on estimating growth from two different tests with differing methodologies, the growth estimates in 2016 will be unreliable at best. If they wanted to improve the system, they would take two to three years to build growth data based on TNReady — that would mean two t0 three years of NO TVAAS data in teacher evaluation.

Alternatively, the state could move to a system of project-based learning and teacher evaluation and professional development based on a Peer Assistance and Review Model. Such an approach would be both student-centered and result in giving teachers the professional respect they deserve. It also carries a price tag — but our students are worth doing the work of both reallocating existing education dollars and finding new ways to invest in our schools.

For more on education politics and policy in Tennessee, follow @TNEdReport

 

 

 

Validating the Invalid?

The Tennessee House of Representatives passed legislation today (HB 108) that makes changes to current practice in teacher evaluation as Tennessee transitions to its new testing regime, TNReady.

The changes adjust the percentage of a teacher’s evaluation that is dependent on TVAAS scores to 10% next year, 20% the following year, and back to the current 35% by the 2017-18 academic year.

This plan is designed to allow for a transition period to the new TNReady tests which will include constructed-response questions and be aligned to the so-called Tennessee standards which match up with the Common Core State Standards.

Here’s the problem: There is no statistically valid way to predict expected growth on a new test based on the historic results of TCAP. First, the new test has (supposedly) not been fully designed. Second, the test is in a different format. It’s both computer-based and it contains constructed-response questions. That is, students must write-out answers and/or demonstrate their work.

Since Tennessee has never had a test like this, it’s impossible to predict growth at all. Not even with 10% confidence. Not with any confidence. It is the textbook definition of comparing apples to oranges.

Clearly, legislators feel like at the very least, this is an improvement. A reasonable accommodation to teachers as our state makes a transition.

But, how is using 10% of an invalid number a good thing? Should any part of a teacher’s evaluation be made up of a number that reveals nothing at all about that teacher’s performance?

While value-added data alone is a relatively poor predictor of teacher performance, the value-added estimate used next year is especially poor because it is not at all valid.

But, don’t just take my word for it. Researchers studying the validity of value-added measures asked whether value-added gave different results depending on the type of question asked. Particularly relevant now because Tennessee is shifting to a new test with different types of questions.

Here’s what Lockwood and McCaffrey (2007) had to say in the Journal of Educational Measurement:

We find that the variation in estimated effects resulting from the different mathematics achievement measures is large relative to variation resulting from choices about model specification, and that the variation within teachers across achievement measures is larger than the variation across teachers. These results suggest that conclusions about individual teachers’ performance based on value-added models can be sensitive to the ways in which student achievement is measured.
These findings align with similar findings by Martineau (2006) and Schmidt et al (2005)
You get different results depending on the type of question you’re measuring.

The researchers tested various VAM models (including the type used in TVAAS) and found that teacher effect estimates changed significantly based on both what was being measured AND how it was measured. 

And they concluded:

Our results provide a clear example that caution is needed when interpreting estimated teacher effects because there is the potential for teacher performance to depend on the skills that are measured by the achievement tests.

If you measure different skills, you get different results. That decreases (or eliminates) the reliability of those results. TNReady is measuring different skills in a different format than TCAP. It’s BOTH a different type of test AND a test on different standards. Any value-added comparison between the two tests is statistically suspect, at best. In the first year, such a comparison is invalid and unreliable. As more years of data become available, it may be possible to make some correlation between past TCAP results and TNReady scores.

Or, if the state is determined to use growth scores (and wants to use them with accuracy), they will wait several years and build completely new growth models based on TNReady alone. At least three years of data would be needed in order to build such a model.

It seems likely that the Senate will follow the House’s lead on Monday and overwhelmingly support the proposed evaluation changes. But in doing so, they should be asking themselves if it’s really ok to base any part of a teacher’s evaluation on numbers that reliably predict nothing.

More on Value-Added:

Real World Harms of Value-Added Data

Struggles with Value-Added Data

 

Is THAT even legal?

That’s the question the Tennessee Education Association is asking about the use of value-added data (TVAAS) in teacher evaluations.

The TEA, joining with the National Education Association, has filed a lawsuit challenging the constitutionality of Tennessee’s use of TVAAS data in teacher evaluations.

According to a press release, TEA is specifically concerned about teachers who receive value-added scores based on students they have never taught. A significant number of Tennessee teachers currently receive a portion of their evaluation score based on TVAAS scores from school-wide or other data, meaning teachers are graded based on students they’ve never taught.

The release states:

More than half of the public school teachers in Tennessee receive evaluations that are based substantially on standardized test scores of students in subjects they do not teach. The lawsuit seeks relief for those teachers from the arbitrary and irrational practice of measuring their effectiveness with statistical estimates based on standardized test scores from students they do not teach and may have never met. 

While Governor Haslam is proposing that the legislature reduce the impact of TVAAS scores on teacher evaluations during the state’s transition to new standardized tests, his proposal does not address the issues of statistical validity with the transition. There is no way to determine how TCAP scores will interface with the scores from a test that has not even been developed yet. To hold teachers accountable for data generated in such an unreliable fashion is not only statistically suspect, it’s disrespectful.

Finally, it’s worth noting that value-added data doesn’t do much in terms of differentiating teacher performance. Of course, even if it did, holding teachers accountable for students they don’t teach defies logic.

For more on education politics and policy in Tennessee, follow @TNEdReport

 

Dear Jim

Tomorrow, Knox County’s Director of Schools, Jim McIntyre, will testify before the Senate HELP Committee as part of ESEA reauthorization hearings being held by Sen. Lamar Alexander.

Ahead of his testimony, 9th District Knox County School Board member Amber Rountree sent McIntyre her thoughts on what he should say. This is her letter:

Dear Jim:
Thank you for the opportunity to give input on your upcoming testimony regarding the reauthorization of No Child Left Behind (“NCLB”).

As you stated in your email to the Board, you have been bestowed an honor to represent our students, our staff and the great state of Tennessee. I know you will share the wonderful innovation happening in Knox County Schools, but I implore you to provide a realistic picture of how NCLB (and its waiver) has impacted our schools.  I hope as you prepare your testimony you find courage to speak hard truths about the current state of our schools, including the following points:

More accountability≠better education. While we need a way to measure student progress, we must discontinue high-stakes testing that is not developmentally appropriate.  Punishing students, teachers and schools for results of these tests is simply unethical, especially while companies like Pearson profit from this punishment.

Restore local control.  Top down mandates from the federal government via NCLB have not led to a better outcome for students.  In fact, in our own district the achievement gap is widening.  Return the decision making to the hands of our state and local boards of education, along with controls to ensure punitive high-stakes testing does not continue.

Rethink the “Teacher Incentive Fund.”  Would you pay a firefighter based on the number of fires they successfully extinguished? Merit pay does not directly correlate to increased student performance.  A wiser choice would be to use the funding for smaller teacher-student ratios, which directly improve student outcomes.

Public dollars, public schools.  Vouchers and charters are a path to privatize public education.  When President Johnson signed ESEA into law, his intent was to help public schools succeed, not see those dollars funneled into private ventures which are not held to the same rigorous standards as public schools.
I concur with President Johnson’s remark that “there is no higher ground than a schoolroom or a more hopeful place than a classroom.”  The brightness of hope for our students and teachers has dimmed under the oppressive mandates of NCLB.  You’ve been given a gift to help restore that hope; my wish is that you use it wisely.
Yours in education,
Amber Rountree,  District 9 Representative

 

For more on education politics and policy in Tennessee, follow @TNEdReport

SPEAK Members Seek KCEA Posts

Lauren Hopson, whose remarks at a Knox County School Board meeting gained national attention, is seeking the Presidency of the Knox County Education Association. Hopson is joined in campaigning by Amy Cate, who is seeking the Vice Presidency, and Linda Holtzclaw, running for Secretary.

Hopson’s speech was the catalyst for a movement that become SPEAK: Students, Parents, and Educators Across Knox County.

The group speaks out on education issues and even recruited and supported some successful candidates in the recent school board election.

Hopson sought to draw attention to Knox County Schools policies that she believed harmed both teachers and their students. Now, SPEAK keeps Knox County citizens informed of relevant education issues and regularly engages local policymakers in discussions about how to improve Knox County Schools.

Here is the video that helped launch Hopson:

 

SPEAK Members Marching:

SPEAK members marching
For more on education politics and policy in Tennessee, follow @TNEdReport

PET Looks to 2015

A response to Governor Haslam’s recently announced teacher support initiatives by JC Bowman and Samantha Bates of Professional Educators of Tennessee (PET)

 

The announcement by Governor Bill Haslam addressing testing, evaluations, local control and teacher input was a much needed statement, as Tennessee is heading into the 2015 legislative session. Keeping in mind that each branch of government has a distinct and separate role, it is appropriate for Governor Haslam to identify changing priorities. As always, the key is in implementation of policies. Many policies sound good. They simply have to be executed correctly.

It is always good to step back and put some political philosophy behind the policy. However, the real message educators need to hear from elected leaders is that they are trusted. We need to start a fresh conversation on evaluating how we assess our educators, which may mean a change in the way we measure engagement.

When did test results became the be-all and end-all of our education experience? Is standardized testing so reliable that it has ended the search for something better to determine the quality of our education experience? And while numbers may help us understand our world, we recognize that they do not tell us the entire story.

Most local school districts understand that ability of their instructional personnel is the only real differentiator between them and other local districts. Therefore, it is imperative that we start treating our educators like one of our most important assets. And it is only common sense that one of the key items policymakers need to address in 2015 will be teacher salaries.

However, educators do not enter this field of public education for the income; they are there for the outcomes. If the perception within Tennessee is that teaching is not a celebrated profession, we certainly will not get the young talented people to pursue a career in public education as a profession.

We have steadfastly maintained that requiring school districts to simultaneously implement new standards, new teacher evaluations and perhaps a new curriculum, as well as new testing demands, will continue to place enormous pressure at the local level. More information and feedback on state assessments to help teachers improve student achievement is a welcome addition to the discussion. The use and/or overuse of testing remain a conversation worthy of public debate.

Tennessee will need to continue allocate resources devoted to the transition of standards. As we have argued, we believe it is time to move beyond the Common Core debate. We need to continuously build state specific standards that are challenging and meet the needs of Tennesseans. This needs to be done with legislative input and with the involvement of Tennessee educators.

The key item we took away from Governor Haslam’s latest proposal is his willingness to hear teacher concerns. It has taken us a long time to get to that point. However, it was a welcome relief to many educators, as we are now positioned to reset the dialogue. The area of improved teacher communication and collaboration has long been needed. We hope a new commissioner of education will truly embrace this concept.

If the right people are brought together for the right purpose, we believe anything is possible for Tennessee children and those who choose to educate our students. Dreaming big should not be just for the children in our classrooms, it should be for the stakeholders and policymakers in our state as well.

For more on education politics and policy in Tennessee, follow @TNEdReport

Little Value Added?

 

That’s the conclusion teacher Jon Alfuth draws about Governor Bill Haslam’s recently announced changes to teacher evaluation and support.

Alfuth notes with frustration that Haslam appears happy to support teachers in ways that don’t involve any new money.

Reducing the weight given TVAAS on a teacher’s evaluation, for example, doesn’t cost anything. Adding a few teachers to a “cabinet” to give feedback on tests is welcome change, but also doesn’t carry a price tag.

Haslam’s changes still unfairly assess teachers in non-tested subjects, in Alfuth’s view:

While reducing the percentage from 25 to 15 percent achievement data for non-EOC teachers is a step in the right direction, I don’t feel that it goes far enough. I personally think it’s unfair to use test scores from courses not taught by a teacher in their evaluation given the concerns surrounding the reliability of these data systems overall.

And, Alfuth says, the financial support teachers and schools need is simply not discussed:

Consider the teacher salary discussion we’ve been having here in Tennessee. This is something that Tennessee Teachers have been clamoring for and which the governor promised but then went back on this past spring. There’s no mention of other initiatives that would require extra funding, such as BEP2.0, which would provide millions of additional dollars to our school districts across the state and do much to help teachers. There’s also no mention of expanding training Common Core trainng, which is essential if we’re going to continue to enable teachers to be successful when the three year phase in of growth scores winds down.

In short, while the proposed changes are step forward, at least in the view of one teacher, much more can be done to truly support teachers and their students.

More on the importance of investing in teacher pay:

Notes on Teacher Pay

More on the state’s broken school funding formula, the BEP:

A BEP Lawsuit?

The Broken BEP

What is BEP 2.0?

For more from Jon Alfuth and education issues in Memphis, follow @BluffCityEd

For more on education politics and policy in Tennessee, follow @TNEdReport

Value Added Changes

 

In what is certain to be welcome news to many teachers across the state, Governor Bill Haslam announced yesterday that he will be proposing changes to the state’s teacher evaluation process in the 2015 legislative session.

Perhaps the most significant proposal is to reduce the weight of value-added data on teacher evaluations during the transition to a new test for Tennessee students.

From the Governor’s press release explaining the proposed changes:

The governor’s proposal would:
•        Adjust the weighting of student growth data in a teacher’s evaluation so that the new state assessments in ELA and math will count 10 percent of the overall evaluation in the first year of
administration (2016), 20 percent in year two (2017) and 35 percent in year
three (2018). Currently 35 percent of an educator’s evaluation is comprised of
student achievement data based on student growth;
•        Lower the weight of student achievement growth for teachers in non-tested grades and subjects
from 25 percent to 15 percent;
•        And make explicit local school district discretion in both the qualitative teacher evaluation model that is used for the observation portion of the evaluation as well as the specific
weight student achievement growth in evaluations will play in personnel
decisions made by the district.

 

The proposal does not go as far as some have proposed, but it does represent a transition period to new tests that teachers have been seeking.  It also provides more local discretion in how evaluations are conducted.

Some educators and critics question the ability of value-added modeling to accurately predict teacher performance.

In fact, the American Statistical Association released a statement on value-added models that says, in part:

Most VAM studies find that teachers account for about 1% to 14% of the variability in test scores

Additional analysis of the ability of value-added modeling to predict significant differences in teacher performance finds that this data doesn’t effectively differentiate among teachers.

I certainly have been critical of the over-reliance on value-added modeling in the TEAM evaluation model used in Tennessee. While the proposed change ultimately returns to using VAM for a significant portion of teacher scores, it also represents an opportunity to both transition to a new test AND explore other options for improving the teacher evaluation system.

For more on value-added modeling and its impact on the teaching profession:

Saving Money and Supporting Teachers

Real World Harms of Value-Added Data

Struggles with Value-Added Data

An Ineffective Teacher?

Principals’ Group Challenges VAM

 

For more on education policy and politics in Tennessee, follow @TNEdReport

Ravitch: Ed Reform is a Hoax

Education scholar and activist Diane Ravitch spoke at Vanderbilt University in Nashville last night at an event hosted by Tennesseans Reclaiming Educational Excellence (TREE), the Tennessee BATs (Badass Teachers), and the Momma Bears.

Ravitch touched on a number of hot-button education issues, including vouchers, charter schools, teacher evaluations, and testing. Many of these issues are seeing plenty of attention in Tennessee public policy circles both on the local and state levels.

She singled out K12, Inc. as a bad actor in the education space, calling the Tennessee Virtual Academy it runs a “sham.”

Attempts have been made to cap enrollment and shut down K12, Inc. in Tennessee, but they are still operating this year. More recently, the Union County School Board defied the State Department of Education and allowed 626 students to remain enrolled in the troubled school. The reason? Union County gets a payoff of $132,000 for their contract with K12.

Ravitch noted that there are good actors in the charter sector, but also said she adamantly opposes for-profit charter schools. Legislation that ultimately failed in 2014 would have allowed for-profit charter management companies to be hired by Tennessee charter schools.

On vouchers, an issue that has been a hot topic in the last two General Assemblies, Ravitch pointed to well-established data from Milwaukee that vouchers have made no difference in overall student performance.

Despite the evidence against vouchers, it seems quite likely they will again be an issue in the 2015 General Assembly. In fact, the Koch Brothers and their allies spent heavily in the recent elections to ensure that vouchers are back on the agenda.

Ravitch told the crowd that using value-added data to evaluate teachers makes no sense. The Tennessee Value-Added Assessment System (TVAAS) has been around since the BEP in 1992. It was created by UT Ag Professor Bill Sanders. Outgoing Commissioner of Education Kevin Huffman made an attempt to tie teacher licenses to TVAAS scores, but that was later repealed by the state board of education. A careful analysis of the claims of value-added proponents demonstrates that the data reveals very little in terms of differentiation among teachers.

Ravitch said that instead of punitive evaluation systems, teachers need resources and support. Specifically, she mentioned Peer Assistance and Review as an effective way to provide support and meaningful development to teachers.

A crowd of around 400 listened and responded positively throughout the hour-long speech. Ravitch encouraged the audience to speak up about the harms of ed reform and rally for the reforms and investments our schools truly need.

For more on education politics and policy in Tennessee, follow @TNEdReport

Why TN Teachers Didn’t Like Kevin Huffman

Kevin Huffman announced yesterday he’s leaving his post as Commissioner of Education. The news was met positively by many teachers around the state. But, why didn’t Tennessee teachers care for Kevin Huffman? Why did a number of local teacher associations vote “no confidence” in Huffman in 2013? Why did Directors from across the state sign a letter telling the Governor that Huffman needed to do a better job?

I wrote a post for a different blog back in 2011, Huffman’s first year, about his remarks on teacher evaluation. In short, he got off to a bad start in terms of communicating with and about teachers, and never recovered.

Here’s that post from 2011 in its entirety, with some notes about what has happened since then included:

Tennessee’s Commissioner of Education, Kevin Huffman, offered his thoughts today on the state’s new evaluation system for teachers which takes effect this year.

 

While I certainly agree that the evaluation system needed significant improvement, I have some concerns about the Commissioner’s statements.

 

Specifically, he notes:

 

Tennessee is now a few weeks into a new era of evaluation. The new system is strong, though not perfect, and it represents a dramatic leap forward over the past system that told nearly all teachers they had succeeded, even when students had failed.

 

This statement assumes that the poor performance of Tennessee students on the National Assessment of Educational Progress (NAEP) was solely or primarily the result of bad teachers. By his calculations, since 70 percent of students failed to meet satisfactory progress on the NAEP, 70 percent of Tennessee teachers must not be performing up to par.

 

What’s missing from his analysis, however, is the reality that until 2010, Tennessee had incredibly low standards relative to the NAEP. In fact, nearly 87% of students were deemed proficient on TCAPs despite only 27% testing proficient on the NAEP. Here’s the deal: Tennessee schools were held accountable under NCLB for hitting TCAP benchmarks. Tennessee policymakers set the standard. And Tennessee teachers were hitting the mark they were told was important. In fact, data suggest more and more Tennessee students were marching toward TCAP proficiency each year. By that indicator, Tennessee teachers were doing a fine job. Policymakers set a target, and Tennessee teachers hit it year after year. Since curriculum and accountability were not tied to NAEP, it seems unreasonable to expect that teachers would be helping students hit NAEP benchmarks.

 

Huffman’s remarks also ignore this reality: Tennessee spends less per student than most of our neighboring states. 8 states test 100% of graduates on the ACT. Tennessee ranks 7th in that group, below every other state that spends MORE per pupil than Tennessee. Kentucky spends about $1500 more per student than Tennessee and gets significantly better results on the NAEP year after year. The point being: teachers can only do so much with limited resources and our state has done a pretty good job of limiting the resources.

 

Huffman also notes:

 

As new student assessments are developed and vetted by Tennessee educators and experts, we expect that next year, it will be possible for 70 percent of teachers to be evaluated by their own student-assessment results. Eventually, more than 90 percent of teachers will have such options.

This dream still hasn’t been realized — Portfolios are available for some non-tested subjects, but are not in wide use due to cost.

So more teachers will have their own value-added data. This means more assessments (TESTS) for Tennessee students. Will there now be TCAP-like tests in grades K-2? As the parent of a Kindergartener, I certainly hope not. What about related arts? Will there be a written test for an instrumental music course? Or is the value-added that a student who previously struggled with the flute now excels? How is that measured? In performance-based art, music, and theatre classes, will more time be spent drilling on concepts so a kid can pass a written test rather than on actually improving one’s ability to draw, sing, or perform?

 

Finally, the new evaluations are time-intensive and do provide regular feedback. That’s a good thing. However, there’s no indication of available funding for meaningful professional development tied to the evaluations. There is yet to be a serious discussion of funding for mentors for early career teachers to help them get up to speed on key concepts and improve their technique. Teach for America (where Huffman worked as a teacher and then as a national organizational leader) relies heavily on intensive support for their Corps members. Lessons are video-taped, coaches are provided, feedback is regular and strategies for improvement are offered. Research suggests that intensive mentoring in the first two years of a teacher’s career not only improves their practice and increases retention, but also results in higher student achievement.

 

Tennessee’s new evaluation system for teachers is no doubt an improvement. But unless that system is coupled with meaningful support for teachers and adequate classroom resources, we’ll still find ourselves far behind the rest of the country.

There’s been no significant commitment to professional development or intensive mentoring by the state. Teachers didn’t get a promised raise this year.

So, Tennessee teachers started off hearing from Huffman that they had failed. Then, resources for support didn’t materialized and the transition to Common Core wasn’t well-communicated. Huffman suggested the same flawed, value-added based evaluations were responsible for a 2013 NAEP boost, and then a promised pay raise was taken away.

Is it any wonder Tennessee teachers aren’t too sad to see Huffman go?

 

For more on education politics and policy in Tennessee, follow @TNEdReport