Value Added Changes

 

In what is certain to be welcome news to many teachers across the state, Governor Bill Haslam announced yesterday that he will be proposing changes to the state’s teacher evaluation process in the 2015 legislative session.

Perhaps the most significant proposal is to reduce the weight of value-added data on teacher evaluations during the transition to a new test for Tennessee students.

From the Governor’s press release explaining the proposed changes:

The governor’s proposal would:
•        Adjust the weighting of student growth data in a teacher’s evaluation so that the new state assessments in ELA and math will count 10 percent of the overall evaluation in the first year of
administration (2016), 20 percent in year two (2017) and 35 percent in year
three (2018). Currently 35 percent of an educator’s evaluation is comprised of
student achievement data based on student growth;
•        Lower the weight of student achievement growth for teachers in non-tested grades and subjects
from 25 percent to 15 percent;
•        And make explicit local school district discretion in both the qualitative teacher evaluation model that is used for the observation portion of the evaluation as well as the specific
weight student achievement growth in evaluations will play in personnel
decisions made by the district.

 

The proposal does not go as far as some have proposed, but it does represent a transition period to new tests that teachers have been seeking.  It also provides more local discretion in how evaluations are conducted.

Some educators and critics question the ability of value-added modeling to accurately predict teacher performance.

In fact, the American Statistical Association released a statement on value-added models that says, in part:

Most VAM studies find that teachers account for about 1% to 14% of the variability in test scores

Additional analysis of the ability of value-added modeling to predict significant differences in teacher performance finds that this data doesn’t effectively differentiate among teachers.

I certainly have been critical of the over-reliance on value-added modeling in the TEAM evaluation model used in Tennessee. While the proposed change ultimately returns to using VAM for a significant portion of teacher scores, it also represents an opportunity to both transition to a new test AND explore other options for improving the teacher evaluation system.

For more on value-added modeling and its impact on the teaching profession:

Saving Money and Supporting Teachers

Real World Harms of Value-Added Data

Struggles with Value-Added Data

An Ineffective Teacher?

Principals’ Group Challenges VAM

 

For more on education policy and politics in Tennessee, follow @TNEdReport

Ravitch: Ed Reform is a Hoax

Education scholar and activist Diane Ravitch spoke at Vanderbilt University in Nashville last night at an event hosted by Tennesseans Reclaiming Educational Excellence (TREE), the Tennessee BATs (Badass Teachers), and the Momma Bears.

Ravitch touched on a number of hot-button education issues, including vouchers, charter schools, teacher evaluations, and testing. Many of these issues are seeing plenty of attention in Tennessee public policy circles both on the local and state levels.

She singled out K12, Inc. as a bad actor in the education space, calling the Tennessee Virtual Academy it runs a “sham.”

Attempts have been made to cap enrollment and shut down K12, Inc. in Tennessee, but they are still operating this year. More recently, the Union County School Board defied the State Department of Education and allowed 626 students to remain enrolled in the troubled school. The reason? Union County gets a payoff of $132,000 for their contract with K12.

Ravitch noted that there are good actors in the charter sector, but also said she adamantly opposes for-profit charter schools. Legislation that ultimately failed in 2014 would have allowed for-profit charter management companies to be hired by Tennessee charter schools.

On vouchers, an issue that has been a hot topic in the last two General Assemblies, Ravitch pointed to well-established data from Milwaukee that vouchers have made no difference in overall student performance.

Despite the evidence against vouchers, it seems quite likely they will again be an issue in the 2015 General Assembly. In fact, the Koch Brothers and their allies spent heavily in the recent elections to ensure that vouchers are back on the agenda.

Ravitch told the crowd that using value-added data to evaluate teachers makes no sense. The Tennessee Value-Added Assessment System (TVAAS) has been around since the BEP in 1992. It was created by UT Ag Professor Bill Sanders. Outgoing Commissioner of Education Kevin Huffman made an attempt to tie teacher licenses to TVAAS scores, but that was later repealed by the state board of education. A careful analysis of the claims of value-added proponents demonstrates that the data reveals very little in terms of differentiation among teachers.

Ravitch said that instead of punitive evaluation systems, teachers need resources and support. Specifically, she mentioned Peer Assistance and Review as an effective way to provide support and meaningful development to teachers.

A crowd of around 400 listened and responded positively throughout the hour-long speech. Ravitch encouraged the audience to speak up about the harms of ed reform and rally for the reforms and investments our schools truly need.

For more on education politics and policy in Tennessee, follow @TNEdReport

Why TN Teachers Didn’t Like Kevin Huffman

Kevin Huffman announced yesterday he’s leaving his post as Commissioner of Education. The news was met positively by many teachers around the state. But, why didn’t Tennessee teachers care for Kevin Huffman? Why did a number of local teacher associations vote “no confidence” in Huffman in 2013? Why did Directors from across the state sign a letter telling the Governor that Huffman needed to do a better job?

I wrote a post for a different blog back in 2011, Huffman’s first year, about his remarks on teacher evaluation. In short, he got off to a bad start in terms of communicating with and about teachers, and never recovered.

Here’s that post from 2011 in its entirety, with some notes about what has happened since then included:

Tennessee’s Commissioner of Education, Kevin Huffman, offered his thoughts today on the state’s new evaluation system for teachers which takes effect this year.

 

While I certainly agree that the evaluation system needed significant improvement, I have some concerns about the Commissioner’s statements.

 

Specifically, he notes:

 

Tennessee is now a few weeks into a new era of evaluation. The new system is strong, though not perfect, and it represents a dramatic leap forward over the past system that told nearly all teachers they had succeeded, even when students had failed.

 

This statement assumes that the poor performance of Tennessee students on the National Assessment of Educational Progress (NAEP) was solely or primarily the result of bad teachers. By his calculations, since 70 percent of students failed to meet satisfactory progress on the NAEP, 70 percent of Tennessee teachers must not be performing up to par.

 

What’s missing from his analysis, however, is the reality that until 2010, Tennessee had incredibly low standards relative to the NAEP. In fact, nearly 87% of students were deemed proficient on TCAPs despite only 27% testing proficient on the NAEP. Here’s the deal: Tennessee schools were held accountable under NCLB for hitting TCAP benchmarks. Tennessee policymakers set the standard. And Tennessee teachers were hitting the mark they were told was important. In fact, data suggest more and more Tennessee students were marching toward TCAP proficiency each year. By that indicator, Tennessee teachers were doing a fine job. Policymakers set a target, and Tennessee teachers hit it year after year. Since curriculum and accountability were not tied to NAEP, it seems unreasonable to expect that teachers would be helping students hit NAEP benchmarks.

 

Huffman’s remarks also ignore this reality: Tennessee spends less per student than most of our neighboring states. 8 states test 100% of graduates on the ACT. Tennessee ranks 7th in that group, below every other state that spends MORE per pupil than Tennessee. Kentucky spends about $1500 more per student than Tennessee and gets significantly better results on the NAEP year after year. The point being: teachers can only do so much with limited resources and our state has done a pretty good job of limiting the resources.

 

Huffman also notes:

 

As new student assessments are developed and vetted by Tennessee educators and experts, we expect that next year, it will be possible for 70 percent of teachers to be evaluated by their own student-assessment results. Eventually, more than 90 percent of teachers will have such options.

This dream still hasn’t been realized — Portfolios are available for some non-tested subjects, but are not in wide use due to cost.

So more teachers will have their own value-added data. This means more assessments (TESTS) for Tennessee students. Will there now be TCAP-like tests in grades K-2? As the parent of a Kindergartener, I certainly hope not. What about related arts? Will there be a written test for an instrumental music course? Or is the value-added that a student who previously struggled with the flute now excels? How is that measured? In performance-based art, music, and theatre classes, will more time be spent drilling on concepts so a kid can pass a written test rather than on actually improving one’s ability to draw, sing, or perform?

 

Finally, the new evaluations are time-intensive and do provide regular feedback. That’s a good thing. However, there’s no indication of available funding for meaningful professional development tied to the evaluations. There is yet to be a serious discussion of funding for mentors for early career teachers to help them get up to speed on key concepts and improve their technique. Teach for America (where Huffman worked as a teacher and then as a national organizational leader) relies heavily on intensive support for their Corps members. Lessons are video-taped, coaches are provided, feedback is regular and strategies for improvement are offered. Research suggests that intensive mentoring in the first two years of a teacher’s career not only improves their practice and increases retention, but also results in higher student achievement.

 

Tennessee’s new evaluation system for teachers is no doubt an improvement. But unless that system is coupled with meaningful support for teachers and adequate classroom resources, we’ll still find ourselves far behind the rest of the country.

There’s been no significant commitment to professional development or intensive mentoring by the state. Teachers didn’t get a promised raise this year.

So, Tennessee teachers started off hearing from Huffman that they had failed. Then, resources for support didn’t materialized and the transition to Common Core wasn’t well-communicated. Huffman suggested the same flawed, value-added based evaluations were responsible for a 2013 NAEP boost, and then a promised pay raise was taken away.

Is it any wonder Tennessee teachers aren’t too sad to see Huffman go?

 

For more on education politics and policy in Tennessee, follow @TNEdReport

The Value of the Report Card on Teacher Training

Every year, the Tennessee Higher Education Commission issues a Report Card on the state’s teacher training program. To evaluate educator effectiveness, THEC uses the Tennessee Value-Added Assessment System.

Which effectively renders the Report Card of little value.

Not included in the report is a teacher’s overall effectiveness score on the TEAM model. That would include both observed scores and value-added data, plus other achievement measures. That would be a more robust score to report, but it’s not included.

I’ve written before on the very limited value of value-added data.

Here are some highlights of why we learn almost nothing from the THEC report in terms of whether or not a teacher education program is actually doing a good job:

Here’s the finding that gets all the attention: A top 5 percent teacher (according to value-added modeling or VAM) can help a classroom of students (28) earn $250,000 more collectively over their lifetime.

Now, a quarter of a million sounds like a lot of money.

But, in their sample, a classroom was 28 students. So, that equates to $8928.57 per child over their lifetime. That’s right, NOT $8928.57 MORE per year, MORE over their whole life.

For more math fun, that’s $297.61 more per year over a thirty year career with a VAM-designated “great” teacher vs. with just an average teacher.

Yep, get your kid into a high value-added teacher’s classroom and they could be living in style, making a whole $300 more per year than their friends who had the misfortune of being in an average teacher’s room.

If we go all the way down to what VAM designates as “ineffective” teaching, you’d likely see that number double, or maybe go a little higher. So, let’s say it doubles plus some. Now, your kid has a low VAM teacher and the neighbor’s kid has a high VAM teacher. What’s that do to his or her life?

Well, it looks like this: The neighbor kid gets a starting job offer of $41,000 and your kid gets a starting offer of $40,000.

So, THEC uses a marginal indicator of educator effectiveness to make a significant determination about whether or not educator training programs are effective. At the very least, such a determination should also include observed scores of these teachers over time or the entire TEAM score.

Until then, the annual Report Card on teacher training will add little value to the education policy discussion in Tennessee.

For more on education politics and policy in Tennessee, follow @TNEdReport

TEA President on Testing and Education Reform

Blake Farmer of WPLN in Nashville has an interview with TEA President Gera Summerford that hits topics including an over-reliance on standardized testing, using value-added data to evaluate teachers, and charter schools.

In the interview, Summerford suggests a move toward common assessments, developed by teachers, to supplement or replace standardized testing.

She notes that the current model of teacher evaluation is not complete, and that multiple measures of effectiveness should included.  And Summerford notes that there are serious concerns about the validity of value-added data and it’s significance in the current teacher evaluation scheme.

The write-up and the entire interview can be found here.

TEA Files TVAAS Lawsuit in Knox County

Use of TVAAS is Arbitrary and Violates 14th Amendment, TEA Alleges

The Tennessee Education Association (TEA) has filed a lawsuit on behalf of a Knox County teacher who was denied a bonus under that school system’s pay plan after Tennessee Value-Added Assessment System (TVAAS) data for 10 of her students was unknowingly attributed to her.

TVAAS is Tennessee’s system of measuring student growth over time. It generates data based on student test scores on TCAP and end of course tests.

In this specific case, the teacher, Lisa Trout, was assigned TVAAS data for 10 students after being told her evaluation would be based on system-wide TVAAS data because she taught at an alternative school.

The TEA lawsuit cites two different memos which indicated that Ms. Trout could expect an evaluation (and bonus eligibility) to be based on system-wide data. At the conclusion of the school year, Ms. Trout was informed that her overall evaluation score, including observations and TVAAS data was a 4, making her eligible for a bonus under the Knox County pay plan.

When she did not receive the bonus as expected, she began asking questions about why the bonus had not been paid.  She ultimately determined that without her knowledge, a school counselor had assigned 10 students to Ms. Trout for the factoring of TVAAS scores.  The students were in an Algebra II course Ms. Trout taught, even though she does not hold an endorsement for teaching Alegbra II.

Though the suit does not specifically mention this, it should be noted that 10 students is a particularly small sample size subject to significant statistical anomaly.

The TEA lawsuit contends that Ms. Trout was owed the bonus based on Knox County School Board policy and in this specific instance, the bonus should have been paid.

Arbitrary?

The TEA goes on to contend that Ms. Trout and similarly situated teachers for whom there is little or no specific TVAAS data are held to an arbitrary standard in violation of the 14th Amendment of the U.S. Constitution.

Specifically, the suit notes: ” … the majority of teachers in the Knox County Schools … have had their eligibility for additional compensation (under the APEX bonus system) determined on the basis of the test scores of students they do not teach and/or the test scores of their students in subjects unrelated to the subjects they teach.”

The suit alleges that such a system violates the Equal Protection Clause of the 14th Amendment because some teachers are evaluated and receive bonuses based on the scores of their own students while other teachers are held accountable for students they do not teach and over which they have no influence or control.

In short, the entire system is flawed and should be discarded.

A spokesperson for TEA confirmed that the organization does not believe that teacher pay should be tied to TVAAS data.

On a related note, the Metro Nashville Public Schools recently announced it is putting plans to pay teachers in part based on TVAAS scores on hold indefinitely.

A TEA press release announcing the Knox County suit indicated that the organization anticipates additional lawsuits along these lines.

For more on Tennessee education politics and policy, follow @TNEdReport

 

Cameras in the Classroom?

Schools Matter has a series of posts up on the Measuring Effectiveness in Teaching project sponsored by the Gates Foundation.

Of note is the use of video cameras to record Tennessee classrooms and transmit the data for use in analyzing teaching behavior.  According to information obtained by Schools Matter, 120 schools in TN are using or have used the cameras to record classrooms.

The cameras were obtained and installed thanks to a grant from the Gates Foundation in the amount of $3.2 million.

The goal of the project is to take into account various measures of teaching practice and then use that information to determine what makes an “effective teacher.”

While not explicitly stated, it seems likely that the project will ultimately match up teachers with high value-added scores and their videotapes so as to determine which practices are most effective.  Teachers will then be encouraged to adopt the model practices as captured on video.

While this in itself is not bad practice, it is important that any data collected in this way is put to good use.  That is, it’s not enough to tape the lessons.  Will the TNDOE use the information to help coach struggling teachers? Will the TNDOE invest funds to provide early career mentoring, a method proven to increase teacher retention and improve teacher performance?

And, while at the outset, the idea behind the project seems to have some merit, the folks at Schools Matter raise some serious concerns.

Do teachers consent to have their classes taped? Are parents informed when a camera is used to tape their student in class? Is the use of this data made clear to both teachers and parents?

According to this piece, the cameras are turned on and off by the teacher and uploaded to the teacher’s account for sharing to appropriate parties. Certainly, that would include an administrator and also the data collection group.  So, it seems the teacher does have some control over when or whether the camera is on — unless there is a district or building policy dictating otherwise.

While the MET project may yet yield some interesting information, it’s not clear what will be done with that information.  And it’s not clear that the implementation of this project in Tennessee has been carried out with full disclosure to both teachers and parents.

If the basis of the project is to match teaching practices to value-added scores, I’d urge at least some caution.

Districts participating should inform parents about the collection of this data and how it may be used.

And, if the project DOES yield useful information, Tennessee should dispatch that information with an investment in training and support of its teachers.

For more on Tennessee education politics and policy, follow @TNEdReport

 

A Broader, Bolder SCORE Report

Today, newly-formed education advocacy group TREE (Tennesseans Reclaiming Educational Excellence) hosted a presentation by Elaine Weiss of the Broader, Bolder Approach to Education.

Weiss discussed recent Tennessee education policy in the context of the drivers of educational inequality.  She pointed to research suggesting that poverty is a significant contributor to student outcomes and noted other research that suggests as much as 2/3 of student outcomes are predicted by factors outside of school.

Later in the day, SCORE (Statewide Collaborative on Reforming Education) released its annual State of Education in Tennessee Report.

Both reports indicate Tennessee has much work to do to improve educational outcomes.  There were some similarities and some differences in the approaches presented, however.

The SCORE report outlined five specific priorities for Tennessee education policy in 2014.  I’ll examine those and note where the Broader, Bolder Approach supported by Weiss matches up and where there are differences.

Here are the SCORE priorities:

  • Maintaining a commitment to rigorous standards and assessments. The report says Tennessee must push forward with the continued implementation of the Common Core State Standards. It also points out that measuring student success with higher standards is needed for effective instruction, so Tennessee must continue its commitment to implementing the Partnership for Assessment of Readiness for College and Career (PARCC) assessments.
  • Strengthening schools through effective leadership. As Tennessee continues to implement student-centered initiatives it is crucial to have strong instructional leadership in every school, the report concludes. To build a pipeline of strong leaders, the state focus should be on creating an aligned, rigorous system for recruiting, training, evaluating and providing ongoing support to school leaders.
  • Expanding student access to great teaching. The report specifically calls for providing teachers with the tools and resources – including instructional coaching, collaborative planning time, and targeted professional learning – that will enable them to be experts in their profession. The report also calls for helping teacher preparation programs implement more selective admissions processes and rigorous curriculum requirements that prioritize the skills and knowledge teachers need to support students in the classroom.
  • Investing in technology to enhance instruction. The report says that although the upcoming online PARCC assessments are a catalyst for increasing technological capabilities in schools and school districts, investing in technology must be an ongoing priority and not just a one-time purchase. Students and teachers need daily access to technology and must be trained on using it, the report says.
  • Supporting students from kindergarten to career. The report points out that in today’s economy most careers require training after high school. It specifically calls for creating a data-rich environment that equips leaders, educators, and parents with the information and tools they need and a data-driven approach to making decisions about policy and practice that will advance student success. It also recommends expanded opportunities for more students to take AP, International Baccalaureate, dual-credit, and dual-enrollment courses and to study science, technology, engineering, and math (STEM) subjects.

And here is some analysis in light of the Broader, Bolder presentation:

Standards/Assessment: Weiss suggests that higher standards alone do not improve student achievement.  She points to persistent achievement gaps over time in spite of increasing standards, particularly in the NCLB era.  She also notes the stress caused to students and parents due to increased testing.  She notes that in some cases, as much as 30 instructional days are lost to testing and test prep. She suggests that raising student achievement over time must not simply be a function of high standards but also must include a commitment to supporting students and families outside of school.

Strengthening Schools Through Effective Leadership: Here, SCORE focuses on providing support for the development of effective school principals.  Weiss also suggests the importance of providing support and development to teachers and school leaders.  She would note that having an effective leader alone won’t close the gap, but that having supported leaders along with strong community supports can make a difference.

Expanding Student Access to Great Teaching: Weiss notes that Tennessee’s teachers are among the lowest paid in the country.  SCORE does not specifically address teacher pay in its report.  SCORE does call for improved professional development and additional collaboration with teachers going forward.  SCORE also calls for continued use of TVAAS to identify quality teachers.  Weiss is clear that value-added modeling is inconsistent and unreliable as a tool for evaluating teachers.  At the same time, SCORE calls for adding growth measures to additional teachers (these may or may not be in the form of tests that feed into the TVAAS formula).

Access to Technology: While Weiss might also place value on technology, she’d also suggest that access to summer learning opportunities and enriching extended learning is important.  She points to research suggesting that low-income students tend to proceed at a rate comparable to their peers but lose significant ground over the summer.  That is, what teachers are doing is working, but outside supports are lacking.  Adding meaningful time to the school calendar is one way to address this.

Supporting Kids from Kindergarten to Career:  Weiss absolutely states that kids need a variety of supports throughout school to ensure their success.  She’d likely expand this recommendation to include supporting kids from Pre-Kindergarten through career.  In fact, Weiss notes that while Tennessee was once moving quickly to grow a high-quality Pre-K program, the state has not added a single Pre-K seat since winning Race to the Top. Weiss explicitly recommends continuing the growth of the state’s Pre-K program in order to provide a proven intervention that closes opportunity gaps.

With the exception of TVAAS, it seems the Broader, Bolder Approach outlined by Weiss would generally be in agreement with the SCORE recommendations.  However, as the name indicates, the approach favored by Weiss would be broader and more expansive.  It would include expanded access to Pre-K. It would provide both targeted support to teachers AND significantly better pay for teachers.  It would examine ways to add valuable learning time to the school calendar.  And it would seek a more balanced approach to administering tests in order to avoid an over-reliance on test-based assessments.

For more on Tennessee education politics and policy, follow us @TNEdReport

 

 

How to Properly Deploy a New Teacher Evaluation System

Like Tennessee, Kentucky has a new teacher evaluation model — The Professional Growth and Effectiveness System (PGES). Similar to reforms in Tennessee, the new model uses multiple measures to evaluate teachers, including classroom observation and student growth.

Unlike Tennessee, Kentucky has rolled out its new evaluation in phases, improving it along the way based on feedback from teachers and administrators.

Here’s a description of how the model, to be fully implemented in 2014-15, has been rolled out:

During the 2012-2013 school year, over 50 school districts in Kentucky have participated in a field test of the new system.  The field test has allowed educator experience and feedback to inform improvements prior to the statewide pilot during the 2013-2014 school year.  During the statewide pilot in 2013-2014, as least 10% of the schools in each district will implement the Professional Growth & Effectiveness System.  In 2014-2015 the system will be fully implemented statewide with full accountability in Spring 2015.

That’s two years of pilot work before a single teacher is held fully accountable for the results of the new system.  Of course, those evaluated get the chance to have their practice informed by the strengths of the new system. But they also are not held back by problems that may need reform or improvement.

Contrast that with Tennessee, which implemented a new evaluation system in 2011-12.  Teachers were responsible for meeting the evaluation standards immediately.  There was no statewide pilot, no partial implementation, testing, and then improvement.  The evaluation has been changed or “improved” along the way, but that process has caused confusion as the standard by which teachers are evaluated seems to change from year to year.

Yes, there are strengths to evaluating teachers through multiple measures. Certainly, the old evaluation system warranted improvement.  But the implementation process directed by the Department of Education failed to adequately take into account teacher and administrator feedback. A more measured approach, as seen in Kentucky, could have helped build educator support and buy-in and could have improved the process without the fear that comes with instant accountability for a previously unused standard.

It’s not too late for Tennessee to “re-launch” it’s evaluation process in light of new Common Core tests.  A suspension of the use of TVAAS for teacher evaluation, as called for by PET and others could allow the state to re-examine the evaluations and phase-in improvements, fully implementing the new system as Common Core tests replace the old TCAP and EOC tests.

Doing so would require a step back from the rapid pace of recent reforms in the state. But the best way forward is not always the fastest. Tennessee would do well to emulate our neighbors, slow down, and focus on getting education reform right.

For more on Tennessee education politics and policy, follow us @TNEdReport

 

Big Monday Coming for McIntyre

On Monday, the Knox County School Board will discuss and possibly vote on a contract extension for embattled Director of Schools Jim McIntyre.

Last night, teachers, parents, and students packed the Board meeting room and some asked the Board not to renew McIntyre’s contract. It’s not clear from available news reports that anyone was present to ask the Board to extend the contract.

McIntyre has come under fire for being an enthusiastic supporter of state-level policy changes to teacher evaluation and for not listening to the concerns of parents and teachers regarding what they call excessive testing and over-reliance on test-based data to evaluate teachers.

That said, the Board recently announced they are working on a resolution calling for more transparency in the TVAAS system used to create scores for teacher evaluation.

Monday’s meeting, focused on the contract extension for McIntyre, will also likely be a contentious one, though it’s not clear whether a significant number of Board members would consider non-renewing the contract.

For more on Tennessee education politics and policy, follow us @TNEdReport