It All Comes Down to a Number

Dan Lawson is the Director of Schools for Tullahoma City Schools. He sent this message and the American Educational Research Association press release to a group of Tennessee lawmakers.

I am the superintendent of Tullahoma City Schools and in light of the media coverage associated with Representative Holt and a dialogue with teachers in west Tennessee I wanted to share a few thoughts with each of who represent teachers in other districts in Tennessee. I am thankful that each of you have a commitment to service and work to cultivate a great relationship with teachers and communities that you represent.

While it is certainly troubling that the standards taught are disconcerting in that developmental appropriateness is in question by many, and that the actual test administration may be a considerable challenge due to hardware, software and capacity concerns, I think one of the major issues has been overlooked and is one that could easily address many concerns and restore a sense of confidence in many of our teachers.

Earlier this week the American Educational Research Association released a statement (see below) cautioning states “against the use of VAM for high-stakes decisions regarding educators.” It seems to me that no matter what counsel I provide, what resources I bring to assist and how much I share our corporate school district priorities, we boil our work and worth as a teacher down to a number. And for many that number is a product of how well they guess on what a school-wide number could be since they don’t have a tested area.

Our teachers are tasked with a tremendous responsibility and our principals who provide direct supervision assign teachers to areas where they are most needed. The excessive reliance on production of a “teacher number” produces stress, a lack of confidence and a drive to first protect oneself rather than best educate the child. As an example, one of my principals joined me in meeting with an exceptional middle school math teacher, Trent Stout. Trent expressed great concerns about the order in which the standards were presented (grade level) and advised that our math department was confident that a different order would better serve our students developmentally and better prepare them for higher level math courses offered in our community. He went on to opine that while he thought we (and he) would take a “hit” on our eighth grade assessment it would serve our students better to adopt the proposed timeline. I agreed. It is important to note that I was able to dialogue with this professional out of a sense of joint respect and trust and with knowledge that his status with our district was solely controlled by local decision makers. He is a recipient of “old tenure.” However, don’t mishear me, I am not requesting the restoration of “old tenure,” simply a modification of the newly enacted statute. I propose that a great deal of confidence in “listening and valuing” teachers could be restored by amending the tenure statute to allow local control rather than state eligibility.

I have teachers in my employ with no test data who guess well and are eligible for the tenure status, while I have others who guess poorly and are not eligible. Certainly, the final decision to award tenure is a local one, but local based on state produced data that may be flawed or based on teachers other than the potential nominee. Furthermore, if we opine that tenure does indeed have value, I am absolutely lost when I attempt to explain to new teachers that if they are not eligible for tenure I may employ them for an unlimited number of added contracts but if they are eligible based on their number and our BOE decides that they will not award tenure to anyone I am compelled to non-renew those who may be highly effective teachers. The thought that statue allows me to reemploy a level 1 teacher while compelling me to non-renew a level 5 teacher seems more than a bit ironic and ridiculous.

I greatly appreciate your service to our state and our future and would love to see an extensive dialogue associated to the adoption of Common Sense.

The American Educational Research Association Statement on Value-Added Modeling:

In a statement released today, the American Educational Research Association (AERA) advises those using or considering use of value-added models (VAM) about the scientific and technical limitations of these measures for evaluating educators and programs that prepare teachers. The statement, approved by AERA Council, cautions against the use of VAM for high-stakes decisions regarding educators.

In recent years, many states and districts have attempted to use VAM to determine the contributions of educators, or the programs in which they were trained, to student learning outcomes, as captured by standardized student tests. The AERA statement speaks to the formidable statistical and methodological issues involved in isolating either the effects of educators or teacher preparation programs from a complex set of factors that shape student performance.

“This statement draws on the leading testing, statistical, and methodological expertise in the field of education research and related sciences, and on the highest standards that guide education research and its applications in policy and practice,” said AERA Executive Director Felice J. Levine.

The statement addresses the challenges facing the validity of inferences from VAM, as well as specifies eight technical requirements that must be met for the use of VAM to be accurate, reliable, and valid. It cautions that these requirements cannot be met in most evaluative contexts.

The statement notes that, while VAM may be superior to some other models of measuring teacher impacts on student learning outcomes, “it does not mean that they are ready for use in educator or program evaluation. There are potentially serious negative consequences in the context of evaluation that can result from the use of VAM based on incomplete or flawed data, as well as from the misinterpretation or misuse of the VAM results.”

The statement also notes that there are promising alternatives to VAM currently in use in the United States that merit attention, including the use of teacher observation data and peer assistance and review models that provide formative and summative assessments of teaching and honor teachers’ due process rights.

The statement concludes: “The value of high-quality, research-based evidence cannot be over-emphasized. Ultimately, only rigorously supported inferences about the quality and effectiveness of teachers, educational leaders, and preparation programs can contribute to improved student learning.” Thus, the statement also calls for substantial investment in research on VAM and on alternative methods and models of educator and educator preparation program evaluation.

The AERA Statement includes 8 technical requirements for the use of VAM:

  1. “VAM scores must only be derived from students’ scores on assessments that meet professional standards of reliability and validity for the purpose to be served…Relevant evidence should be reported in the documentation supporting the claims and proposed uses of VAM results, including evidence that the tests used are a valid measure of growth [emphasis added] by measuring the actual subject matter being taught and the full range of student achievement represented in teachers’ classrooms” (p. 3).
  2. “VAM scores must be accompanied by separate lines of evidence of reliability and validity that support each [and every] claim and interpretative argument” (p. 3).
  3. “VAM scores must be based on multiple years of data from sufficient numbers of students…[Related,] VAM scores should always be accompanied by estimates of uncertainty to guard against [simplistic] overinterpretation[s] of [simple] differences” (p. 3).
  4. “VAM scores must only be calculated from scores on tests that are comparable over time…[In addition,] VAM scores should generally not be employed across transitions [to new, albeit different tests over time]” (AERA Council, 2015, p. 3).
  5. “VAM scores must not be calculated in grades or for subjects where there are not standardized assessments that are accompanied by evidence of their reliability and validity…When standardized assessment data are not available across all grades (K–12) and subjects (e.g., health, social studies) in a state or district, alternative measures (e.g., locally developed assessments, proxy measures, observational ratings) are often employed in those grades and subjects to implement VAM. Such alternative assessments should not be used unless they are accompanied by evidence of reliability and validity as required by the AERA, APA, and NCME Standards for Educational and Psychological Testing” (p. 3).
  6. “VAM scores must never be used alone or in isolation in educator or program evaluation systems…Other measures of practice and student outcomes should always be integrated into judgments about overall teacher effectiveness” (p. 3).
  7. “Evaluation systems using VAM must include ongoing monitoring for technical quality and validity of use…Ongoing monitoring is essential to any educator evaluation program and especially important for those incorporating indicators based on VAM that have only recently been employed widely. If authorizing bodies mandate the use of VAM, they, together with the organizations that implement and report results, are responsible for conducting the ongoing evaluation of both intended and unintended consequences. The monitoring should be of sufficient scope and extent to provide evidence to document the technical quality of the VAM application and the validity of its use within a given evaluation system” (AERA Council, 2015, p. 3).
  8. “Evaluation reports and determinations based on VAM must include statistical estimates of error associated with student growth measures and any ratings or measures derived from them…There should be transparency with respect to VAM uses and the overall evaluation systems in which they are embedded. Reporting should include the rationale and methods used to estimate error and the precision associated with different VAM scores. Also, their reliability from year to year and course to course should be reported. Additionally, when cut scores or performance levels are established for the purpose of evaluative decisions, the methods used, as well as estimates of classification accuracy, should be documented and reported. Justification should [also] be provided for the inclusion of each indicator and the weight accorded to it in the evaluation process…Dissemination should [also] include accessible formats that are widely available to the public, as well as to professionals” ( p. 3-4).

The bottom line:  Tennessee’s use of TVAAS in teacher evaluations is highly problematic.

More on TVAAS:

Not Yet TNReady

The Worst Teachers

Validating the Invalid

More on Peer Assistance and Review:

Is PAR a Worthy Investment?

For more on education politics and policy in Tennessee, follow @TNEdReport

 

Testing Time

While Tennessee teachers are raising concerns about the amount of time spent on testing and test preparation, the Department of Education is lauding the new TNReady tests as an improvement for Tennessee students.

According to an AP story:

However, the survey of nearly 37,000 teachers showed 60 percent say they spend too much time helping students prepare for statewide exams, and seven out of ten believe their students spend too much time taking exams.

“What teachers recognize is the unfortunate fact that standardized testing is the only thing valued by the state,” said Jim Wrye, assistant executive director of the Tennessee Education Association, the state’s largest teachers’ union.

“Teachers and parents know there are so many things that affect future student success that are not measured by these tests, like social and emotional skills, cooperative behaviors, and academic abilities that do not lend themselves to be measured this way.”

Despite teacher concerns, the Department of Education says the new tests will be better indicators of student performance, noting that it will be harder for students to “game” the tests. That’s because the tests will include some open-ended questions.

What they don’t mention is that the company administering the tests, Measurement, Inc., is seeking test graders on Craigslist. And, according to a recent New York Times story, graders of tests like TNReady have, “…the possibility of small bonuses if they hit daily quality and volume targets.”  The more you grade, the more you earn, in other words.

Chalkbeat summarizes the move to TNReady like this:

The state was supposed to move in 2015 to the PARCC, a Common Core-aligned assessment shared by several states, but the legislature voted in 2014 to stick to its multiple-choice TCAP test while state education leaders searched for a test similar to the PARCC but designed exclusively for Tennessee students.

Except the test is not exactly exclusive to Tennessee.  That’s because Measurement, Inc. has a contract with AIR to use test questions already in use in Utah for tests in Florida, Arizona, and Tennessee.

And, for those concerned that students already spend too much time taking standardized tests, the DOE offers this reassurance about TNReady:

The estimated time for TNReady includes 25-50 percent more time per question than on the prior TCAP for English and math. This ensures that all students have plenty of time to answer each test question, while also keeping each TNReady test short enough to fit into a school’s regular daily schedule.

According to the schedule, the first phase of testing will start in February/March and the second phase in April/May. That means the tests are not only longer, but they also start earlier and consume more instructional time.

For teachers, that means it is critical to get as much curriculum covered as possible by February. This is because teachers are evaluated in part based on TVAAS — Tennessee Value-Added Assessment System — a particularly problematic statistical formula that purports to measure teacher impact on student learning.

So, if you want Tennessee students to spend more time preparing for and taking tests that will be graded by people recruited on Craigslist and paid bonuses based on how quickly they grade, TNReady is for you. And, you’re in luck, because testing time will start earlier than ever this year.

Interestingly, the opt-out movement hasn’t gotten much traction in Tennessee yet. TNReady may be just the catalyst it needs.

For more on education politics and policy in Tennessee, follow @TNEdReport

The Worst Teachers?

“There is a decently large percentage of teachers who are saying that they feel evaluation isn’t fair,” he (state data guru Nate Schwartz) said. “That’s something we need to think about in the process we use to evaluate teachers … and what we can do to make clear to teachers how this process works so they feel more secure about it.”

This from a story about the recently released 2015 Educator Survey regarding teacher attitudes in Tennessee.

One reason teachers might feel the evaluation is unfair is the continued push to align observation scores with TVAAS (Tennessee Value-Added Assessment System) data – data that purportedly captures student growth and thereby represents an indicator of teacher performance.

From WPLN:

Classroom observation scores calculated by principals should roughly line up with how a teacher’s students do on standardized tests. That’s what state education officials believe. But the numbers on the state’s five point scale don’t match up well.

“The gap between observation and individual growth largely exists because we see so few evaluators giving 1s or 2s on observation,” the report states.

“The goal is not perfect alignment,” Department of Education assistant commissioner Paul Fleming says, acknowledging that a teacher could be doing many of the right things at the front of the class and still not get the test results to show for it. But the two figures should be close.

In order to be better at aligning observation scores with TVAAS scores, principals could start by assigning lower scores to sixth and seventh grade teachers. At least, that’s what the findings of a study by Jessica Holloway-Libell published in June in the Teachers College Record suggest.

Holloway-Libell studied value-added scores assigned to individual schools in 10 Tennessee districts — Urban and suburban — and found:

In ELA in 2013, schools were, across the board, much more likely to receive positive value-added scores for ELA in fourth and eighth grades than in other grades (see Table 1). Simultaneously, districts struggled to yield positive value-added scores for their sixth and seventh grades in the same subject-areas. Fifth grade scores fell consistently in the middle range, while the third-grade scores varied across districts

Table 1. Percent of Schools that had Positive Value-Added Scores in English/language arts by Grade and District (2013) (Districts which had less than 25% of schools indicate positive growth are in bold)
District      Third      Fourth    Fifth     Sixth     Seventh      Eighth
Memphis      41%       43%        45%      19%        14%           76%
Nashville      NA        43%        28%      16%        15%           74%
Knox             72%       79%        47%      14%         7%            73%
Hamilton     38%      64%        48%      33%      29%            81%
Shelby           97%     76%         61%       6%        50%            69%
Sumner         77%     85%         42%       17%      33%            83%
Montgomery NA      71%         62%       0%        0%              71%
Rutherford     83%   92%         63%      15%     23%             85%
Williamson    NA      88%        58%      11%      33%           100%
Murfreesboro NA     90%        50%     30%     NA              NA

SOURCE: Teachers College Record, Date Published: June 08, 2015
http://www.tcrecord.org ID Number: 17987, Date Accessed: 7/27/2015

In examining three-year averages, Holloway-Libell found:

The three-year composite scores were similar except even more schools received positive value-added scores for the fifth and eighth grades. In fact, in each of the nine districts that had a composite score for eighth grade, at least 86% of their schools received positive value-added scores at the eighth-grade level.

By contrast, results in math were consistently positive across grade level and district type:

In particular, the fourth and seventh grade-level scores were consistently higher than those of the third, fifth, sixth, and eighth grades, which illustrated much greater variation across districts. The three-year composite scores were similar. In fact, a majority of schools across the state received positive value-added scores in mathematics across all grade levels.

So, what does this mean?

Well, it could mean that Tennessee’s 6th and 7th grade ELA teachers are the worst in the state. Or, it could mean that math teachers in Tennessee are better teachers than ELA teachers. Or, it could mean that 8th grade ELA teachers are rock stars.

Alternatively, one might suspect that the results of Holloway-Libell’s analysis suggest both grade level and subject matter bias in TVAAS.

In short, TVAAS is an unreliable predictor of teacher performance. Or, teaching 6th and 7th grade students reading is really hard.

Holloway-Libell’s findings are consistent with those of Lockwood and McCaffrey (2007) published in the Journal of Educational Measurement:

The researchers tested various VAM models and found that teacher effect estimates changed significantly based on both what was being measured AND how it was measured.

That is, it’s totally consistent with VAM to have different estimates for math and ELA teachers, for example. Math questions are often asked in a different manner than ELA questions and the assessment is covering different subject matter.

So, TVAAS is like other VAM models in this respect. Which means, as Lockwood and McCaffrey suggest, “caution is needed when interpreting estimated teacher effects” when using VAM models (like TVAAS).

In other words: TVAAS is not a reliable predictor of teacher performance.

Which begs the question: Why is the Tennessee Department of Education attempting to force correlation between observed teacher behavior and a flawed, unreliable measure of teacher performance? More importantly, why is such an unreliable measure being used to evaluate (and in some districts, reward with salary increases) teachers?

Don’t Tennessee’s students and parents deserve a teacher evaluation system that actually reveals strong teaching and provides support for teachers who need improvement?

Aren’t Tennessee’s teachers deserving of meaningful evaluation based on sound evidence instead of a system that is consistent only in its unreliability?

The American Statistical Association has said value-added models generally are unreliable as predictors of teacher performance. Now, there’s Tennessee-specific evidence that suggests strongly that TVAAS is biased, unreliable, and not effective as a predictor of teacher performance.

Unless, that is, you believe that 6th and 7th grade ELA teachers are our state’s worst.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

 

 

Can We Retire the Bad Teacher Narrative Already?

This article was submitted by Becca Leech, a Tennessee teacher with more than 20 years of experience. Read more about her here.

There’s an old story that goes something like this: American schools are in trouble. Our students lag so horribly behind the rest of the world academically that soon we won’t be able to keep pace. And why? Because our schools are full of bad teachers: lazy, mean, stupid teachers who don’t care about students and just want a cushy job where they have summers off. Unions protect these bad teachers so that schools can’t fire them and replace them with better ones, so our schools have become permeated with useless teachers who ruin the whole system and hold our students down.

I’ve taught special education in public schools since 1993, from inner-city Nashville to suburban Murfreesboro to rural McMinnville and, while every school has a few ineffective teachers, most of the teachers I have worked with have been smart, dedicated, and hard-working. These teachers continue to work every day to improve our schools, despite the demoralizing wounds of repeated volleys of the “bad teacher” narrative, so easily lobbed at us by the media, parents, students, and administrators. Last week, I even read it repeated by (of all people) a fellow special educator, in (of all places) this blog that I believed to be pro education – the Tennessee Education Report. Here is what Zack Barnes wrote:

“I am a special education teacher at a North Nashville middle school. Our fifth graders come into fifth grade already behind. It’s our job to catch them up during the middle school years before we send them off to high school. That shows me that we have dropped the ball along the way to middle school. We have come to a point where it’s okay that students come in to middle school behind. That shouldn’t be okay.

There are bad teachers and they should not be in the classroom. There isn’t more I can say about this. Every career field has bad workers, and the teaching profession is no different.”

This from a teacher who has been hired for the sole purpose of helping students with disabilities. When he accepted his position, was he unaware that students are only referred for our special education services when they have academic delays that cannot be addressed by the general curriculum? Was he unaware that student achievement, like all other human characteristics, spans a broad spectrum?

Our job as special education teachers is to accept the students referred to us – all of them, as they are – to assess their strengths and weaknesses, and to develop and implement appropriate educational plans to help them all learn. It’s not our job, and is counterproductive for our schools, to look for someone to blame for student delays. We don’t blame the persistence of illness in our communities on the bad doctors (although we know that some exist), so how do we find it so easy to blame all academic delays on bad teachers?

 

The Origins of the Narrative

Most of us were once students, and teachers were the face of the educational system to us. We experienced schools and teachers through immature eyes and often developed misperceptions of the roles and motivations of the teachers who taught us. Students who didn’t have good school experiences often caricaturized teachers as mean task-masters who didn’t like kids and just wanted to make them work hard or get them into trouble. Although it was often the structural problems with our educational system that we found frustrating or unfair as students – problems that were at least equally frustrating for our teachers – it was the teachers we saw as the cause. It’s easy for those who want to undermine public education for political or personal gain to play on these unconscious prejudices and transfer the problems with our educational structure directly into the laps of teachers as individuals.

 

In recent years, there has been a growing effort underway to undermine public education for just such purposes, and it is taking the form of a direct attack on teachers. The documentary Waiting for Superman brought the “bad teacher” narrative into our public consciousness. Politicians and news commentators have repeated the story as fact, ignoring all evidence offered against it. I have only recently begun to hear a backlash – teachers, parents, and students stepping forward to question the truth and usefulness of the tale.

 

How the “Bad Teacher” Narrative is Harming Education

The narrative of the bad teacher diminishes respect for the profession of teaching and gives ammunition to those working to decrease job security and protections for teachers. With less respect and job security, our schools have even more difficulty attracting and retaining qualified teachers. We have recently seen an increase in the number of experienced teachers leaving the profession and a decrease in applicants to new teacher education programs. In fact, many of these teachers and would-be teachers cite the lack of respect and poor job security as reasons for staying away from the classroom. When schools don’t have a pool of strong applicants to fill teaching vacancies, they must resort to hiring unqualified or less qualified candidates. This situation creates a self-fulfilling prophecy where students are more likely to fall behind, and critics find even more examples of “bad teachers” to further blame and diminish teaching.

 

Our Responsibility as a Community

It is true that there are some ineffective teachers among us and some teacher training programs that are not adequately preparing teachers for the classroom, but blame, judgement, and punitive accountability measures are not the answers. Most of the ineffective teachers I have taught with were either new teachers who needed more on-the-job guidance and mentoring from experienced teachers, or were teachers who had once been dedicated to their craft, but were now exhausted and weakened by the difficult environments they taught in and felt powerless to change.

 

There is no time in our school schedules to provide the support, mentoring, and quality training to help teachers in these situations to improve. Our school systems must find ways to make more non-teaching time in our school day for teachers to collaborate and support one another. Opportunities for teacher creativity, growth, and leadership will also raise teacher quality and improve working conditions.

 

To make real change, we must recognize that the problems with education are much broader than simply problems of teacher quality. We have to address poverty and inequality as the greatest challenges to education. We have to reorganize our funding structures to provide environments that attract and retain teachers. Most importantly, though, our communities must recognize teachers as the experts who can provide solutions for our schools – not the problems to be solved. If we are to improve the overall quality of education, we, as individuals and as a community, have a responsibility to support and defend the profession of teaching.

For more on education politics and policy in Tennessee, follow @TNEdReport

We welcome submissions from educators — if you have a story idea, send it to andy@spearsstrategy.com

 

 

 

A Little Less Bad

From a story in Chalkbeat:

Tennessee’s teacher evaluation system is more accurate than ever in measuring teacher quality…

That’s the conclusion drawn from a report on the state’s teacher evaluation system conducted by the State Department of Education.

The idea is that the system is improving.

Here’s the evidence the report uses to justify the claim of an improving evaluation system:

1) Teacher observation scores now more closely align with teacher TVAAS scores — TVAAS is the value-added modeling system used to determine a teacher’s impact on student growth

2) More teachers in untested subjects are now being evaluated using the portfolio system rather than TVAAS data from students they never taught

On the second item, I’d note that previously, 3 districts were using the a portfolio model and now 11 districts use it. This model allows related-arts teachers and those in other untested subjects to present a portfolio of student work to demonstrate that teacher’s impact on growth. The model is generally applauded by teachers who have a chance to use it.

However, there are 141 districts in Tennessee and 11 use this model. Part of the reason is the time it takes to assess portfolios well and another reason is the cost associated with having trained evaluators assess the portfolios. Since the state has not (yet) provided funding for the use of portfolios, it’s no surprise more districts haven’t adopted the model. If the state wants the evaluation model to really improve (and thereby improve teaching practice), they should support districts in their efforts to provide meaningful evaluation to teachers.

A portfolio system could work well for all teachers, by the way. The state could move to a system of project-based learning and thus provide a rich source of material for both evaluating student mastery of concepts AND teacher ability to impact student learning.

On to the issue of TVAAS and observation alignment. Here’s what the report noted:

Among the findings, state education leaders are touting the higher correlation between a teacher’s value-added score (TVAAS), which estimates how much teachers contribute to students’ growth on statewide assessments, and observation scores conducted primarily by administrators.

First, the purpose of using multiple measures of teacher performance is not to find perfect alignment, or even strong correlation, but to utilize multiple inputs to assess performance. Pushing for alignment suggests that the department is actually looking for a way to make TVAAS the central input driving teacher evaluation.

Advocates of this approach will tell suggest that student growth can be determined accurately by TVAAS and that TVAAS is a reliable predictor of teacher performance.

I would suggest that TVAAS, like most value-added models, is not a significant differentiator of teacher performance. I’ve written before about the need for caution when using value-added data to evaluate teachers.

More recently, I wrote about the problems inherent in attempting to assign growth scores when shifting to a new testing regime, as Tennessee will do next year when it moves from TCAP to TNReady. In short, it’s not possible to assign valid growth scores when comparing two entirely different tests.  Researchers at RAND noted:

We find that the variation in estimated effects resulting from the different mathematics achievement measures is large relative to variation resulting from choices about model specification, and that the variation within teachers across achievement measures is larger than the variation across teachers. These results suggest that conclusions about individual teachers’ performance based on value-added models can be sensitive to the ways in which student achievement is measured.
These findings align with similar findings by Martineau (2006) and Schmidt et al (2005)
You get different results depending on the type of question you’re measuring.

The researchers tested various VAM models (including the type used in TVAAS) and found that teacher effect estimates changed significantly based on both what was being measured AND how it was measured. 

And they concluded:

Our results provide a clear example that caution is needed when interpreting estimated teacher effects because there is the potential for teacher performance to depend on the skills that are measured by the achievement tests.

So, even if you buy the idea that TVAAS is a significant differentiator of teacher performance, drawing meaningful conclusions from next year’s TNReady simply is not reliable.

The state is touting improvement in a flawed system that may now be a little less bad.  And because they insist on estimating growth from two different tests with differing methodologies, the growth estimates in 2016 will be unreliable at best. If they wanted to improve the system, they would take two to three years to build growth data based on TNReady — that would mean two t0 three years of NO TVAAS data in teacher evaluation.

Alternatively, the state could move to a system of project-based learning and teacher evaluation and professional development based on a Peer Assistance and Review Model. Such an approach would be both student-centered and result in giving teachers the professional respect they deserve. It also carries a price tag — but our students are worth doing the work of both reallocating existing education dollars and finding new ways to invest in our schools.

For more on education politics and policy in Tennessee, follow @TNEdReport

 

 

 

Validating the Invalid?

The Tennessee House of Representatives passed legislation today (HB 108) that makes changes to current practice in teacher evaluation as Tennessee transitions to its new testing regime, TNReady.

The changes adjust the percentage of a teacher’s evaluation that is dependent on TVAAS scores to 10% next year, 20% the following year, and back to the current 35% by the 2017-18 academic year.

This plan is designed to allow for a transition period to the new TNReady tests which will include constructed-response questions and be aligned to the so-called Tennessee standards which match up with the Common Core State Standards.

Here’s the problem: There is no statistically valid way to predict expected growth on a new test based on the historic results of TCAP. First, the new test has (supposedly) not been fully designed. Second, the test is in a different format. It’s both computer-based and it contains constructed-response questions. That is, students must write-out answers and/or demonstrate their work.

Since Tennessee has never had a test like this, it’s impossible to predict growth at all. Not even with 10% confidence. Not with any confidence. It is the textbook definition of comparing apples to oranges.

Clearly, legislators feel like at the very least, this is an improvement. A reasonable accommodation to teachers as our state makes a transition.

But, how is using 10% of an invalid number a good thing? Should any part of a teacher’s evaluation be made up of a number that reveals nothing at all about that teacher’s performance?

While value-added data alone is a relatively poor predictor of teacher performance, the value-added estimate used next year is especially poor because it is not at all valid.

But, don’t just take my word for it. Researchers studying the validity of value-added measures asked whether value-added gave different results depending on the type of question asked. Particularly relevant now because Tennessee is shifting to a new test with different types of questions.

Here’s what Lockwood and McCaffrey (2007) had to say in the Journal of Educational Measurement:

We find that the variation in estimated effects resulting from the different mathematics achievement measures is large relative to variation resulting from choices about model specification, and that the variation within teachers across achievement measures is larger than the variation across teachers. These results suggest that conclusions about individual teachers’ performance based on value-added models can be sensitive to the ways in which student achievement is measured.
These findings align with similar findings by Martineau (2006) and Schmidt et al (2005)
You get different results depending on the type of question you’re measuring.

The researchers tested various VAM models (including the type used in TVAAS) and found that teacher effect estimates changed significantly based on both what was being measured AND how it was measured. 

And they concluded:

Our results provide a clear example that caution is needed when interpreting estimated teacher effects because there is the potential for teacher performance to depend on the skills that are measured by the achievement tests.

If you measure different skills, you get different results. That decreases (or eliminates) the reliability of those results. TNReady is measuring different skills in a different format than TCAP. It’s BOTH a different type of test AND a test on different standards. Any value-added comparison between the two tests is statistically suspect, at best. In the first year, such a comparison is invalid and unreliable. As more years of data become available, it may be possible to make some correlation between past TCAP results and TNReady scores.

Or, if the state is determined to use growth scores (and wants to use them with accuracy), they will wait several years and build completely new growth models based on TNReady alone. At least three years of data would be needed in order to build such a model.

It seems likely that the Senate will follow the House’s lead on Monday and overwhelmingly support the proposed evaluation changes. But in doing so, they should be asking themselves if it’s really ok to base any part of a teacher’s evaluation on numbers that reliably predict nothing.

More on Value-Added:

Real World Harms of Value-Added Data

Struggles with Value-Added Data

 

Is THAT even legal?

That’s the question the Tennessee Education Association is asking about the use of value-added data (TVAAS) in teacher evaluations.

The TEA, joining with the National Education Association, has filed a lawsuit challenging the constitutionality of Tennessee’s use of TVAAS data in teacher evaluations.

According to a press release, TEA is specifically concerned about teachers who receive value-added scores based on students they have never taught. A significant number of Tennessee teachers currently receive a portion of their evaluation score based on TVAAS scores from school-wide or other data, meaning teachers are graded based on students they’ve never taught.

The release states:

More than half of the public school teachers in Tennessee receive evaluations that are based substantially on standardized test scores of students in subjects they do not teach. The lawsuit seeks relief for those teachers from the arbitrary and irrational practice of measuring their effectiveness with statistical estimates based on standardized test scores from students they do not teach and may have never met. 

While Governor Haslam is proposing that the legislature reduce the impact of TVAAS scores on teacher evaluations during the state’s transition to new standardized tests, his proposal does not address the issues of statistical validity with the transition. There is no way to determine how TCAP scores will interface with the scores from a test that has not even been developed yet. To hold teachers accountable for data generated in such an unreliable fashion is not only statistically suspect, it’s disrespectful.

Finally, it’s worth noting that value-added data doesn’t do much in terms of differentiating teacher performance. Of course, even if it did, holding teachers accountable for students they don’t teach defies logic.

For more on education politics and policy in Tennessee, follow @TNEdReport

 

Dear Jim

Tomorrow, Knox County’s Director of Schools, Jim McIntyre, will testify before the Senate HELP Committee as part of ESEA reauthorization hearings being held by Sen. Lamar Alexander.

Ahead of his testimony, 9th District Knox County School Board member Amber Rountree sent McIntyre her thoughts on what he should say. This is her letter:

Dear Jim:
Thank you for the opportunity to give input on your upcoming testimony regarding the reauthorization of No Child Left Behind (“NCLB”).

As you stated in your email to the Board, you have been bestowed an honor to represent our students, our staff and the great state of Tennessee. I know you will share the wonderful innovation happening in Knox County Schools, but I implore you to provide a realistic picture of how NCLB (and its waiver) has impacted our schools.  I hope as you prepare your testimony you find courage to speak hard truths about the current state of our schools, including the following points:

More accountability≠better education. While we need a way to measure student progress, we must discontinue high-stakes testing that is not developmentally appropriate.  Punishing students, teachers and schools for results of these tests is simply unethical, especially while companies like Pearson profit from this punishment.

Restore local control.  Top down mandates from the federal government via NCLB have not led to a better outcome for students.  In fact, in our own district the achievement gap is widening.  Return the decision making to the hands of our state and local boards of education, along with controls to ensure punitive high-stakes testing does not continue.

Rethink the “Teacher Incentive Fund.”  Would you pay a firefighter based on the number of fires they successfully extinguished? Merit pay does not directly correlate to increased student performance.  A wiser choice would be to use the funding for smaller teacher-student ratios, which directly improve student outcomes.

Public dollars, public schools.  Vouchers and charters are a path to privatize public education.  When President Johnson signed ESEA into law, his intent was to help public schools succeed, not see those dollars funneled into private ventures which are not held to the same rigorous standards as public schools.
I concur with President Johnson’s remark that “there is no higher ground than a schoolroom or a more hopeful place than a classroom.”  The brightness of hope for our students and teachers has dimmed under the oppressive mandates of NCLB.  You’ve been given a gift to help restore that hope; my wish is that you use it wisely.
Yours in education,
Amber Rountree,  District 9 Representative

 

For more on education politics and policy in Tennessee, follow @TNEdReport

SPEAK Members Seek KCEA Posts

Lauren Hopson, whose remarks at a Knox County School Board meeting gained national attention, is seeking the Presidency of the Knox County Education Association. Hopson is joined in campaigning by Amy Cate, who is seeking the Vice Presidency, and Linda Holtzclaw, running for Secretary.

Hopson’s speech was the catalyst for a movement that become SPEAK: Students, Parents, and Educators Across Knox County.

The group speaks out on education issues and even recruited and supported some successful candidates in the recent school board election.

Hopson sought to draw attention to Knox County Schools policies that she believed harmed both teachers and their students. Now, SPEAK keeps Knox County citizens informed of relevant education issues and regularly engages local policymakers in discussions about how to improve Knox County Schools.

Here is the video that helped launch Hopson:

 

SPEAK Members Marching:

SPEAK members marching
For more on education politics and policy in Tennessee, follow @TNEdReport

PET Looks to 2015

A response to Governor Haslam’s recently announced teacher support initiatives by JC Bowman and Samantha Bates of Professional Educators of Tennessee (PET)

 

The announcement by Governor Bill Haslam addressing testing, evaluations, local control and teacher input was a much needed statement, as Tennessee is heading into the 2015 legislative session. Keeping in mind that each branch of government has a distinct and separate role, it is appropriate for Governor Haslam to identify changing priorities. As always, the key is in implementation of policies. Many policies sound good. They simply have to be executed correctly.

It is always good to step back and put some political philosophy behind the policy. However, the real message educators need to hear from elected leaders is that they are trusted. We need to start a fresh conversation on evaluating how we assess our educators, which may mean a change in the way we measure engagement.

When did test results became the be-all and end-all of our education experience? Is standardized testing so reliable that it has ended the search for something better to determine the quality of our education experience? And while numbers may help us understand our world, we recognize that they do not tell us the entire story.

Most local school districts understand that ability of their instructional personnel is the only real differentiator between them and other local districts. Therefore, it is imperative that we start treating our educators like one of our most important assets. And it is only common sense that one of the key items policymakers need to address in 2015 will be teacher salaries.

However, educators do not enter this field of public education for the income; they are there for the outcomes. If the perception within Tennessee is that teaching is not a celebrated profession, we certainly will not get the young talented people to pursue a career in public education as a profession.

We have steadfastly maintained that requiring school districts to simultaneously implement new standards, new teacher evaluations and perhaps a new curriculum, as well as new testing demands, will continue to place enormous pressure at the local level. More information and feedback on state assessments to help teachers improve student achievement is a welcome addition to the discussion. The use and/or overuse of testing remain a conversation worthy of public debate.

Tennessee will need to continue allocate resources devoted to the transition of standards. As we have argued, we believe it is time to move beyond the Common Core debate. We need to continuously build state specific standards that are challenging and meet the needs of Tennesseans. This needs to be done with legislative input and with the involvement of Tennessee educators.

The key item we took away from Governor Haslam’s latest proposal is his willingness to hear teacher concerns. It has taken us a long time to get to that point. However, it was a welcome relief to many educators, as we are now positioned to reset the dialogue. The area of improved teacher communication and collaboration has long been needed. We hope a new commissioner of education will truly embrace this concept.

If the right people are brought together for the right purpose, we believe anything is possible for Tennessee children and those who choose to educate our students. Dreaming big should not be just for the children in our classrooms, it should be for the stakeholders and policymakers in our state as well.

For more on education politics and policy in Tennessee, follow @TNEdReport