Would You Like Some Pie?

Chalkbeat’s Laura Faith Kebede reports on how the United Education Association, which represents teachers in Shelby County, describes the current TNReady situation:

“It’s as if you had a piece of pie, and I find a piece of glass in it,” she said. “But I cut somebody else a piece of that same pie and assure you ‘You don’t have glass in yours.’ Are you going to trust me and eat that piece of pie knowing that there’s a piece of glass in mine?”

That’s how UEA President Tikela Rucker described the current state of TNReady given what she cited as years of problems:

“This is the third year in a row that we’ve experienced issues regarding TNReady, which leads us to have zero confidence in TNReady, Commissioner McQueen and the Tennessee Department of Education,” said Shelby County UEA President Tikeila Rucker on behalf of the union’s 2,000 members.

Shelby County Director of Schools Dorsey Hopson also suggested the state needed to be more accountable:

“We stand in solidarity with our teachers. We know the (state) Department of Education is working very hard,” Hopson told reporters. “But given the high-stakes nature of the test, we just want to be accurate. And when they’re not accurate, it just casts a cloud of doubt over the whole process.”

Hopson stopped short of calling for test scores to be invalidated. “I wouldn’t necessarily jump to that conclusion,” he said, “but I do agree with our teaching colleagues that the results need to be accurate and timely.”

The UEA is calling for all scores from this year to be invalidated and for a moratorium on using TNReady scores in the state’s accountability system until 2021.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

 

Apples and Oranges

Here’s what Director of Schools Dorsey Hopson had to say amid reports that schools in his Shelby County district showed low growth according to recently released state test data:

Hopson acknowledged concerns over how the state compares results from “two very different tests which clearly are apples and oranges,” but he added that the district won’t use that as an excuse.

“Notwithstanding those questions, it’s the system upon which we’re evaluated on and judged,” he said.

State officials stand by TVAAS. They say drops in proficiency rates resulting from a harder test have no impact on the ability of teachers, schools and districts to earn strong TVAAS scores, since all students are experiencing the same change.

That’s all well and good, except when the system upon which you are evaluated is seriously flawed, it seems there’s an obligation to speak out and fight back.

Two years ago, ahead of what should have been the first year of TNReady, I wrote about the challenges of creating valid TVAAS scores while transitioning to a new test. TNReady was not just a different test, it was (is) a different type of test than the previous TCAP test. For example, it included constructed response questions instead of simply multiple choice bubble-in questions.

Here’s what I wrote:

Here’s the problem: There is no statistically valid way to predict expected growth on a new test based on the historic results of TCAP. First, the new test has (supposedly) not been fully designed. Second, the test is in a different format. It’s both computer-based and it contains constructed-response questions. That is, students must write-out answers and/or demonstrate their work.

Since Tennessee has never had a test like this, it’s impossible to predict growth at all. Not even with 10% confidence. Not with any confidence. It is the textbook definition of comparing apples to oranges.

Here’s a statement from the academic article I cited to support this claim:

Here’s what Lockwood and McCaffrey (2007) had to say in the Journal of Educational Measurement:

We find that the variation in estimated effects resulting from the different mathematics achievement measures is large relative to variation resulting from choices about model specification, and that the variation within teachers across achievement measures is larger than the variation across teachers.
You get different value-added results depending on the type of test you use. That is, you can’t just say this is a new test but we’ll compare peer groups from the old test and see what happens. Plus, TNReady presents the added challenge of not having been fully administered last year, so you’re now looking at data from two years ago and extrapolating to this year’s results.
Of course, the company paid millions to crunch the TVAAS numbers says that this transition presents no problem at all. Here’s what their technical document has to say about the matter:
In 2015-16, Tennessee implemented new End-of-Course (EOC) assessments in math and English/language arts. Redesigned assessments in Math and English/language arts were also implemented in grades 3-8 during the 2016-17 school year. Changes in testing regimes occur at regular intervals within any state, and these changes need not disrupt the continuity and use of value-added reporting by educators and policymakers. Based on twenty years of experience with providing valueadded and growth reporting to Tennessee educators, EVAAS has developed several ways to accommodate changes in testing regimes.
Prior to any value-added analyses with new tests, EVAAS verifies that the test’s scaling properties are suitable for such reporting. In addition to the criteria listed above, EVAAS verifies that the new test is related to the old test to ensure that the comparison from one year to the next is statistically reliable. Perfect correlation is not required, but there should be a strong relationship between the new test and old test. For example, a new Algebra I exam should be correlated to previous math scores in grades seven and eight and to a lesser extent other grades and subjects such as English/language arts and science. Once suitability of any new assessment has been confirmed, it is possible to use both the historical testing data and the new testing data to avoid any breaks or delays in value-added reporting.
A couple of problems with this. First, there was NO complete administration of a new testing regime in 2015-16. It didn’t happen.
Second, EVAAS doesn’t get paid if there’s not a way to generate these “growth scores” so it is in their interest to find some justification for comparing the two very different tests.
Third, researchers who study value-added modeling are highly skeptical of the reliability of comparisons between different types of tests when it comes to generating value-added scores. I noted Lockwood and McCaffrey (2007) above. Here are some more:
John Papay (2011) did a similar study using three different reading tests, with similar results. He stated his conclusion as follows: [T]he correlations between teacher value-added estimates derived from three separate reading tests — the state test, SRI [Scholastic Reading Inventory], and SAT [Stanford Achievement Test] — range from 0.15 to 0.58 across a wide range of model specifications. Although these correlations are moderately high, these assessments produce substantially different answers about individual teacher performance and do not rank individual teachers consistently. Even using the same test but varying the timing of the baseline and outcome measure introduces a great deal of instability to teacher rankings.
Two points worth noting here: First, different tests yield different value-added scores. Second, even using the same test but varying the timing can create instability in growth measures.
Then, there’s data from the Measures of Effective Teaching (MET) Project, which included data from Memphis. In terms of reliability when using value-added among different types of tests, here’s what MET reported:
Once more, the MET study offered corroborating evidence. The correlation between value-added scores based on two different mathematics tests given to the same students the same year was only .38. For 2 different reading tests, the correlation was .22 (the MET Project, 2010, pp. 23, 25).
Despite the claims of EVAAS, the academic research raises significant concerns about extrapolating results from different types of tests. In short, when you move to a different test, you get different value-added results. As I noted in 2015:

If you measure different skills, you get different results. That decreases (or eliminates) the reliability of those results. TNReady is measuring different skills in a different format than TCAP. It’s BOTH a different type of test AND a test on different standards. Any value-added comparison between the two tests is statistically suspect, at best. In the first year, such a comparison is invalid and unreliable. As more years of data become available, it may be possible to make some correlation between past TCAP results and TNReady scores.

Or, if the state is determined to use growth scores (and wants to use them with accuracy), they will wait several years and build completely new growth models based on TNReady alone. At least three years of data would be needed in order to build such a model.

Dorsey Hopson and other Directors of Schools should be pushing back aggressively. Educators should be outraged. After all, this unreliable data will be used as a portion of their teacher evaluations this year. Schools are being rated on a 1-5 scale based on a growth model grounded in suspect methods.

How much is this apple like last year’s orange? How much will this apple ever be like last year’s orange?

If we’re determined to use value-added modeling to measure school-wide growth or district performance, we should at least be determined to do it in a way that ensures valid, reliable results.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Doing the Right Thing

Shelby County’s Director of Schools Dorsey Hopson announced that all teachers will receive a three percent raise this year, not just those who meet certain scores on the state’s flawed value-added evaluation system.

More from Chalkbeat:

Hopson told the district’s educators in an email Thursday that they’ll see the raise reflected in their Nov. 18 paychecks. The pay hikes will be retroactive and will also go to librarians, counselors, instructional facilitators, coaches, social workers, physical/speech therapists and psychologists.

The decision came after Hopson learned that the district won’t receive the state’s testing data until December.

The decision by Hopson came about as a result of last year’s TNReady debacle. It also came in the same week that Knox County’s School Board asked the state for a waiver from included this year’s TNReady test results in student grades and teacher evaluations.

Hopson made the right decision — it is unfair to ask teachers to wait to receive pay raises because of the state’s mistakes with TNReady. It’s also unfair to use data from last year’s mess of a test administration to evaluate teachers. While I’ve expressed doubts about the usefulness of value-added data in evaluating teachers, even those who haven’t should acknowledge that using data from last year (or this year) is problematic.

Shelby County educators will all see a raise this year. The next question: Will the school board there join Knox County in requesting a waiver from using test data for students and teachers this year?

For more on education politics and policy in Tennessee, follow @TNEdReport


 

ASD Flexes Muscles in Memphis

Jon Alfuth over at Bluff City Ed has an update on the ASD and its actions in Memphis and in Nashville.

Alfuth’s report includes information on the teacher group organizing against ASD takevovers and concerns expressed by Shelby County Schools Superintenent Dorsey Hopson.

Most troubling is the notes about Chris Barbic’s remarks asserting the ASD could take all 85 schools on the priority list and noting that the ASD has been essentially playing nice up to this point.

Here’s the excerpt:

The most interesting and worrisome part of the last article for me are two quotes from Chris Barbic, the ASD’s Superintendent. First, Roberts quotes him as saying

“I think its important to remind everyone that a lot of things we are doing are by choice. If we wanted to, we could take over all 85 schools next year (bold added by me for emphasis).”

Second, he also states that they don’t have to do a matching process but that they have chosen to do so, and laments that they are being “beaten up” for what they are doing. He also says that they (presumably the community) “would be beating us up for not doing it (the matching process).”

These quotes trouble me because they perpetuate the message Memphians have been getting that “we (the ASD) have the power, and you should be thankful that we’re including you.” While I don’t know what else was said in the interview, it worries me that this is the type of rhetorical language that we’re seeing in the wake of a strong anti-ASD outpouring. Barbic does qualify his first quote by stating that they’ve chosen to work with SCS instead of pursing the total takeover course of action, but the lead sentence makes it clear who really has the decision making power.

In the end, the ASD opposition is all about who has the power to improve our schools, and quotes like this don’t do anything to alleviate the idea growing locally that the ASD is engaged in a hostile takeover of Memphis.

The entire update is worth a read.

I tend to agree with Jon that the challenge to the ASD is essentially about power. Parents, teachers, and even SCS leaders feel like they lack the power to make decisions about the schools.

For more on education politics and policy in Tennessee, follow @TNEdReport

Struggling ASD to Takeover 9 More Memphis Schools

The Memphis Commercial Appeal reports:

Nine more schools in Memphis will be taken over by the state-run Achievement School District next fall, including Wooddale Middle, Raleigh-Egypt High and South Side Middle, which have already been assigned to charter schools. Nine others — Florida-Kansas Elementary, Denver Elementary, Airways Middle, Brookmeade Elementary, American Way Middle, Hawkins Mills Elementary, LaRose Elementary, A. Maceo Walker Middle and A.B. Hill Elementary — are eligible for takeover, although only six of those will be under new management. The six to be taken over will be determined in part by a community vetting process that suggests which charter operators are best-suited to each school’s needs. The ASD will announce the final matches in December.

A recent analysis of the ASD’s performance indicates that the schools it has taken over in Shelby County would have been better off if they had remained in district hands. The ASD’s student achievement numbers have failed to meet their own ambitious targets and also failed to grow at a rate consistent with that of district schools.

An additional analysis compared schools in the iZone to ASD schools and found that the iZone model consistenly out-performs the ASD model.

When asked recently if he planned to convert iZone schools to charter schools, Shelby County Director of Schools Dorsey Hopson called the idea “absurd.”

Hopson’s statement is noteworthy because converting district schools to charters is exactly what the ASD plans to do.

While it may be fair to give the ASD more time to prove it can be effective with its existing schools, it seems irresponsible to allow the ASD to take on more schools. The leaders of the ASD would not be likely to put more students into a classroom of a teacher who failed to meet their desired student achievement targets. Why should more schools be handed over to a model that’s not only not living up to its own hype, but also failing to outperform the district schools it has taken over?

For more on education politics and policy in Tennessee, follow @TNEdReport

Hopson: Turning iZone over to Charters “Absurd”

Amid reports that the Shelby County Schools iZone may turn over some of its schools to charter operators due to financial concerns, Director of Schools Dorsey Hopson told the Memphis Daily News, “That’s absurd. I just want to be clear on that.”

Instead, Hopson indicated he plans to seek additional grants and/or private funding to continue the successful iZone efforts.

A recent analysis indicates that iZone schools are outperforming their Achievement School District counterparts. In short, the iZone is working. And Hopson’s comments acknowledge that while also making clear his commitment to find a way to stick with what’s working to help improve outcomes for students.

 

For more on education politics and policy in Tennessee, follow @TNEdReport