Muddy Waters

Laura Faith Kebede of Chalkbeat reports on the challenges in generating reliable TVAAS scores as a result of TNReady trouble last year. Her story cites a statistician from the Center for Assessment who explains the issue this way:

Damian Betebenner, a senior associate at Center for Assessment that regularly consults with state departments, said missing data on top of a testing transition “muddies the water” on results.

“When you look at growth over two years, so how much the student grew from third to fifth grade, then it’s probably going to be a meaningful quantity,” he said. “But to then assert that it isolates the school contribution becomes a pretty tenuous assertion… It adds another thing that’s changing underneath the scene.”

In other words, it’s difficult to get a meaningful result given the current state of testing in Tennessee. I wrote recently about this very issue and the problem with the validity of the growth scores this year.

Additionally, two years ago, I pointed out the challenges the state would face when shifting to a new test. Keep in mind, this was before all the TNReady trouble that further muddied the waters. Here’s what I said in March of 2015:

Here’s the problem: There is no statistically valid way to predict expected growth on a new test based on the historic results of TCAP. First, the new test has (supposedly) not been fully designed. Second, the test is in a different format. It’s both computer-based and it contains constructed-response questions. That is, students must write-out answers and/or demonstrate their work.

Since Tennessee has never had a test like this, it’s impossible to predict growth at all. Not even with 10% confidence. Not with any confidence. It is the textbook definition of comparing apples to oranges.

The way to address this issue? Build multiple years of data in order to obtain reliable results:

If you measure different skills, you get different results. That decreases (or eliminates) the reliability of those results. TNReady is measuring different skills in a different format than TCAP. It’s BOTH a different type of test AND a test on different standards. Any value-added comparison between the two tests is statistically suspect, at best. In the first year, such a comparison is invalid and unreliable. As more years of data become available, it may be possible to make some correlation between past TCAP results and TNReady scores.

So, now we have two challenges: We have two different types of tests AND we have a missing year of data. Either one of these challenges creates statistical problems. The combination of the two calls for a serious reset of the state’s approach to accountability.

As I suggested yesterday, taking the time to get this right would mean not using the TNReady data for accountability for teachers, students, or schools until 2019 at the earliest. If our state is committed to TNReady, we should be committed to getting it right. We’re spending a lot of money on both TNReady and on TVAAS. If we’re going to invest in these approaches, we should also take the time to be sure that investment yields useful, reliable information.

Why does any of this matter? Because, as Kebede points out:

At the same time, TVAAS scores for struggling schools will be a significant factor to determine which improvement tracks they will be be placed on under the state’s new accountability system as outlined in its plan to comply with the federal Every Student Succeeds Act. For some schools, their TVAAS score will be the difference between continuing under a local intervention model or being eligible to enter the state-run Achievement School District. The school growth scores will also determine which charter schools are eligible for a new pot of state money for facilities.

TVAAS scores also count in teacher evaluations. TNReady scores were expected to count in student grades until the quick scores weren’t back in time. If all goes well with the online administration of TNReady this year, the scores will count for students.

The state says TNReady matters. The state evaluates schools based on TVAAS scores. The state teacher evaluation formula includes TVAAS scores for teachers and TNReady scores as one measure of achievement that can be selected.

In short: Getting this right matters.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Chattanooga in Talks for Expansion Team

The state’s fourth-largest school district will soon be in talks to become the next location of an expansion franchise in the school takeover league known as the Achievement School District (ASD). With 33 schools under its control, the ASD is considered the major league in the school takeover world.

Laura Faith Kebede reports on this development at Chalkbeat:

Leaders of the Achievement School District will begin talks with district and community leaders in Hamilton County in the coming months, according to Robert S. White, the ASD’s chief of external affairs.

The development comes despite a lack of current data due to the failure of this year’s administration of TNReady.

League leaders say a lack of data won’t slow them down as they aggressively pursue expansion in 2018 and beyond:

The ASD’s next steps have been made more challenging by the lack of test score data across Tennessee due the state’s late-spring cancellation of most of its TNReady tests. But after the hiatus year, White said he expects the state-run district to continue to take control of priority schools, even as the state rolls out a new assessment by a new test maker this coming year.

“You won’t see that two years in a row,” he said of the takeover hiatus.

The league also didn’t rule out an expansion in Nashville, where a contentious battle in 2014 resulted in Neely’s Bend Middle School “winning” ASD franchise status.

Despite an initial plan focused on stellar turnarounds of struggling schools, the ASD has a reputation for taking low-performing schools, handing them over to charter-operator general managers, and watching as the results are rarely better than under previous management teams.

The Memphis franchise(s) have been plagued with unrest from “fans” expecting the league to live up to its promises.

In fact, one local group has asked the league to stop looking to Memphis for new expansion teams. Due to what it considers market saturation, that’s a request the league is likely to honor in the short term.

Meanwhile, Chattanooga awaits word regarding which eligible school(s) could get the call from the school takeover major league.

For more on education politics and policy in Tennessee, follow @TNEdReport