“No Confidence” in TNReady

Just days after members of the Knox County School Board took the Tennessee Department of Education to task for “incompetence” and an “abject failure” to measure student achievement or teacher performance, the Directors of the state’s two largest school districts, Nashville and Memphis, sent a letter to Education Commissioner Candice McQueen and Governor Bill Haslam stating they had “no confidence” in TNReady and asking the state to pause the test.

The letter, signed by Nashville’s Shawn Joseph and Shelby County’s Dorsey Hopson, says in part:

“We respectfully ask the State to hit the pause button on TNReady in order to allow the next Governor and Commissioner to convene a statewide working group of educators to sort out the myriad challenges in a statewide, collaborative conversation.”

The two leaders, whose districts represent 20 percent of all students in Tennessee, note:

“We are challenged to explain to teachers, parents, and students why they must accept the results of a test that has not been effectively deployed.”

The language from these two directors is the strongest yet from any district and the first to call for an outright stop to administration of the TNReady test while the state explores other options. Johnson City’s school board sent a proposal asking for a significant reduction in testing while Wilson County is exploring the possibility of administering a different test altogether. At the same time, Williamson County Director of Schools Mike Looney expressed concern about the poor administration of this year’s test.

It seems clear there is growing concern among educators about the continued use of TNReady. As Joseph and Hopson note, taxpayer resources have been invested in a test that is poorly implemented and yields suspect results. Taking their suggestion of a pause could give the state and a new Governor and Education Commissioner time to actually develop a process for administering an aligned assessment that does not disrupt instruction and does return useful, meaningful results to teachers, parents, and students.

Here’s the letter:

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Support TNEdReport!

Haslam’s TNReady Pit

Chalkbeat has a story demonstrating just how out of touch outgoing Governor Bill Haslam is. The story details Haslam’s belief that tying teacher evaluation to TNReady results is a key element in Tennessee’s recent education success.

Here’s some truth: Over the past few years, Tennessee has seen high school graduation rates and average ACT scores climb while also seeing the number of students requiring remediation at state schools decline. All of that is encouraging. All of it happened in a climate where the TNReady test was unreliable and poorly administered. In other words, Tennessee’s testing system had nothing to do with student performance. All other indicators point to teachers getting the job done and students hitting ever higher marks.

Here’s more truth:

Does basing teacher evaluation on student test scores get results that impact student outcomes?

No.

That’s the conclusion from a years-long study funded by the Gates Foundation that included Memphis/Shelby County Schools.

It’s also worth noting that while Haslam touts the “fastest-improving” NAEP results from back in 2013, further evidence suggests the results then were likely an outlier.

Here’s more from Chalkbeat:

Gov. Bill Haslam says he had a “pit” in his stomach every day of Tennessee’s testing season this spring when a parade of technical problems vexed students and teachers in the bumpy transition to computerized exams.

He also worries that three straight years of frustrations with the state’s 3-year-old standardized assessment, TNReady, could unravel policies that he believes led to students’ gains on national tests.

“Do we really want to go back? Do we really want to go back to when Tennessee was in the 40s out of the states ranked 1 to 50?”  the outgoing Republican governor asked recently in an exclusive interview with Chalkbeat.

First, no serious policymaker is suggesting Tennessee adopt weaker or lower standards for students.

Second, as noted above, other significant indicators demonstrate Tennessee students are improving — even without a reliable annual test.

Third, Haslam’s “beliefs” about policies have not been tested on a statewide level – in part due to the failure of his own Administration to execute the tests. Haslam has allowed Commissioner of Education Candice McQueen to keep her job despite multiple testing failures with different vendors. In fact, Haslam joined McQueen in touting a “new” testing vendor that turned out to actually be the parent company of the current vendor.

More from Chalkbeat:

“Hopefully Tennessee and the new administration won’t have the same struggles we’ve had this year with testing. But there will be some struggles; there just are by the very nature of it,” he said. “I worry that the struggles will cause us to say, ‘OK, we give. We’re no longer going to have an evaluation that’s tied to an assessment.’

To this, I’d note that experts suggest no state has had a more tumultuous transition to online testing than Tennessee:

“I’m not aware of a state that has had a more troubled transition” to online testing, said Douglas A. Levin of the consulting group EdTech Strategies.

In terms of an evaluation tied to an assessment, even if TNReady had gone well, the results in the initial years would not be in any way valid for use in teacher evaluation. That’s because the nature of value-added assessment requires multiple years of similar testing in order to produce results that are even vaguely reliable predictors of teacher performance. Here’s a bit more on that:

Here’s what Lockwood and McCaffrey (2007) had to say in the Journal of Educational Measurement:

We find that the variation in estimated effects resulting from the different mathematics achievement measures is large relative to variation resulting from choices about model specification, and that the variation within teachers across achievement measures is larger than the variation across teachers. These results suggest that conclusions about individual teachers’ performance based on value-added models can be sensitive to the ways in which student achievement is measured.
These findings align with similar findings by Martineau (2006) and Schmidt et al (2005)
You get different results depending on the type of question you’re measuring.

The researchers tested various VAM models (including the type used in TVAAS) and found that teacher effect estimates changed significantly based on both what was being measured AND how it was measured.

And they concluded:

Our results provide a clear example that caution is needed when interpreting estimated teacher effects because there is the potential for teacher performance to depend on the skills that are measured by the achievement tests.

If you measure different skills, you get different results. That decreases (or eliminates) the reliability of those results. TNReady is measuring different skills in a different format than TCAP. It’s BOTH a different type of test AND a test on different standards. Any value-added comparison between the two tests is statistically suspect, at best. In the first year, such a comparison is invalid and unreliable. As more years of data become available, it may be possible to make some correlation between past TCAP results and TNReady scores.

But, TNReady hasn’t gone well. At all. It’s been so bad, the Department of Education has been unveiling a bunch of pie charts to demonstrate how they are attempting to correlate test scores and teacher evaluation. First, it went like this:

Second, this chart is crazy. A teacher’s growth score is factored on tests from three different years and three types of tests.

15% of the growth score comes from the old TCAP (the test given in 2014-15, b/c the 2015-16 test had some problems). Then, 10% comes from last year’s TNReady, which was given on paper and pencil. Last year was the first year of a full administration of TNReady, and there were a few problems with the data calculation. A final 10% comes from this year’s TNReady, given online.

So, you have data from the old test, a skipped year, data from last year’s test (the first time TNReady had truly been administered), and data from this year’s messed up test.

There is no way this creates any kind of valid score related to teacher performance. At all.

After teachers expressed outrage that the DOE was going to count this year’s scores in their evaluations, the legislature finally took action and passed legislation that said teachers could face “no adverse action” based on this year’s test results.

So, now the Department of Education has more pie charts and a lot of explanations:

What is included in teacher evaluation generally?

There are many factors that go into a teacher’s overall evaluation. One of those, the individual growth component (in gray in the charts in this document), is typically based on a three-year TVAAS measure if data is available. However, for the phase-in period there are two key items to note for the growth component:

• If the current single-year year growth score – in this case, 2017-18 data – provides the educator with a higher overall composite, it will be used as the full growth score.

• Additionally, if a teacher has 2017-18 TNReady data included in any part of their evaluation, they will be able to nullify their entire LOE this year.

What is included in teacher evaluation in 2017-18 for a teacher with 3 years of TVAAS data?

There are three composite options for this teacher:

• Option 1: TVAAS data from 2017-18 will be factored in at 10%, TVAAS data from 2016-17 will be factored in at 10% and TVAAS data from 2015-16 will be factored in at 15% if it benefits the teacher.

• Option 2: TVAAS data from 2017-18 and 2016-17 will be factored in at 35%.

• Option 3: TVAAS data from 2017-18 will be factored in at 35%. The option that results in the highest LOE for the teacher will be automatically applied. Since 2017-18 TNReady data is included in this calculation, this teacher may nullify his or her entire LOE this year.

And if you only have one or two years of TVAAS data or if you teach in a non-tested subject? Well, the key line continues to apply: Since 2017-18 TNReady data is included in this calculation, this teacher may nullify his or her entire LOE this year.

What does this mean? Well, it means you’d have a year with no evaluation score. Sounds fine, right? No. It’s not fine. In order to achieve tenure, a teacher must have consecutive years of evaluation scores at Level 4 or 5. But a year with no score at all means that teacher would then need to have TWO MORE YEARS of high scores in order to be tenure eligible. While it seems unlikely a teacher would choose to nullify their entire score if they achieved a high rank, it also seems only fair to allow that teacher to simply exclude the TNReady data and receive their LOE rating based on all the other factors that go into a TEAM rating.

But wait, excluding 2017-18 TNReady data is NOT an option provided. It’s either count it as 10%, count it as 35%, or nullify your entire LOE score. Doing so could certainly have an adverse impact on a teacher.

In short, the TNReady mess has made teacher evaluation a mess. Still, a host of indicators suggest Tennessee’s teachers are hitting the mark. One might conclude that tying a suspect teacher evaluation model to an unreliable test is, in fact, not the key to educational progress in our state. Unfortunately, Governor Bill Haslam has concluded the opposite.

For more on education politics and policy in Tennessee, follow @TNEdReport

Help keep the education news coming!


 

Tempered Enthusiasm

Following last week’s release of TNReady results, Williamson County Director of Schools Mike Looney offered words of caution in interpreting the results.

The Williamson Herald has the story:

Looney said he was proud of how well WCS students, parents, teachers and staff responded to the testing in light of its documented flaws, and he was pleased with the fact that the district remained in the top five in every test and grade level.

“However,” he said in a statement released by WCS, “it would be disingenuous to fully celebrate without acknowledging the problems experienced by students, parents and teachers during last year’s testing process.”

While clearly frustrated with continued TNReady problems, Looney offered hope for a reliable assessment in the future:

“While I am so sorry that our students and teachers had to endure last year’s State testing experience, moving forward, we are optimistic that our students will be able to show what they know with a reliable and functional assessment. As a district, we will continue to be laser focused on success for all students.”

MORE on TNReady:

It’s all been a pack of lies

Beyond TNReady

Definitely something wrong

One glaring exception

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Meaningless Scores Make Big Splash

The Tennessee Department of Education today released results from this year’s troubled administration of TNReady testing.

This Tennessean story indicates Commissioner McQueen is taking the results seriously in spite of what has become an annual inability to get the job done right:

“We see reason to be encouraged, but we also have a lot of work to do to meet our higher expectations for all students,” Education Commissioner Candice McQueen said in a statement. “While we’ve focused extensively on early grades reading and are starting to see a shift in the right direction, we know middle school remains a statewide challenge across the board. TNReady serves as a vital feedback loop for teachers, parents, and administrators to tell us where we are, and the results inform what steps we need to take to help all students and schools succeed.”

Umm…what?? Are you even serious, Candice? THE TESTS DIDN’T WORK! Also, you were caught lying time and again about WHY they didn’t work.

The legislature passed legislation at the end of session, during testing to account for the failure of TNReady.

This isn’t even the first year the tests didn’t work. It happens EVERY year.

Here’s the deal: These scores can’t credibly be used to tell us ANYTHING. Students in schools all across the state faced disruptions caused by dump trucks and hacking (also known as the ineptitude of DOE leadership). Some kids got the wrong test and the extent of that problem is not totally known.

Note to district leaders: If you use these results to say your district or a given school are “doing well,” you have ZERO credibility. Treating these results as anything other than the complete trash that they are is unacceptable.

Some districts have already begun pushing for change in either how TNReady happens next year or even a totally different type of test.

Oh, and note that the DOE makes it sound like they are making improvements:

The Tennessee Department of Education also announced changes after this year’s problems, including searching for a new vendor. It also adjusted how fast it will phase-in online testing.

That sounds great, but the truth is, the DOE and Governor Haslam are treating Tennesseans like we’re stupid:

Let’s get this straight: Governor Haslam and Commissioner McQueen think no one in Tennessee understands Google? They are “firing” the company that messed up this year’s testing and hiring a new company that owns the old one and that also has a reputation for messing up statewide testing.

So, we had a testing season full of lies, deception, disruption, and mixed-up tests but we’re supposed to look at the “results” of those tests and take them seriously? No thanks.

 

For more on education politics and policy in Tennessee, follow @TNEdReport

Help keep the education news coming!


 

Beyond TNReady

At least one school system in Tennessee is taking steps to move beyond TNReady. According to a story in the Wilson Post, Wilson County Schools is seeking legislative action that would allow them to choose and administer their own annual tests in place of the state-mandated TNReady.

Here’s more:

Wilson County Director of Schools Dr. Donna Wright told county commissioners Monday the local school system is pursuing a private act from the state Legislature that would allow it to use an assessment other than the one currently mandated.

In her monthly report to the commission, Wright expressed her dissatisfaction with the TNReady test, saying that, “We are four years in without any or little actionable data that teachers can use.”

Wright added that while district leaders support accountability, the lack of timely, reliable data from the state tests is problematic:

“We are absolutely advocates of accountability because that’s how we know what to improve and where to improve,” Wright said adamantly. “But the fallacy in all this is that we haven’t had an effective system in four years, but we still keep using information that is not only in error, but late in coming.”

The action in Wilson County follows a resolution passed in Johnson City calling for a significant reduction in state-mandated testing.

The movement to reduce or replace TNReady follows yet another year of testing problems and a litany of excuses offered by the Department of Education and the state’s testing vendor.

Wright is correct that mishaps in testing and the late return of results call the usefulness of the data into question. However, even in the best of circumstances, it would be difficult to arrive at valid, actionable data based on the early years of a new test.

It will be interesting to see if other school systems follow the lead of Johnson City and Wilson County. Perhaps we’re finally seeing district leaders stand up and say “enough!”

For more on education politics and policy in Tennessee, follow @TNEdReport

Your support keeps the education news coming!


 

It’s All Been a Pack of Lies

By now, it should come as no surprise that our Commissioner of Education and the department she leads has a troubled relationship with the truth. That said, today’s revelation at a legislative hearing that an alleged hack of the state’s TNReady test didn’t actually happen again raises the question: Why does Candice McQueen still have a job?

Back on April 17th, the day after TNReady failed to work on day one of this year’s testing, the Tennessee Department of Education noted that the Day 2 failures were related to someone hacking the vendor:

At a legislative hearing today, the Tennessee Bureau of Investigation (TBI) indicated there was no evidence of a hack.

Additionally, the Department of Education issued this statement, which notes:

  • It appears, thankfully, that there was not an outside actor who attacked Questar’s data system. No student data was breached.
  • It is now clear that the event that Questar initially thought presented like a denial of service attack on Tuesday, April 17 was not created by an external actor with malicious intent, but, rather, can be traced in large part to the caching issues connected to how text-to-speech was configured by Questar.
  • Questar implemented a significant and unauthorized change to text-to-speech, which had previously operated successfully during the state’s fall administration. We now know this decision led to the severity of other issues we experienced during online testing.
  • Questar continues their internal investigation and is cooperating with additional external audits to make sure we have all of the facts.

Questar’s Chief Operating Officer Brad Baumgartner has provided this statement: “Questar’s internal and external investigations indicate that the source of the anomalous data pattern is believed to be the result of a configuration with the cache server. We have applied a configuration change and believe to have resolved the issue. We will continue to work with our internal technology team and external partners to validate this.”

The text-to-speech feature was also blamed for students receiving the wrong tests.

While at the time, the hacking excuse sounded pretty far-fetched, today’s hearing confirms that the Department advanced a lie offered by the state’s testing vendor. Of course, later on in the testing cycle, a dump truck was blamed for disrupting testing. That excuse was also later proven untrue.

All of this may explain why at least one school district is calling for a significant reduction in TNReady testing next year.

If this year had been the first time our state had faced testing challenges, one might understand (and forgive) the excuse-making. However, this is now the fifth consecutive year of some sort of problem and the fourth year testing administration has been, to say the least, a challenge.

One may recall the saga of Measurement, Inc. The company that hired test graders from Craigslist and was ultimately fired in 2016 after that year’s TNReady test failed.

The bottom line: If TNEdu tells you something about testing, you should question it. The track record shows that to our state’s Department of Education, truth is a relative concept.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Your support keeps the education news coming!

If this is what success looks like…

In a story about the Tennessee Department of Education scaling back the requirements for online testing next year in light of this year’s testing challenges, this caught my attention:

Even with the problems this year, it was one of the most successful online administrations for the state to date. More than 2.5 million TNReady tests were administered this spring, with about 300,000 students taking the test online. Only high school students were required to take the online version this year.

What does the word “success” mean? Because my recollection of this year’s TNReady administration is that it was a debacle.

I’m not the only one. As I noted last week:

While lots of states are moving to online testing, one expert says Tennessee is unique:

“I’m not aware of a state that has had a more troubled transition” to online testing, said Douglas A. Levin of the consulting group EdTech Strategies.

And there’s this helpful explainer:

Why is Tennessee in the unique position of having the worst online testing transition in the country?

The reality is that Tennessee’s online-testing mess has left everyone in a difficult position, said Chad Aldeman, a principal at Bellwether Education Partners, a consulting organization.

“The state has not [made] stability a key priority in their testing vendors,” Aldeman said.

Nevertheless, Commissioner of Education Candice McQueen says:

The state will put out a request for contract proposals in the fall, with a new vendor to be identified in the spring. Questar Assessment could again win the contract, but McQueen said who wins the proposal will have to show the ability and history of seamlessly administering an online test.

“We look for a company with a track record of success in administering online testing and who can manage our test well.”

Haven’t we heard that before?

For more on education politics and policy, follow @TNEdReport

Keep the education news coming!


 

Definitely Something Wrong

The Texas Tribune reports:

A couple of weeks after Texas penalized its main testing vendor over glitches with thousands of standardized tests, another potential testing mishap is under investigation after more than 100 students in a high-performing Houston-area high school received zeros on their English essays.

Valerie Vogt, chief academic officer at Lamar Consolidated Independent School District, said she was confused this spring when about 157 students at George Ranch High School, which generally performs higher than state average on standardized tests, received zeros on their English 1 and English 2 essays. In the other four high schools in the district, just 10 or fewer students received zeros on the essays.

“There’s definitely something wrong,” she said.

The testing vendor responsible is Educational Testing Services (ETS), which owns Tennessee’s testing vendor, Questar. This is the latest in a series of problems with ETS in Texas:

Last month, the TEA levied a $100,000 penalty against ETS after tens of thousands of Texas students were kicked out of the testing software or encountered connection problems when taking computerized tests in April and May. The agency also announced it would throw out the scores of students who experienced those glitches and reduce their effect on state accountability ratings for schools and districts.

Tennessee’s Department of Education announced recently ETS would be taking over more responsibility for TNReady after Questar’s administration of the testing this year was plagued by hackers and dump trucks.

Of course, ETS is not without a history of test administration problems. Edsurge.com notes:

The changes highlight a possible strategic shift for ETS whose reputation came under fire last year when the nonprofit had to pay $20.7 million dollars in damages and upgrades after multiple testing problems in Texas.

A recent analysis of the transition to online testing in the states indicates it is going well in most places, with Tennessee being the one glaring exception.

So, of course Tennessee hires the parent company of Questar — a company that has experienced consecutive years of testing problems in Texas — to come in and … make things right?

Yep, there’s definitely something wrong.

For more on education politics and policy in Tennessee, follow @TNEdReport

Your support keeps the education news coming!


 

Shrinking TNReady

That’s the hope behind a resolution passed by the Johnson City Board of Education this week.

The Johnson City Press reports:

Some changes included shrinking testing timing back from three weeks to one week for grades 3 through 8, pushing the writing assessment back to February to give the state more time to get grades in by the end of the school year, and drawing back on pre-K and kindergarten ELA assessments to be less time-consuming for teachers.

The proposal comes after another year of testing trouble in Tennessee. In fact, a recent report noted that while most states transitioning to online testing are doing so smooth, Tennessee is the one glaring exception.

Broad Support?

Now that the Johnson City School Board has given unanimous approval to this proposal, the Director of Schools hopes to spread the message to other districts and build support for changing TNReady:

What I’d like to do if the board approves this resolution is reach out to all the other school superintendents and talk to them about the resolution and get feedback from them,” Barnett said at the meeting. “I think we’d have some support.”

It’s possible this is the beginning of a move that will see district leaders stand up to the state and say “Enough!”

The Board also referenced the problematic implementation of portfolios to evaluate teachers in Pre-K/Kindergarten:

Anderson said that the state estimated those assessments would take about 15 to 17 hours, but some teachers reported spending as many as 44 hours on the project, most of that time being spent in the English Language Arts component of the assessment.

She added that portfolio assessment is considered an appropriate avenue to track student learning in those early grades, and the portfolios can be completed with video or audio taping or with written assessment.

“I don’t think anybody has anything against the concept of portfolios for pre-K (and kindergarten),” she said. “Though the piloting process went fairly well, it ended up morphing into a process this past year that I think was just very complicated and very unwieldy.

It will be interesting to see how the state moves forward in revising those portfolios and if there is any move toward making significant change in the TNReady tests.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

One Glaring Exception

That’s how this article in Education Week defines the TNReady testing experience.

It starts like this: Just about everything that could go wrong did go wrong with online state testing this year in Tennessee.

Yep.

The piece walks through the saga that has been TNReady. Here are some highlights:

Then, thanks to human error at some schools, about 1,400 students ended up taking the wrong version of the TNReady exam

Except it wasn’t human error at the schools. As I reported on April 26th, the Department of Education said about the issue:

 

“There was a poorly designed feature of the online testing system that contributed to some users accidentally administering a test to students that was below their grade level, including those at Norris Middle School. We’ve provided guidance to the district staff and the building testing coordinator to invalidate these tests. Students are not required to re-test, and their tests will not be scored.

Then, again with the dump truck:

And a rogue dump truck severed one of the state’s main fiber-optic cables, causing temporary connectivity problems during the testing period.

Except not really:

“There is no evidence this was anything other than a side effect of the issue with the fiber cut, but we continue to look into it,” Sara Gast, a spokeswoman for the state education department, said last week.

But internet provider Education Networks of America disputes that, saying that the West Tennessee issues were not related to the cable cut.

What happened in those cases remains a mystery, for now.

Unanticipated?

The article says:

On the second day of testing, Questar was flooded with unanticipated traffic that overwhelmed the company’s servers and prevented some students from connecting to the TNReady testing platform.

How was the testing traffic unanticipated? Was Questar counting on a bunch of students missing school on the second day of testing? Did they not know how many students would be logging on ant the relative times that would happen? They were paid $30 million to figure that out… and didn’t.

While lots of states are moving to online testing, one expert says Tennessee is unique:

“I’m not aware of a state that has had a more troubled transition” to online testing, said Douglas A. Levin of the consulting group EdTech Strategies.

 

Why is Tennessee in the unique position of having the worst online testing transition in the country?

The reality is that Tennessee’s online-testing mess has left everyone in a difficult position, said Chad Aldeman, a principal at Bellwether Education Partners, a consulting organization.

“The state has not [made] stability a key priority in their testing vendors,” Aldeman said.

 

Ultimately, responsibility should rest at the feet of Education Commissioner Candice McQueen, who so far has avoided any accountability for the ongoing testing mishaps in the state.

For more on education politics and policy in Tennessee, follow @TNEdReport

Help keep the education news coming!