Battle Lines Being Drawn

Last week, the School Superintendents in Memphis and Nashville wrote a letter to Commissioner of Education Candice McQueen calling for a pause in TNReady. The letter indicated the leaders had “no confidence” in TNReady. Following the letter, the Knox County School Board voted 8-1 to send a letter to Governor Haslam stating they had “no confidence” in the Department of Education. Later that week, the Director of Schools in Maury County said he agreed with the idea of pausing TNReady and suggested moving to the ACT suite of assessments.

Today, Commissioner McQueen issued a response. According to Chalkbeat, her response indicates that pausing TNReady would be “illegal and inconsistent with our values as a state.”

McQueen cites federal and state law requiring test administration. Here’s the deal: The entity that determines the penalty for violating state law regarding testing is the Department of Education. The penalty they can use is withholding BEP funds. This is the threat they used back in 2016 to force districts to back down on threats to halt testing then.

Let’s be clear: The Tennessee Department of Education is the enforcer of the state testing mandate. The DOE could refuse to penalize districts who paused testing OR the DOE could take the suggestion made by Dorsey Hopson of Memphis and Shawn Joseph of Nashville and just hit the pause button for this year and work toward an effective administration of testing for 2019-20.

Next, McQueen cites federal law. I’ve written about why this is misguided. Here’s more:

There’s just one problem: The federal government has not (yet) penalized a single district for failing to hit the 95% benchmark. In fact, in the face of significant opt-outs in New York last year (including one district where 89% of students opted-out), the U.S. Department of Education communicated a clear message to New York state education leaders:  Districts and states will not suffer a loss of federal dollars due to high test refusal rates. The USDOE left it up to New York to decide whether or not to penalize districts financially.

That’s right, the federal government tends to leave decisions regarding punishment up to the states. Of course, Tennessee could also request a 1-year waiver of ESSA requirements in order to further clarify the need to get testing right. In short, the only problem now is McQueen’s unwillingness to admit failure and take aggressive steps to make improvement.

McQueen also says that halting testing is “inconsistent” with Tennessee values.

While in McQueen’s world, halting testing is inconsistent with our state’s values, lying about why testing isn’t working is apparently perfectly fine. Oh, and playing a game with testing vendors? No problem!

McQueen claims that we need the tests to help identify gaps in education delivery in traditionally under-served students. Yes, having a working annual assessment can be a helpful tool in identifying those gaps. But, when the test doesn’t work — when students get the wrong test, when the testing climate is not consistent — then we get results that are unreliable. That helps no one.

What should be consistent with Tennessee values is taking the time to get testing right. That means ensuring it’s not disruptive to the instruction process and the results are useful and returned to students, teachers, and parents in a timely fashion.

Will McQueen’s letter deter other district leaders from speaking out on TNReady? Will there be additional fallout from the DOE’s failure to effectively administer Pre-K/K portfolios?

Stay tuned.

 

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Blood in the Water

The Director of Schools in Maury County has joined those in Memphis and Nashville in calling for a pause in TNReady as a result of repeated problems with the testing platform.

The Columbia Daily Herald reports Maury County Director of Schools Chris Marczak said he agrees with the letter sent by Dorsey Hopson of Memphis and Shawn Joseph of Nashville. Marczak offered an alternative:

“I believe it would be best for us to focus solely on the ACT and align ourselves with outcomes that can affect students’ college acceptance and scholarship ability,” Marczak said.

Maury County district leadership has indicated the results from this year’s botched test administration are of limited value:

“Due to the issues with testing, we will not be adding TNReady/EOC data to the Keys’ scorecards for either the district or the school levels when they eventually come in,” Marczak said in an email sent to staff in July. “In light of the numerous testing issues, please know that the results of the assessments will be used to inform conversation only. These are the conversations we will have with principals and the principals will have with teachers/staffs.”

In response to the ongoing testing issues, Marczak shared accounts of students completing 75-minute long examinations in 10 minutes. When reviewing the examinations, Marczak said the district had over 600 missing scores. Questar, the contractor hired by the state to administer the test, reported that 600 individual assessments were incomplete.

Despite the growing concern over the inability to effectively administer the TNReady test, Commissioner of Education Candice McQueen has said the test is still an important tool:

TNReady serves as a vital feedback loop for teachers, parents, and administrators to tell us where we are, and the results inform what steps we need to take to help all students and schools succeed.”

TNReady might be an important feedback loop if it ever worked the way it was intended. But it hasn’t. Instead, it’s been fraught with problems since the beginning. Now, education leaders are standing up and speaking out.

The push to pause TNReady and possibly move forward with a different measure comes at the same time the TDOE is being taken to task for a failure to properly execute Pre-K/Kindergarten portfolios. Knox County’s School Board last night voted to send a message that they have “no confidence” in the portfolio process or in the TDOE.

The push against TNReady from key district leaders figures to make the test and overall administration of the Department of Education a key issue in the 2018 gubernatorial and state legislative elections.

 

 

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Stand Up, Fight Back

Just days after the state’s two largest school districts sent a letter to Governor Haslam and Education Commissioner Candice McQueen expressing “no confidence in TNReady, the school board in the third largest district (Knox County) voted 8-1 to have their Director of Schools send a letter expressing “no confidence” in the Pre-K/Kindergarten portfolio process and in the Tennessee Department of Education as a whole.

The move comes after a study session last week in which board members characterized the TN DOE’s administration of portfolios and of teacher evaluation as an “abject failure.”

While the DOE blames the problems with scores on this year’s Pre-K/K portfolios on teachers, individual teachers continue to provide evidence they followed every instruction and guideline from DOE and yet still faced sections of their portfolio submissions that were not scored at all. When a section was not scored, teachers saw their score for that section default to a “1”, the lowest possible score.

I’ve reported before on the discrepancies between rubrics provided to teachers and those provided reviewers. Reviewers received rubrics reflecting more difficult standards, meaning teachers who complied with the rubrics they were given likely lost ground in the final scoring.

I’ve since talked with teachers who indicated they received scores of “5” on three sections and a score of “1” on another. While this created a composite score of “4,” it’s not a logical outcome. It’s highly unlikely that a teacher who receives the top score in three categories would then receive the lowest possible score on the fourth.

As I learn more about this issue, it seems clear that many teachers had submissions that simply weren’t scored at all. The problems in May and June with submission review indicate the state was ill-prepared to execute the scoring of this year’s portfolios.

Now, the TDOE faces significant criticism from the state’s three largest districts in terms of how it handles both student assessment and teacher evaluation. It will be interesting to see if additional districts follow suit.

 

For more on education politics and policy in Tennessee, follow @TNEdReport


 

“No Confidence” in TNReady

Just days after members of the Knox County School Board took the Tennessee Department of Education to task for “incompetence” and an “abject failure” to measure student achievement or teacher performance, the Directors of the state’s two largest school districts, Nashville and Memphis, sent a letter to Education Commissioner Candice McQueen and Governor Bill Haslam stating they had “no confidence” in TNReady and asking the state to pause the test.

The letter, signed by Nashville’s Shawn Joseph and Shelby County’s Dorsey Hopson, says in part:

“We respectfully ask the State to hit the pause button on TNReady in order to allow the next Governor and Commissioner to convene a statewide working group of educators to sort out the myriad challenges in a statewide, collaborative conversation.”

The two leaders, whose districts represent 20 percent of all students in Tennessee, note:

“We are challenged to explain to teachers, parents, and students why they must accept the results of a test that has not been effectively deployed.”

The language from these two directors is the strongest yet from any district and the first to call for an outright stop to administration of the TNReady test while the state explores other options. Johnson City’s school board sent a proposal asking for a significant reduction in testing while Wilson County is exploring the possibility of administering a different test altogether. At the same time, Williamson County Director of Schools Mike Looney expressed concern about the poor administration of this year’s test.

It seems clear there is growing concern among educators about the continued use of TNReady. As Joseph and Hopson note, taxpayer resources have been invested in a test that is poorly implemented and yields suspect results. Taking their suggestion of a pause could give the state and a new Governor and Education Commissioner time to actually develop a process for administering an aligned assessment that does not disrupt instruction and does return useful, meaningful results to teachers, parents, and students.

Here’s the letter:

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Support TNEdReport!

Haslam’s TNReady Pit

Chalkbeat has a story demonstrating just how out of touch outgoing Governor Bill Haslam is. The story details Haslam’s belief that tying teacher evaluation to TNReady results is a key element in Tennessee’s recent education success.

Here’s some truth: Over the past few years, Tennessee has seen high school graduation rates and average ACT scores climb while also seeing the number of students requiring remediation at state schools decline. All of that is encouraging. All of it happened in a climate where the TNReady test was unreliable and poorly administered. In other words, Tennessee’s testing system had nothing to do with student performance. All other indicators point to teachers getting the job done and students hitting ever higher marks.

Here’s more truth:

Does basing teacher evaluation on student test scores get results that impact student outcomes?

No.

That’s the conclusion from a years-long study funded by the Gates Foundation that included Memphis/Shelby County Schools.

It’s also worth noting that while Haslam touts the “fastest-improving” NAEP results from back in 2013, further evidence suggests the results then were likely an outlier.

Here’s more from Chalkbeat:

Gov. Bill Haslam says he had a “pit” in his stomach every day of Tennessee’s testing season this spring when a parade of technical problems vexed students and teachers in the bumpy transition to computerized exams.

He also worries that three straight years of frustrations with the state’s 3-year-old standardized assessment, TNReady, could unravel policies that he believes led to students’ gains on national tests.

“Do we really want to go back? Do we really want to go back to when Tennessee was in the 40s out of the states ranked 1 to 50?”  the outgoing Republican governor asked recently in an exclusive interview with Chalkbeat.

First, no serious policymaker is suggesting Tennessee adopt weaker or lower standards for students.

Second, as noted above, other significant indicators demonstrate Tennessee students are improving — even without a reliable annual test.

Third, Haslam’s “beliefs” about policies have not been tested on a statewide level – in part due to the failure of his own Administration to execute the tests. Haslam has allowed Commissioner of Education Candice McQueen to keep her job despite multiple testing failures with different vendors. In fact, Haslam joined McQueen in touting a “new” testing vendor that turned out to actually be the parent company of the current vendor.

More from Chalkbeat:

“Hopefully Tennessee and the new administration won’t have the same struggles we’ve had this year with testing. But there will be some struggles; there just are by the very nature of it,” he said. “I worry that the struggles will cause us to say, ‘OK, we give. We’re no longer going to have an evaluation that’s tied to an assessment.’

To this, I’d note that experts suggest no state has had a more tumultuous transition to online testing than Tennessee:

“I’m not aware of a state that has had a more troubled transition” to online testing, said Douglas A. Levin of the consulting group EdTech Strategies.

In terms of an evaluation tied to an assessment, even if TNReady had gone well, the results in the initial years would not be in any way valid for use in teacher evaluation. That’s because the nature of value-added assessment requires multiple years of similar testing in order to produce results that are even vaguely reliable predictors of teacher performance. Here’s a bit more on that:

Here’s what Lockwood and McCaffrey (2007) had to say in the Journal of Educational Measurement:

We find that the variation in estimated effects resulting from the different mathematics achievement measures is large relative to variation resulting from choices about model specification, and that the variation within teachers across achievement measures is larger than the variation across teachers. These results suggest that conclusions about individual teachers’ performance based on value-added models can be sensitive to the ways in which student achievement is measured.
These findings align with similar findings by Martineau (2006) and Schmidt et al (2005)
You get different results depending on the type of question you’re measuring.

The researchers tested various VAM models (including the type used in TVAAS) and found that teacher effect estimates changed significantly based on both what was being measured AND how it was measured.

And they concluded:

Our results provide a clear example that caution is needed when interpreting estimated teacher effects because there is the potential for teacher performance to depend on the skills that are measured by the achievement tests.

If you measure different skills, you get different results. That decreases (or eliminates) the reliability of those results. TNReady is measuring different skills in a different format than TCAP. It’s BOTH a different type of test AND a test on different standards. Any value-added comparison between the two tests is statistically suspect, at best. In the first year, such a comparison is invalid and unreliable. As more years of data become available, it may be possible to make some correlation between past TCAP results and TNReady scores.

But, TNReady hasn’t gone well. At all. It’s been so bad, the Department of Education has been unveiling a bunch of pie charts to demonstrate how they are attempting to correlate test scores and teacher evaluation. First, it went like this:

Second, this chart is crazy. A teacher’s growth score is factored on tests from three different years and three types of tests.

15% of the growth score comes from the old TCAP (the test given in 2014-15, b/c the 2015-16 test had some problems). Then, 10% comes from last year’s TNReady, which was given on paper and pencil. Last year was the first year of a full administration of TNReady, and there were a few problems with the data calculation. A final 10% comes from this year’s TNReady, given online.

So, you have data from the old test, a skipped year, data from last year’s test (the first time TNReady had truly been administered), and data from this year’s messed up test.

There is no way this creates any kind of valid score related to teacher performance. At all.

After teachers expressed outrage that the DOE was going to count this year’s scores in their evaluations, the legislature finally took action and passed legislation that said teachers could face “no adverse action” based on this year’s test results.

So, now the Department of Education has more pie charts and a lot of explanations:

What is included in teacher evaluation generally?

There are many factors that go into a teacher’s overall evaluation. One of those, the individual growth component (in gray in the charts in this document), is typically based on a three-year TVAAS measure if data is available. However, for the phase-in period there are two key items to note for the growth component:

• If the current single-year year growth score – in this case, 2017-18 data – provides the educator with a higher overall composite, it will be used as the full growth score.

• Additionally, if a teacher has 2017-18 TNReady data included in any part of their evaluation, they will be able to nullify their entire LOE this year.

What is included in teacher evaluation in 2017-18 for a teacher with 3 years of TVAAS data?

There are three composite options for this teacher:

• Option 1: TVAAS data from 2017-18 will be factored in at 10%, TVAAS data from 2016-17 will be factored in at 10% and TVAAS data from 2015-16 will be factored in at 15% if it benefits the teacher.

• Option 2: TVAAS data from 2017-18 and 2016-17 will be factored in at 35%.

• Option 3: TVAAS data from 2017-18 will be factored in at 35%. The option that results in the highest LOE for the teacher will be automatically applied. Since 2017-18 TNReady data is included in this calculation, this teacher may nullify his or her entire LOE this year.

And if you only have one or two years of TVAAS data or if you teach in a non-tested subject? Well, the key line continues to apply: Since 2017-18 TNReady data is included in this calculation, this teacher may nullify his or her entire LOE this year.

What does this mean? Well, it means you’d have a year with no evaluation score. Sounds fine, right? No. It’s not fine. In order to achieve tenure, a teacher must have consecutive years of evaluation scores at Level 4 or 5. But a year with no score at all means that teacher would then need to have TWO MORE YEARS of high scores in order to be tenure eligible. While it seems unlikely a teacher would choose to nullify their entire score if they achieved a high rank, it also seems only fair to allow that teacher to simply exclude the TNReady data and receive their LOE rating based on all the other factors that go into a TEAM rating.

But wait, excluding 2017-18 TNReady data is NOT an option provided. It’s either count it as 10%, count it as 35%, or nullify your entire LOE score. Doing so could certainly have an adverse impact on a teacher.

In short, the TNReady mess has made teacher evaluation a mess. Still, a host of indicators suggest Tennessee’s teachers are hitting the mark. One might conclude that tying a suspect teacher evaluation model to an unreliable test is, in fact, not the key to educational progress in our state. Unfortunately, Governor Bill Haslam has concluded the opposite.

For more on education politics and policy in Tennessee, follow @TNEdReport

Help keep the education news coming!


 

Tempered Enthusiasm

Following last week’s release of TNReady results, Williamson County Director of Schools Mike Looney offered words of caution in interpreting the results.

The Williamson Herald has the story:

Looney said he was proud of how well WCS students, parents, teachers and staff responded to the testing in light of its documented flaws, and he was pleased with the fact that the district remained in the top five in every test and grade level.

“However,” he said in a statement released by WCS, “it would be disingenuous to fully celebrate without acknowledging the problems experienced by students, parents and teachers during last year’s testing process.”

While clearly frustrated with continued TNReady problems, Looney offered hope for a reliable assessment in the future:

“While I am so sorry that our students and teachers had to endure last year’s State testing experience, moving forward, we are optimistic that our students will be able to show what they know with a reliable and functional assessment. As a district, we will continue to be laser focused on success for all students.”

MORE on TNReady:

It’s all been a pack of lies

Beyond TNReady

Definitely something wrong

One glaring exception

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Meaningless Scores Make Big Splash

The Tennessee Department of Education today released results from this year’s troubled administration of TNReady testing.

This Tennessean story indicates Commissioner McQueen is taking the results seriously in spite of what has become an annual inability to get the job done right:

“We see reason to be encouraged, but we also have a lot of work to do to meet our higher expectations for all students,” Education Commissioner Candice McQueen said in a statement. “While we’ve focused extensively on early grades reading and are starting to see a shift in the right direction, we know middle school remains a statewide challenge across the board. TNReady serves as a vital feedback loop for teachers, parents, and administrators to tell us where we are, and the results inform what steps we need to take to help all students and schools succeed.”

Umm…what?? Are you even serious, Candice? THE TESTS DIDN’T WORK! Also, you were caught lying time and again about WHY they didn’t work.

The legislature passed legislation at the end of session, during testing to account for the failure of TNReady.

This isn’t even the first year the tests didn’t work. It happens EVERY year.

Here’s the deal: These scores can’t credibly be used to tell us ANYTHING. Students in schools all across the state faced disruptions caused by dump trucks and hacking (also known as the ineptitude of DOE leadership). Some kids got the wrong test and the extent of that problem is not totally known.

Note to district leaders: If you use these results to say your district or a given school are “doing well,” you have ZERO credibility. Treating these results as anything other than the complete trash that they are is unacceptable.

Some districts have already begun pushing for change in either how TNReady happens next year or even a totally different type of test.

Oh, and note that the DOE makes it sound like they are making improvements:

The Tennessee Department of Education also announced changes after this year’s problems, including searching for a new vendor. It also adjusted how fast it will phase-in online testing.

That sounds great, but the truth is, the DOE and Governor Haslam are treating Tennesseans like we’re stupid:

Let’s get this straight: Governor Haslam and Commissioner McQueen think no one in Tennessee understands Google? They are “firing” the company that messed up this year’s testing and hiring a new company that owns the old one and that also has a reputation for messing up statewide testing.

So, we had a testing season full of lies, deception, disruption, and mixed-up tests but we’re supposed to look at the “results” of those tests and take them seriously? No thanks.

 

For more on education politics and policy in Tennessee, follow @TNEdReport

Help keep the education news coming!


 

Beyond TNReady

At least one school system in Tennessee is taking steps to move beyond TNReady. According to a story in the Wilson Post, Wilson County Schools is seeking legislative action that would allow them to choose and administer their own annual tests in place of the state-mandated TNReady.

Here’s more:

Wilson County Director of Schools Dr. Donna Wright told county commissioners Monday the local school system is pursuing a private act from the state Legislature that would allow it to use an assessment other than the one currently mandated.

In her monthly report to the commission, Wright expressed her dissatisfaction with the TNReady test, saying that, “We are four years in without any or little actionable data that teachers can use.”

Wright added that while district leaders support accountability, the lack of timely, reliable data from the state tests is problematic:

“We are absolutely advocates of accountability because that’s how we know what to improve and where to improve,” Wright said adamantly. “But the fallacy in all this is that we haven’t had an effective system in four years, but we still keep using information that is not only in error, but late in coming.”

The action in Wilson County follows a resolution passed in Johnson City calling for a significant reduction in state-mandated testing.

The movement to reduce or replace TNReady follows yet another year of testing problems and a litany of excuses offered by the Department of Education and the state’s testing vendor.

Wright is correct that mishaps in testing and the late return of results call the usefulness of the data into question. However, even in the best of circumstances, it would be difficult to arrive at valid, actionable data based on the early years of a new test.

It will be interesting to see if other school systems follow the lead of Johnson City and Wilson County. Perhaps we’re finally seeing district leaders stand up and say “enough!”

For more on education politics and policy in Tennessee, follow @TNEdReport

Your support keeps the education news coming!


 

It’s All Been a Pack of Lies

By now, it should come as no surprise that our Commissioner of Education and the department she leads has a troubled relationship with the truth. That said, today’s revelation at a legislative hearing that an alleged hack of the state’s TNReady test didn’t actually happen again raises the question: Why does Candice McQueen still have a job?

Back on April 17th, the day after TNReady failed to work on day one of this year’s testing, the Tennessee Department of Education noted that the Day 2 failures were related to someone hacking the vendor:

At a legislative hearing today, the Tennessee Bureau of Investigation (TBI) indicated there was no evidence of a hack.

Additionally, the Department of Education issued this statement, which notes:

  • It appears, thankfully, that there was not an outside actor who attacked Questar’s data system. No student data was breached.
  • It is now clear that the event that Questar initially thought presented like a denial of service attack on Tuesday, April 17 was not created by an external actor with malicious intent, but, rather, can be traced in large part to the caching issues connected to how text-to-speech was configured by Questar.
  • Questar implemented a significant and unauthorized change to text-to-speech, which had previously operated successfully during the state’s fall administration. We now know this decision led to the severity of other issues we experienced during online testing.
  • Questar continues their internal investigation and is cooperating with additional external audits to make sure we have all of the facts.

Questar’s Chief Operating Officer Brad Baumgartner has provided this statement: “Questar’s internal and external investigations indicate that the source of the anomalous data pattern is believed to be the result of a configuration with the cache server. We have applied a configuration change and believe to have resolved the issue. We will continue to work with our internal technology team and external partners to validate this.”

The text-to-speech feature was also blamed for students receiving the wrong tests.

While at the time, the hacking excuse sounded pretty far-fetched, today’s hearing confirms that the Department advanced a lie offered by the state’s testing vendor. Of course, later on in the testing cycle, a dump truck was blamed for disrupting testing. That excuse was also later proven untrue.

All of this may explain why at least one school district is calling for a significant reduction in TNReady testing next year.

If this year had been the first time our state had faced testing challenges, one might understand (and forgive) the excuse-making. However, this is now the fifth consecutive year of some sort of problem and the fourth year testing administration has been, to say the least, a challenge.

One may recall the saga of Measurement, Inc. The company that hired test graders from Craigslist and was ultimately fired in 2016 after that year’s TNReady test failed.

The bottom line: If TNEdu tells you something about testing, you should question it. The track record shows that to our state’s Department of Education, truth is a relative concept.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Your support keeps the education news coming!

If this is what success looks like…

In a story about the Tennessee Department of Education scaling back the requirements for online testing next year in light of this year’s testing challenges, this caught my attention:

Even with the problems this year, it was one of the most successful online administrations for the state to date. More than 2.5 million TNReady tests were administered this spring, with about 300,000 students taking the test online. Only high school students were required to take the online version this year.

What does the word “success” mean? Because my recollection of this year’s TNReady administration is that it was a debacle.

I’m not the only one. As I noted last week:

While lots of states are moving to online testing, one expert says Tennessee is unique:

“I’m not aware of a state that has had a more troubled transition” to online testing, said Douglas A. Levin of the consulting group EdTech Strategies.

And there’s this helpful explainer:

Why is Tennessee in the unique position of having the worst online testing transition in the country?

The reality is that Tennessee’s online-testing mess has left everyone in a difficult position, said Chad Aldeman, a principal at Bellwether Education Partners, a consulting organization.

“The state has not [made] stability a key priority in their testing vendors,” Aldeman said.

Nevertheless, Commissioner of Education Candice McQueen says:

The state will put out a request for contract proposals in the fall, with a new vendor to be identified in the spring. Questar Assessment could again win the contract, but McQueen said who wins the proposal will have to show the ability and history of seamlessly administering an online test.

“We look for a company with a track record of success in administering online testing and who can manage our test well.”

Haven’t we heard that before?

For more on education politics and policy, follow @TNEdReport

Keep the education news coming!