Battle Lines Being Drawn

Last week, the School Superintendents in Memphis and Nashville wrote a letter to Commissioner of Education Candice McQueen calling for a pause in TNReady. The letter indicated the leaders had “no confidence” in TNReady. Following the letter, the Knox County School Board voted 8-1 to send a letter to Governor Haslam stating they had “no confidence” in the Department of Education. Later that week, the Director of Schools in Maury County said he agreed with the idea of pausing TNReady and suggested moving to the ACT suite of assessments.

Today, Commissioner McQueen issued a response. According to Chalkbeat, her response indicates that pausing TNReady would be “illegal and inconsistent with our values as a state.”

McQueen cites federal and state law requiring test administration. Here’s the deal: The entity that determines the penalty for violating state law regarding testing is the Department of Education. The penalty they can use is withholding BEP funds. This is the threat they used back in 2016 to force districts to back down on threats to halt testing then.

Let’s be clear: The Tennessee Department of Education is the enforcer of the state testing mandate. The DOE could refuse to penalize districts who paused testing OR the DOE could take the suggestion made by Dorsey Hopson of Memphis and Shawn Joseph of Nashville and just hit the pause button for this year and work toward an effective administration of testing for 2019-20.

Next, McQueen cites federal law. I’ve written about why this is misguided. Here’s more:

There’s just one problem: The federal government has not (yet) penalized a single district for failing to hit the 95% benchmark. In fact, in the face of significant opt-outs in New York last year (including one district where 89% of students opted-out), the U.S. Department of Education communicated a clear message to New York state education leaders:  Districts and states will not suffer a loss of federal dollars due to high test refusal rates. The USDOE left it up to New York to decide whether or not to penalize districts financially.

That’s right, the federal government tends to leave decisions regarding punishment up to the states. Of course, Tennessee could also request a 1-year waiver of ESSA requirements in order to further clarify the need to get testing right. In short, the only problem now is McQueen’s unwillingness to admit failure and take aggressive steps to make improvement.

McQueen also says that halting testing is “inconsistent” with Tennessee values.

While in McQueen’s world, halting testing is inconsistent with our state’s values, lying about why testing isn’t working is apparently perfectly fine. Oh, and playing a game with testing vendors? No problem!

McQueen claims that we need the tests to help identify gaps in education delivery in traditionally under-served students. Yes, having a working annual assessment can be a helpful tool in identifying those gaps. But, when the test doesn’t work — when students get the wrong test, when the testing climate is not consistent — then we get results that are unreliable. That helps no one.

What should be consistent with Tennessee values is taking the time to get testing right. That means ensuring it’s not disruptive to the instruction process and the results are useful and returned to students, teachers, and parents in a timely fashion.

Will McQueen’s letter deter other district leaders from speaking out on TNReady? Will there be additional fallout from the DOE’s failure to effectively administer Pre-K/K portfolios?

Stay tuned.

 

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Blood in the Water

The Director of Schools in Maury County has joined those in Memphis and Nashville in calling for a pause in TNReady as a result of repeated problems with the testing platform.

The Columbia Daily Herald reports Maury County Director of Schools Chris Marczak said he agrees with the letter sent by Dorsey Hopson of Memphis and Shawn Joseph of Nashville. Marczak offered an alternative:

“I believe it would be best for us to focus solely on the ACT and align ourselves with outcomes that can affect students’ college acceptance and scholarship ability,” Marczak said.

Maury County district leadership has indicated the results from this year’s botched test administration are of limited value:

“Due to the issues with testing, we will not be adding TNReady/EOC data to the Keys’ scorecards for either the district or the school levels when they eventually come in,” Marczak said in an email sent to staff in July. “In light of the numerous testing issues, please know that the results of the assessments will be used to inform conversation only. These are the conversations we will have with principals and the principals will have with teachers/staffs.”

In response to the ongoing testing issues, Marczak shared accounts of students completing 75-minute long examinations in 10 minutes. When reviewing the examinations, Marczak said the district had over 600 missing scores. Questar, the contractor hired by the state to administer the test, reported that 600 individual assessments were incomplete.

Despite the growing concern over the inability to effectively administer the TNReady test, Commissioner of Education Candice McQueen has said the test is still an important tool:

TNReady serves as a vital feedback loop for teachers, parents, and administrators to tell us where we are, and the results inform what steps we need to take to help all students and schools succeed.”

TNReady might be an important feedback loop if it ever worked the way it was intended. But it hasn’t. Instead, it’s been fraught with problems since the beginning. Now, education leaders are standing up and speaking out.

The push to pause TNReady and possibly move forward with a different measure comes at the same time the TDOE is being taken to task for a failure to properly execute Pre-K/Kindergarten portfolios. Knox County’s School Board last night voted to send a message that they have “no confidence” in the portfolio process or in the TDOE.

The push against TNReady from key district leaders figures to make the test and overall administration of the Department of Education a key issue in the 2018 gubernatorial and state legislative elections.

 

 

For more on education politics and policy in Tennessee, follow @TNEdReport


 

“No Confidence” in TNReady

Just days after members of the Knox County School Board took the Tennessee Department of Education to task for “incompetence” and an “abject failure” to measure student achievement or teacher performance, the Directors of the state’s two largest school districts, Nashville and Memphis, sent a letter to Education Commissioner Candice McQueen and Governor Bill Haslam stating they had “no confidence” in TNReady and asking the state to pause the test.

The letter, signed by Nashville’s Shawn Joseph and Shelby County’s Dorsey Hopson, says in part:

“We respectfully ask the State to hit the pause button on TNReady in order to allow the next Governor and Commissioner to convene a statewide working group of educators to sort out the myriad challenges in a statewide, collaborative conversation.”

The two leaders, whose districts represent 20 percent of all students in Tennessee, note:

“We are challenged to explain to teachers, parents, and students why they must accept the results of a test that has not been effectively deployed.”

The language from these two directors is the strongest yet from any district and the first to call for an outright stop to administration of the TNReady test while the state explores other options. Johnson City’s school board sent a proposal asking for a significant reduction in testing while Wilson County is exploring the possibility of administering a different test altogether. At the same time, Williamson County Director of Schools Mike Looney expressed concern about the poor administration of this year’s test.

It seems clear there is growing concern among educators about the continued use of TNReady. As Joseph and Hopson note, taxpayer resources have been invested in a test that is poorly implemented and yields suspect results. Taking their suggestion of a pause could give the state and a new Governor and Education Commissioner time to actually develop a process for administering an aligned assessment that does not disrupt instruction and does return useful, meaningful results to teachers, parents, and students.

Here’s the letter:

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Support TNEdReport!

Would You Like Some Pie?

Chalkbeat’s Laura Faith Kebede reports on how the United Education Association, which represents teachers in Shelby County, describes the current TNReady situation:

“It’s as if you had a piece of pie, and I find a piece of glass in it,” she said. “But I cut somebody else a piece of that same pie and assure you ‘You don’t have glass in yours.’ Are you going to trust me and eat that piece of pie knowing that there’s a piece of glass in mine?”

That’s how UEA President Tikela Rucker described the current state of TNReady given what she cited as years of problems:

“This is the third year in a row that we’ve experienced issues regarding TNReady, which leads us to have zero confidence in TNReady, Commissioner McQueen and the Tennessee Department of Education,” said Shelby County UEA President Tikeila Rucker on behalf of the union’s 2,000 members.

Shelby County Director of Schools Dorsey Hopson also suggested the state needed to be more accountable:

“We stand in solidarity with our teachers. We know the (state) Department of Education is working very hard,” Hopson told reporters. “But given the high-stakes nature of the test, we just want to be accurate. And when they’re not accurate, it just casts a cloud of doubt over the whole process.”

Hopson stopped short of calling for test scores to be invalidated. “I wouldn’t necessarily jump to that conclusion,” he said, “but I do agree with our teaching colleagues that the results need to be accurate and timely.”

The UEA is calling for all scores from this year to be invalidated and for a moratorium on using TNReady scores in the state’s accountability system until 2021.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

 

Apples and Oranges

Here’s what Director of Schools Dorsey Hopson had to say amid reports that schools in his Shelby County district showed low growth according to recently released state test data:

Hopson acknowledged concerns over how the state compares results from “two very different tests which clearly are apples and oranges,” but he added that the district won’t use that as an excuse.

“Notwithstanding those questions, it’s the system upon which we’re evaluated on and judged,” he said.

State officials stand by TVAAS. They say drops in proficiency rates resulting from a harder test have no impact on the ability of teachers, schools and districts to earn strong TVAAS scores, since all students are experiencing the same change.

That’s all well and good, except when the system upon which you are evaluated is seriously flawed, it seems there’s an obligation to speak out and fight back.

Two years ago, ahead of what should have been the first year of TNReady, I wrote about the challenges of creating valid TVAAS scores while transitioning to a new test. TNReady was not just a different test, it was (is) a different type of test than the previous TCAP test. For example, it included constructed response questions instead of simply multiple choice bubble-in questions.

Here’s what I wrote:

Here’s the problem: There is no statistically valid way to predict expected growth on a new test based on the historic results of TCAP. First, the new test has (supposedly) not been fully designed. Second, the test is in a different format. It’s both computer-based and it contains constructed-response questions. That is, students must write-out answers and/or demonstrate their work.

Since Tennessee has never had a test like this, it’s impossible to predict growth at all. Not even with 10% confidence. Not with any confidence. It is the textbook definition of comparing apples to oranges.

Here’s a statement from the academic article I cited to support this claim:

Here’s what Lockwood and McCaffrey (2007) had to say in the Journal of Educational Measurement:

We find that the variation in estimated effects resulting from the different mathematics achievement measures is large relative to variation resulting from choices about model specification, and that the variation within teachers across achievement measures is larger than the variation across teachers.
You get different value-added results depending on the type of test you use. That is, you can’t just say this is a new test but we’ll compare peer groups from the old test and see what happens. Plus, TNReady presents the added challenge of not having been fully administered last year, so you’re now looking at data from two years ago and extrapolating to this year’s results.
Of course, the company paid millions to crunch the TVAAS numbers says that this transition presents no problem at all. Here’s what their technical document has to say about the matter:
In 2015-16, Tennessee implemented new End-of-Course (EOC) assessments in math and English/language arts. Redesigned assessments in Math and English/language arts were also implemented in grades 3-8 during the 2016-17 school year. Changes in testing regimes occur at regular intervals within any state, and these changes need not disrupt the continuity and use of value-added reporting by educators and policymakers. Based on twenty years of experience with providing valueadded and growth reporting to Tennessee educators, EVAAS has developed several ways to accommodate changes in testing regimes.
Prior to any value-added analyses with new tests, EVAAS verifies that the test’s scaling properties are suitable for such reporting. In addition to the criteria listed above, EVAAS verifies that the new test is related to the old test to ensure that the comparison from one year to the next is statistically reliable. Perfect correlation is not required, but there should be a strong relationship between the new test and old test. For example, a new Algebra I exam should be correlated to previous math scores in grades seven and eight and to a lesser extent other grades and subjects such as English/language arts and science. Once suitability of any new assessment has been confirmed, it is possible to use both the historical testing data and the new testing data to avoid any breaks or delays in value-added reporting.
A couple of problems with this. First, there was NO complete administration of a new testing regime in 2015-16. It didn’t happen.
Second, EVAAS doesn’t get paid if there’s not a way to generate these “growth scores” so it is in their interest to find some justification for comparing the two very different tests.
Third, researchers who study value-added modeling are highly skeptical of the reliability of comparisons between different types of tests when it comes to generating value-added scores. I noted Lockwood and McCaffrey (2007) above. Here are some more:
John Papay (2011) did a similar study using three different reading tests, with similar results. He stated his conclusion as follows: [T]he correlations between teacher value-added estimates derived from three separate reading tests — the state test, SRI [Scholastic Reading Inventory], and SAT [Stanford Achievement Test] — range from 0.15 to 0.58 across a wide range of model specifications. Although these correlations are moderately high, these assessments produce substantially different answers about individual teacher performance and do not rank individual teachers consistently. Even using the same test but varying the timing of the baseline and outcome measure introduces a great deal of instability to teacher rankings.
Two points worth noting here: First, different tests yield different value-added scores. Second, even using the same test but varying the timing can create instability in growth measures.
Then, there’s data from the Measures of Effective Teaching (MET) Project, which included data from Memphis. In terms of reliability when using value-added among different types of tests, here’s what MET reported:
Once more, the MET study offered corroborating evidence. The correlation between value-added scores based on two different mathematics tests given to the same students the same year was only .38. For 2 different reading tests, the correlation was .22 (the MET Project, 2010, pp. 23, 25).
Despite the claims of EVAAS, the academic research raises significant concerns about extrapolating results from different types of tests. In short, when you move to a different test, you get different value-added results. As I noted in 2015:

If you measure different skills, you get different results. That decreases (or eliminates) the reliability of those results. TNReady is measuring different skills in a different format than TCAP. It’s BOTH a different type of test AND a test on different standards. Any value-added comparison between the two tests is statistically suspect, at best. In the first year, such a comparison is invalid and unreliable. As more years of data become available, it may be possible to make some correlation between past TCAP results and TNReady scores.

Or, if the state is determined to use growth scores (and wants to use them with accuracy), they will wait several years and build completely new growth models based on TNReady alone. At least three years of data would be needed in order to build such a model.

Dorsey Hopson and other Directors of Schools should be pushing back aggressively. Educators should be outraged. After all, this unreliable data will be used as a portion of their teacher evaluations this year. Schools are being rated on a 1-5 scale based on a growth model grounded in suspect methods.

How much is this apple like last year’s orange? How much will this apple ever be like last year’s orange?

If we’re determined to use value-added modeling to measure school-wide growth or district performance, we should at least be determined to do it in a way that ensures valid, reliable results.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Doing the Right Thing

Shelby County’s Director of Schools Dorsey Hopson announced that all teachers will receive a three percent raise this year, not just those who meet certain scores on the state’s flawed value-added evaluation system.

More from Chalkbeat:

Hopson told the district’s educators in an email Thursday that they’ll see the raise reflected in their Nov. 18 paychecks. The pay hikes will be retroactive and will also go to librarians, counselors, instructional facilitators, coaches, social workers, physical/speech therapists and psychologists.

The decision came after Hopson learned that the district won’t receive the state’s testing data until December.

The decision by Hopson came about as a result of last year’s TNReady debacle. It also came in the same week that Knox County’s School Board asked the state for a waiver from included this year’s TNReady test results in student grades and teacher evaluations.

Hopson made the right decision — it is unfair to ask teachers to wait to receive pay raises because of the state’s mistakes with TNReady. It’s also unfair to use data from last year’s mess of a test administration to evaluate teachers. While I’ve expressed doubts about the usefulness of value-added data in evaluating teachers, even those who haven’t should acknowledge that using data from last year (or this year) is problematic.

Shelby County educators will all see a raise this year. The next question: Will the school board there join Knox County in requesting a waiver from using test data for students and teachers this year?

For more on education politics and policy in Tennessee, follow @TNEdReport


 

ASD Flexes Muscles in Memphis

Jon Alfuth over at Bluff City Ed has an update on the ASD and its actions in Memphis and in Nashville.

Alfuth’s report includes information on the teacher group organizing against ASD takevovers and concerns expressed by Shelby County Schools Superintenent Dorsey Hopson.

Most troubling is the notes about Chris Barbic’s remarks asserting the ASD could take all 85 schools on the priority list and noting that the ASD has been essentially playing nice up to this point.

Here’s the excerpt:

The most interesting and worrisome part of the last article for me are two quotes from Chris Barbic, the ASD’s Superintendent. First, Roberts quotes him as saying

“I think its important to remind everyone that a lot of things we are doing are by choice. If we wanted to, we could take over all 85 schools next year (bold added by me for emphasis).”

Second, he also states that they don’t have to do a matching process but that they have chosen to do so, and laments that they are being “beaten up” for what they are doing. He also says that they (presumably the community) “would be beating us up for not doing it (the matching process).”

These quotes trouble me because they perpetuate the message Memphians have been getting that “we (the ASD) have the power, and you should be thankful that we’re including you.” While I don’t know what else was said in the interview, it worries me that this is the type of rhetorical language that we’re seeing in the wake of a strong anti-ASD outpouring. Barbic does qualify his first quote by stating that they’ve chosen to work with SCS instead of pursing the total takeover course of action, but the lead sentence makes it clear who really has the decision making power.

In the end, the ASD opposition is all about who has the power to improve our schools, and quotes like this don’t do anything to alleviate the idea growing locally that the ASD is engaged in a hostile takeover of Memphis.

The entire update is worth a read.

I tend to agree with Jon that the challenge to the ASD is essentially about power. Parents, teachers, and even SCS leaders feel like they lack the power to make decisions about the schools.

For more on education politics and policy in Tennessee, follow @TNEdReport

Struggling ASD to Takeover 9 More Memphis Schools

The Memphis Commercial Appeal reports:

Nine more schools in Memphis will be taken over by the state-run Achievement School District next fall, including Wooddale Middle, Raleigh-Egypt High and South Side Middle, which have already been assigned to charter schools. Nine others — Florida-Kansas Elementary, Denver Elementary, Airways Middle, Brookmeade Elementary, American Way Middle, Hawkins Mills Elementary, LaRose Elementary, A. Maceo Walker Middle and A.B. Hill Elementary — are eligible for takeover, although only six of those will be under new management. The six to be taken over will be determined in part by a community vetting process that suggests which charter operators are best-suited to each school’s needs. The ASD will announce the final matches in December.

A recent analysis of the ASD’s performance indicates that the schools it has taken over in Shelby County would have been better off if they had remained in district hands. The ASD’s student achievement numbers have failed to meet their own ambitious targets and also failed to grow at a rate consistent with that of district schools.

An additional analysis compared schools in the iZone to ASD schools and found that the iZone model consistenly out-performs the ASD model.

When asked recently if he planned to convert iZone schools to charter schools, Shelby County Director of Schools Dorsey Hopson called the idea “absurd.”

Hopson’s statement is noteworthy because converting district schools to charters is exactly what the ASD plans to do.

While it may be fair to give the ASD more time to prove it can be effective with its existing schools, it seems irresponsible to allow the ASD to take on more schools. The leaders of the ASD would not be likely to put more students into a classroom of a teacher who failed to meet their desired student achievement targets. Why should more schools be handed over to a model that’s not only not living up to its own hype, but also failing to outperform the district schools it has taken over?

For more on education politics and policy in Tennessee, follow @TNEdReport

Hopson: Turning iZone over to Charters “Absurd”

Amid reports that the Shelby County Schools iZone may turn over some of its schools to charter operators due to financial concerns, Director of Schools Dorsey Hopson told the Memphis Daily News, “That’s absurd. I just want to be clear on that.”

Instead, Hopson indicated he plans to seek additional grants and/or private funding to continue the successful iZone efforts.

A recent analysis indicates that iZone schools are outperforming their Achievement School District counterparts. In short, the iZone is working. And Hopson’s comments acknowledge that while also making clear his commitment to find a way to stick with what’s working to help improve outcomes for students.

 

For more on education politics and policy in Tennessee, follow @TNEdReport