Wrong Answer

In the never ending saga that is testing in Tennessee, the latest chapter spins a familiar but frustrating tale. It seems the state’s testing vendor incorrectly scored thousands of TNReady tests, impacting student score reports and teacher evaluation scores based on those student scores.

Jennifer Pignolet and Jason Gonzales have more:

About 9,400 TNReady tests across the state were scored incorrectly, according to the Tennessee Department of Education.

The scoring issue impacted about 70 schools in 33 districts. Just over 1,000 of the incorrectly scored tests were in Shelby County Schools, according to an email from Superintendent Dorsey Hopson to his board on Friday.

Approximately 1,700 of the total incorrect tests scores, once corrected, changed what scoring category that test fell into, possibly affecting whether a student passed the test.

The error also impacted value-added scores for up to 230 teachers. A separate problem could impact TVAAS scores for as many as 900 teachers.

The scope of the error means scores in nearly 25% of the state’s school districts will need to be corrected. The Department of Education says the testing vendor, Questar, is re-scoring the tests.

UPDATE — Here’s a list of districts impacted:

  • Achievement School District
  • Anderson County
  • Benton County
  • Bradley County
  • Bristol City
  • Carter County
  • Cocke County
  • Collierville City
  • Crockett County
  • Davidson County
  • Elizabethton City
  • Giles County
  • Hamilton County
  • Hardin County
  • Henry County
  • Huntingdon Special School District
  • Jackson-Madison County
  • Knox County
  • Lewis County
  • Lincoln County
  • Marshall County
  • Maryville City
  • Monroe County
  • Montgomery County
  • Obion County
  • Putnam County
  • Roane County
  • Rutherford County
  • Shelby County
  • Smith County
  • Sumner County
  • Union City
  • Weakley County

The State of Tennessee has spent millions of dollars on a new testing regime supposedly better able to assess student mastery of state standards. So far, all most students, teachers, and parents have seen is problems.

The first set of problems happened on day one of the initial online administration of the test in 2016. Then, a series of missed deadlines led to the state firing then-vendor Measurement, Inc. That’s the same company that hired test scorers via ads on Craigslist.

Of course, this is the same Department of Education that has repeatedly had issues with test score data.

If only there had been warning signs or calls to take the time to phase-in TNReady so that it best serves students and educators.

You know, something like:

TNReady is measuring different skills in a different format than TCAP. It’s BOTH a different type of test AND a test on different standards. Any value-added comparison between the two tests is statistically suspect, at best. In the first year, such a comparison is invalid and unreliable. As more years of data become available, it may be possible to make some correlation between past TCAP results and TNReady scores.

Or, if the state is determined to use growth scores (and wants to use them with accuracy), they will wait several years and build completely new growth models based on TNReady alone. At least three years of data would be needed in order to build such a model.

That’s from an article I wrote in March of 2015 about TNReady data and the challenges of adapting to a new test using our current accountability system.

That was BEFORE the 2016 TNReady mess. It was before the state had a problem getting data back this year.

How many warning signs will be ignored? How important is the test that it must be administered at all costs and the mistakes must be excused away because “accountability” demands it?

How can you hold students and teachers and schools accountable when no one is holding the Department of Education accountable? How long will legislators tolerate a testing regime that creates nightmares for our students and headaches for our teachers while yielding little in terms of educational value?

At least one school board has complained about the state’s handling of TNReady data this year. I suspect more will follow in the wake of this latest mistake.

So far, TNReady has sent one clear message: Accountability is a one way street in Tennessee and students, teachers, and districts are on the wrong end.

 

For more on education politics and policy in Tennessee, follow @TNEdReport


 

 

TC Talks Testing

Nashville education blogger TC Weber talks about testing (and a lot of other things) in his latest post.

Specifically, he talks about the release of data on TNReady tests and the comparisons being made to previous TCAP tests.

Keep in mind: We didn’t have a complete administration of TNReady in 2016. Which means the 2017 test was the first year for TNReady. It also means the comparisons being made are based on different tests taken two years ago. So, you have analysis of 5th grade results and “growth” on TNReady being made in comparison to 3rd grade results on TCAP.

It’s apples and oranges. 

Here’s what TC has to say:

Let’s approach this in a different manner though. Say I annually run a 5k race and each year my timing goes up a little bit, so I’m feeling like  I want something different. After year 5 I change to a 10k race. My time for that race is substantially lower. What conclusions can I draw from that difference in time? Am I really not that good a 5k runner? Is the course really that much harder than the 5k I was running? Is my training off? Am I not that good a runner?
I’d say there are very few conclusions, based on comparing the results between my 5k and my 10k time, that can be drawn. It could be that the length of the course was a bigger adjustment than anticipated. It could be that conditions were worse on the day I ran the 10k vs the 5k. It could be that one course was flatter and one was hillier. A kid could be good at bubble in questions but not write ins. How do we know that improvement isn’t contingent just on familiarity with the course? Or the test?
I know people will argue that we should all be training to run hills instead of a flat races. But does running hills well really indicate that I am a better runner? Terrain is just another variable. My liberal arts education always explained to me that in order to get the most accurate measurement possible you need to remove as many of the variables as possible.
One year of data is not a real indication of anything other than, kid’s are not very good at taking this test. In order to draw any meaningful conclusions, you would have to have a set of data that you could analyze for trends. Simply taking a 10k race and comparing it’s results to a 5k race’s results, just because both are races, is not a valid means to draw conclusions about a runners abilities. The same holds true for students and testing.
If TNReady really is the amazing test we’ve all been waiting for, why not take the time to build a reliable set of data? The results from year one don’t really tell us much of anything. Because we skipped* 2016, it’s even MORE difficult to draw meaningful conclusions about the transition from TCAP to TNReady.
TC talks about these challenges and more issues. Check it out.
*We didn’t actually skip the 2016 test. Instead, many students attempted to take the test only to face glitches with the online system. Schools then were given various new times for testing to start only to have those dates changed and ultimately, to see the test cancelled. 
Kids were jerked around with messages about how the “important test” was coming up next week only to have it not happen. Teachers were told they’d be proctoring tests and instead had to quickly plan lessons. Our schools and students adapted, to be sure. But, there is no way to give back the instructional time lost in 2016.
Now, we have students taking THE test in 2017 only to see a slow drip of data come back. Students are told the test matters, it will count toward their grades. Teachers have growth scores based on it. Schools are assigned ratings based on it. But, getting it right doesn’t matter. Well, unless it does.
Oh, and we spend a lot of money on a testing system that produces questionable results with data coming back at a time that reduces usefulness.
What’s next? This year, we’ll try again to administer TNReady online across the state. That didn’t work so well with the previous vendor, but maybe it will this time. Of course, online administration adds another variable to the mix. So, 2018 will be the first time many students have taken a fully online TNReady test. Assuming it works, online administration could address the challenges of getting results back in a timely fashion. But, the transition could impact student performance, once again calling into question the legitimacy of growth scores assigned to students and schools.
For more on education politics and policy in Tennessee, follow @TNEdReport


 

Flight of the Dreamers

I’ve written before about the challenges facing DACA recipients unless Congress acts and noted Tennessee’s Attorney General changed his stance after some focused advocacy.

Now, some Tennessee Dreamers are joining others from around the country in lobbying Congress to take immediate action.

Here’s more from a press release on the Tennessee connection:

Dreamer Yenin E., a student at Trevecca University, wants to be a Christian bilingual counselor, but if she loses DACA protection, she will be out of luck.

 

Yenin, who lives in Smyrna, is one of three Nashville-area young people who are in Washington today through Thursday to talk to members of the Tennessee congressional delegation about passing the Dream Act to allow them to remain in the country. They will be joining approximately 100 Dreamers from 25 states around the country.

 

“Tennessee has been my home ever since I was 4 years old. I have been serving my community through the National Honor Society, Rotary, and BETA club, and I graduated with honors as a Tennessee Scholar,” she said. “I work two jobs, one as a cashier in my community and also as a preschool teacher to pay my way through school.”

 

“Losing DACA would leave me without the ability to pay for my tuition and pursue my dreams of becoming a licensed professional counselor.”

 

Yenin is one of nearly 700,000 individuals brought to this country, undocumented, as children. As of now, these Dreamers, as they are known, can stay and work under limited conditions without being deported, under the protection of the Deferred Action for Childhood Arrivals (DACA) program.

 

But the Trump administration has decided to rescind DACA, and protection for Dreamers could end. The Dream Act, if passed, would allow them to continue their work and their studies in this country.

 

Members of Congress and their staffs will hear stories first-hand from Dreamers like Yenin, who was born in Latin America.

 

“I want to be a Christian bilingual counselor so that I can help those who have been impacted by our immigration system, other tragic life events, or are struggling with mental health,” she said. “My dream of helping those in need would not be possible without DACA.”

 

Also on today’s trip to Washington is Molly Haynes with Equal Chance for Education, a local foundation that works with Dreamers to help them go to college and succeed.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

A Warning

Nashville School Board members Amy Frogge and Will Pinkston took to the blog Seattle Education to issue a warning about the Center for Reinventing Public Education (CRPE) housed at the University of Washington Bothell.

Here’s some of what they had to say:

CRPE’s list of “senior research affiliates” reads like a Who’s Who of special interests determined to tear down public schools and replace them with publicly funded, privately run charter schools. As members of the local school board in Nashville who are fighting against the devastating effects of school privatization, we are writing this column to advise Washington public education advocates — including the leadership and faculty at UW Bothell — that you have an enemy in your midst.

Here’s how they describe the CRPE’s role in recent Nashville education battles:

Political and business interests aligned with the charter movement seized on the CRPE compact to attempt a wholesale privatization of Nashville’s public school system. Some even shamefully referred to their plan as “New Orleans without the hurricane” — a reference to the charterization of Crescent City schools in the wake of Hurricane Katrina. CRPE, led by pro-privatization director Robin Lake, cheered the effort.

Fortunately, the voters of Nashville ultimately rejected CRPE and Lake’s agenda by overwhelmingly electing and re-electing a strong pro-public education contingent to the Nashville school board. Yet the well-funded CRPE threat persists, in our city and elsewhere in the U.S.

The post goes on to alert Washingtonians of what Frogge and Pinkston describe as a clear threat to public education. The warning is well worth a read. 

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Muddy Waters

Laura Faith Kebede of Chalkbeat reports on the challenges in generating reliable TVAAS scores as a result of TNReady trouble last year. Her story cites a statistician from the Center for Assessment who explains the issue this way:

Damian Betebenner, a senior associate at Center for Assessment that regularly consults with state departments, said missing data on top of a testing transition “muddies the water” on results.

“When you look at growth over two years, so how much the student grew from third to fifth grade, then it’s probably going to be a meaningful quantity,” he said. “But to then assert that it isolates the school contribution becomes a pretty tenuous assertion… It adds another thing that’s changing underneath the scene.”

In other words, it’s difficult to get a meaningful result given the current state of testing in Tennessee. I wrote recently about this very issue and the problem with the validity of the growth scores this year.

Additionally, two years ago, I pointed out the challenges the state would face when shifting to a new test. Keep in mind, this was before all the TNReady trouble that further muddied the waters. Here’s what I said in March of 2015:

Here’s the problem: There is no statistically valid way to predict expected growth on a new test based on the historic results of TCAP. First, the new test has (supposedly) not been fully designed. Second, the test is in a different format. It’s both computer-based and it contains constructed-response questions. That is, students must write-out answers and/or demonstrate their work.

Since Tennessee has never had a test like this, it’s impossible to predict growth at all. Not even with 10% confidence. Not with any confidence. It is the textbook definition of comparing apples to oranges.

The way to address this issue? Build multiple years of data in order to obtain reliable results:

If you measure different skills, you get different results. That decreases (or eliminates) the reliability of those results. TNReady is measuring different skills in a different format than TCAP. It’s BOTH a different type of test AND a test on different standards. Any value-added comparison between the two tests is statistically suspect, at best. In the first year, such a comparison is invalid and unreliable. As more years of data become available, it may be possible to make some correlation between past TCAP results and TNReady scores.

So, now we have two challenges: We have two different types of tests AND we have a missing year of data. Either one of these challenges creates statistical problems. The combination of the two calls for a serious reset of the state’s approach to accountability.

As I suggested yesterday, taking the time to get this right would mean not using the TNReady data for accountability for teachers, students, or schools until 2019 at the earliest. If our state is committed to TNReady, we should be committed to getting it right. We’re spending a lot of money on both TNReady and on TVAAS. If we’re going to invest in these approaches, we should also take the time to be sure that investment yields useful, reliable information.

Why does any of this matter? Because, as Kebede points out:

At the same time, TVAAS scores for struggling schools will be a significant factor to determine which improvement tracks they will be be placed on under the state’s new accountability system as outlined in its plan to comply with the federal Every Student Succeeds Act. For some schools, their TVAAS score will be the difference between continuing under a local intervention model or being eligible to enter the state-run Achievement School District. The school growth scores will also determine which charter schools are eligible for a new pot of state money for facilities.

TVAAS scores also count in teacher evaluations. TNReady scores were expected to count in student grades until the quick scores weren’t back in time. If all goes well with the online administration of TNReady this year, the scores will count for students.

The state says TNReady matters. The state evaluates schools based on TVAAS scores. The state teacher evaluation formula includes TVAAS scores for teachers and TNReady scores as one measure of achievement that can be selected.

In short: Getting this right matters.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Seeping Scores Sour School Board

Members of the Murfreesboro City School Board are not happy with the slow pace of results coming from the state’s new TNReady test. All seven elected board members sent a letter to Commissioner of Education Candice McQueen expressing their concerns.

The Daily News Journal reports:

“However, currently those test scores seep ever-so-slowly back to their source of origin from September until January,” the letter states. “And every year, precious time is lost. We encourage you to do everything possible to get test results — all the test results — to schools in a timely manner.

“We also encourage you to try to schedule distribution of those results at one time so that months are not consumed in interpreting, explaining and responding to those results,” the letter continued.

A Department of Education spokesperson suggested the state wants the results back sooner, too:

“We know educators, families and community members want these results so they can make key decisions and improve, and we want them to be in their hands as soon as possible,” Gast said.. “We, at the department, also desire these results sooner.”

Of course, this is the same department that continues to have trouble releasing quick score data in time for schools to use it in student report cards. In fact, this marked the fourth consecutive year there’s been a problem with end of year data — either timely release of that data or clear calculation of the data.

TDOE spokesperson Sara Gast went further in distancing the department from blame, saying:

Local schools should go beyond TNReady tests in determining student placement and teacher evaluations, Gast said.

“All personnel decisions, including retaining, placing, and paying educators, are decisions that are made locally, and they are not required to be based on TNReady results,” Gast said. “We hope that local leaders use multiple sources of feedback in making those determinations, not just one source, but local officials have discretion on their processes for those decisions.”

Here’s the problem with that statement: This is THE test. It is the test that determines a school’s achievement and growth score. It is THE test used to calculate an (albeit invalid) TVAAS score for teachers. It is THE test used in student report cards (when the quick scores come back on time). This is THE test.

Teachers are being asked RIGHT NOW to make choices about the achievement measure they will be evaluated on for their 2017-18 TEAM evaluation. One choice: THE test. The TNReady test. But there aren’t results available to allow teachers and principals to make informed choices.

One possible solution to the concern expressed by the Murfreesboro School Board is to press the pause button. That is, get the testing right before using it for any type of accountability measure. Build some data in order to establish the validity of the growth scores. Administer the test, get the results back, and use the time to work out any challenges. Set a goal of 2019 to have full use of TNReady results.

Another solution is to move to a different set of assessments. Students in Tennessee spend a lot of time taking tests. Perhaps a set of assessments that was less time-consuming could allow for both more instructional time and more useful feedback. I’ve heard some educators suggest the ACT suite of assessments could be adapted in a way that’s relevant to Tennessee classrooms.

It will be interesting to see if more school districts challenge the Department of Education on the current testing situation.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

What’s in the Water?

At some schools in Nashville, the answer is unacceptable amounts of lead.

Phil Williams reports:

A NewsChannel 5 investigation discovers information potentially affecting the health of school children across Nashville — information that has not been shared with parents.

It reveals children are still drinking lead-contaminated water when they go to school — despite the district’s assurances that there’s nothing to worry about.

This past summer, Metro Schools tested every water fountain in the district after questions raised by NewsChannel 5 Investigates.

As the school year started, officials only shared the worst results with the public.

But we obtained the raw data, which shows there’s a lot more to the story.

Doctors and health officials suggest MNPS needs to do more — which may involve replacing contaminated pipes and finding alternative water sources in the meantime.

Read more about the risks posed and the challenge of addressing this problem.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Disappointing

That’s the word from Commissioner of Education Candice McQueen in response to a refusal by both Shelby County and Nashville school districts to hand over student data.

As the Data Wars continue, Chalkbeat reports on McQueen’s reaction:

“We are disappointed that these districts are choosing to withhold information from parents about the options that are available to their students while routinely saying they desire more parental engagement,” she said. “Allowing parents to be informed of their educational options is the epitome of family engagement and should be embraced by every school official.”

McQueen seemed to indicate that firmer consequences could lie ahead. “We must consider all options available in situations where a district actively chooses to ignore the law,” she said in the statement. McQueen told lawmakers in a conference call last month that she was not discussing withholding state funds as a penalty at the time, according to Rep. John Clemmons, who was on the call.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

TC Talks Chattanooga

Nashville-based education blogger TC Weber takes some time to explain a bit more about what’s happening with Chattanooga and the state’s Achievement School District in a recent post.

Here’s how he explains what’s happening since the threat of an ASD expansion team in Hamilton County became more real:

Let’s take a quick trip down to Chattanooga where last night a historic vote took place. The Hamilton County School Board voted 7 -2 to continue the conversation about creating a partnership zone with the Tennessee Department of Education. In case you are not familiar with the Partnership Zone plan, it’s the latest quick fix scheme developed by the TNDOE because people have started to catch on to the dumpster fire that is the Achievement School District. Under the Partnership Zone plan, both the county and the state would work together to improve underperforming schools in the district.

The plan calls for the a creation of an appointed board that would oversee the Partner Zone. This creates a bit of a conundrum. Under current law, schools governing boards can only be elected entities. So this would require a change in legislation. A change that could open a virtual pandora’s box because what’s to stop other districts from switching to an all appointed board, a hybrid, or turn control over to the mayor or other appointed officials?

The term partner is a little bit of a misnomer. The state is making it perfectly clear who wears the pants in this relationship right from the out set. The HCS Board was told that they could choose not to pursue the “Partnership Zone” but if they didn’t State Superintendent Candice McQueen would take all 5 of the priority schools plus two more schools and dump them in the Achievement District. If this is in fact a threat she was prepared to follow through with, it’s a little troubling and a clear sign that she’s willing to play politics with kids. The ASD is an unmitigated failure that should be ended this legislative session not used a stick to ensure district compliance.

As Weber points out, McQueen is using the threat of aggressive state action (takeovers, fines) to attempt to get her way lately. So far, that has not resulted in yielding in Nashville or Memphis. It will be interesting to see how the Partnership Zone plays out in Chattanooga.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

The Data Wars: A New Hope?

The ongoing Data Wars between the state’s two largest school districts and the Tennessee Department of Education continue, with today being the deadline set by Commissioner Candice McQueen for districts to hand over the data or face consequences.

Yesterday, Anna Shepherd and Chris Caldwell, chairs of the Boards of Education in Nashville and Memphis respectively, penned an op-ed detailing their opposition to the data demand from McQueen.

They wrote:

Tennessee Education Commissioner Candice McQueen has demanded that Metro Nashville Public Schools and Shelby County Schools surrender personal contact information for a large number of students and families in our school systems, which represent approximately 20 percent of Tennessee’s K-12 public school students.

Her argument: A new state law requires us to hand over personal information to ASD charter schools so these taxpayer-funded private schools can use the data to fill thousands of empty seats by recruiting students away from public schools.

In addition to violating student and family privacy — the right to privacy is a fundamental American principle — the problem with McQueen’s data demand is this: The ASD now is universally viewed as a failed experiment in education reform.

Shepherd and Caldwell contend that their district’s students will not be well-served by marketing efforts from charter schools operating under the banner of the Achievement School District:

Instead, McQueen proposes to shift the cost burden of the failing ASD to local taxpayers in Memphis and Nashville. She wants to confiscate our student data and information in order to stage marketing raids on our schools — which would redirect local taxpayer funds to the ASD and its charter operators at the expense of our school systems.

With today’s deadline looming, it appears school leaders in Memphis and Nashville are locked down against releasing the data demanded by McQueen. Should that position hold, the question will be: What will McQueen do about it? Will she unleash her ultimate weapon and withhold state funds from these districts as punishment?

For more on education politics and policy in Tennessee, follow @TNEdReport