Wrong Answer

In the never ending saga that is testing in Tennessee, the latest chapter spins a familiar but frustrating tale. It seems the state’s testing vendor incorrectly scored thousands of TNReady tests, impacting student score reports and teacher evaluation scores based on those student scores.

Jennifer Pignolet and Jason Gonzales have more:

About 9,400 TNReady tests across the state were scored incorrectly, according to the Tennessee Department of Education.

The scoring issue impacted about 70 schools in 33 districts. Just over 1,000 of the incorrectly scored tests were in Shelby County Schools, according to an email from Superintendent Dorsey Hopson to his board on Friday.

Approximately 1,700 of the total incorrect tests scores, once corrected, changed what scoring category that test fell into, possibly affecting whether a student passed the test.

The error also impacted value-added scores for up to 230 teachers. A separate problem could impact TVAAS scores for as many as 900 teachers.

The scope of the error means scores in nearly 25% of the state’s school districts will need to be corrected. The Department of Education says the testing vendor, Questar, is re-scoring the tests.

UPDATE — Here’s a list of districts impacted:

  • Achievement School District
  • Anderson County
  • Benton County
  • Bradley County
  • Bristol City
  • Carter County
  • Cocke County
  • Collierville City
  • Crockett County
  • Davidson County
  • Elizabethton City
  • Giles County
  • Hamilton County
  • Hardin County
  • Henry County
  • Huntingdon Special School District
  • Jackson-Madison County
  • Knox County
  • Lewis County
  • Lincoln County
  • Marshall County
  • Maryville City
  • Monroe County
  • Montgomery County
  • Obion County
  • Putnam County
  • Roane County
  • Rutherford County
  • Shelby County
  • Smith County
  • Sumner County
  • Union City
  • Weakley County

The State of Tennessee has spent millions of dollars on a new testing regime supposedly better able to assess student mastery of state standards. So far, all most students, teachers, and parents have seen is problems.

The first set of problems happened on day one of the initial online administration of the test in 2016. Then, a series of missed deadlines led to the state firing then-vendor Measurement, Inc. That’s the same company that hired test scorers via ads on Craigslist.

Of course, this is the same Department of Education that has repeatedly had issues with test score data.

If only there had been warning signs or calls to take the time to phase-in TNReady so that it best serves students and educators.

You know, something like:

TNReady is measuring different skills in a different format than TCAP. It’s BOTH a different type of test AND a test on different standards. Any value-added comparison between the two tests is statistically suspect, at best. In the first year, such a comparison is invalid and unreliable. As more years of data become available, it may be possible to make some correlation between past TCAP results and TNReady scores.

Or, if the state is determined to use growth scores (and wants to use them with accuracy), they will wait several years and build completely new growth models based on TNReady alone. At least three years of data would be needed in order to build such a model.

That’s from an article I wrote in March of 2015 about TNReady data and the challenges of adapting to a new test using our current accountability system.

That was BEFORE the 2016 TNReady mess. It was before the state had a problem getting data back this year.

How many warning signs will be ignored? How important is the test that it must be administered at all costs and the mistakes must be excused away because “accountability” demands it?

How can you hold students and teachers and schools accountable when no one is holding the Department of Education accountable? How long will legislators tolerate a testing regime that creates nightmares for our students and headaches for our teachers while yielding little in terms of educational value?

At least one school board has complained about the state’s handling of TNReady data this year. I suspect more will follow in the wake of this latest mistake.

So far, TNReady has sent one clear message: Accountability is a one way street in Tennessee and students, teachers, and districts are on the wrong end.

 

For more on education politics and policy in Tennessee, follow @TNEdReport


 

 

WTF, TNReady?

In what is sure to be surprising news to parents, students, and school district leaders, TNReady quick score results are delayed yet again. Hard to predict this type of disaster since it only happens every year for the past four years now.

This delay is due to a problem the vendor is having with new test scanning equipment.

Here’s a statement from Hamilton County Schools explaining the news they received from the Tennessee Department of Education:

Tennessee Education Commissioner Dr. Candice McQueen informed school directors state-wide today that TNReady assessment vendor Questar has experienced challenges with the scanning software and capacity to scan testing materials; therefore, the company will not be able to meet the raw score deadlines set.

Dr. McQueen has asked Questar for updates on district-specific timing, and says the company currently estimates the scores will be delivered for the state’s upload process no later than the week of June 12.

Questar released a statement, which I have attached to this email.

The message from Dr. McQueen says she plans to meet with several groups this summer, including the TOSS Board, to come up with solutions that may prevent Tennessee testing vendors from missing the deadline set for report cards for a third year in a row. Her office promises to keep districts posted as the state receives more information.

To their credit, Questar is taking responsibility. That’s a change in tone from previous testing vendor Measurement, Inc.

Here’s what Questar had to say:

A Statement from Questar Regarding Raw Score Delivery

Questar committed to spring TNReady raw score delivery on a timeline and terms we set and which were communicated extensively by the Tennessee Department of Education (TDOE), however, our company has experienced scan set-up difficulties that will affect that timeline. Questar recently completed an upgrade to our scanning programs for paper tests, and we underestimated the time needed to fully integrate these upgrades. Therefore, any delays to the timeline are not the fault of the Tennessee Department of Education or the districts across the state. We take complete responsibility for this project, and we are actively working to meet our commitments to the state, districts, and educators. The purpose of this communication is to provide an update on the progress we are making to publish raw scores for grades 3-8 and End-of-Course assessments.

End-of-Course Status Update: Nearly all paper and online EOC raw scores are uploaded to EdTools. We appreciate the communication from the TDOE to reprioritize EOC delivery to ensure districts had these results as soon as possible.

3-8 Status Update: Questar is currently working through delays in scan processing for the 3-8 raw scores, which impact the timeline we previously provided to the TDOE for raw score delivery. We have experienced several software delays with our scan programming that we are actively resolving, and we are working to increase capacity within our scanning operations. We are committed to working with TDOE to mitigate these challenges in order to scan and score TNReady 3-8 assessments as quickly as possible. We will provide additional details and timelines, including district-specific estimations, as we make progress over the week.

The raw score delays will not impact the timeline for delivery of final score reports. Our scoring operations are unaffected by the scanning software issues, and we anticipate providing EOC final score reports this summer and 3-8 score reports this fall after the cut score process is complete, as previously announced.

We understand the importance of having the raw scores to communicate information to educators, students, and families, and we apologize for the inconvenience our delays have caused TDOE and our district partners in getting this information on the timeline we committed to months ago.

Questar sincerely appreciates our partnership with TDOE and the Tennessee educators and students we serve.

Tennessee has unique timing requirements for raw score delivery as compared to other states so we will continue to work with the department and districts to ensure that the future score return process is successful.

-Brad Baumgartner, Chief Partner Officer

At least we’re now paying millions of taxpayer dollars to a vendor who will take some heat, right?
This part of the Hamilton County statement is interesting:
The message from Dr. McQueen says she plans to meet with several groups this summer, including the TOSS Board, to come up with solutions that may prevent Tennessee testing vendors from missing the deadline set for report cards for a third year in a row.
Here’s a solution: Don’t count TNReady scores in student final grades. In fact, since McQueen readily admits that raw score data is of little actual value and since different methods of factoring quick scores lead to vastly different scores assigned to students, why even bother?
It’s clearly been a problem to get reliable quick score data back to districts. The data is of dubious value. Legislation should advance that eliminates the use of TNReady data in student grades.
As I’ve said before, TNReady may well be a solid test properly aligned to standards. Such a test can yield useful information. A more student-centered approach to testing would focus on project-based assessment and use tools like TNReady to provide diagnostic information for students, teachers, and schools.
For more on education politics and policy in Tennessee, follow @TNEdReport


 

 

An $18.5 Million Emergency

As a result of the failure of Measurement Inc. to deliver on its TNReady promises, the State of Tennessee has awarded a contract to Pearson to grade tests completed by students this year, including high school EOC tests and Part I tests that were completed. The contract pays $18.5 million and the estimated completion date for grading is December.

Grace Tatter has the details:

The state’s contract with Pearson goes through December for scoring and reporting of 2015-16 assessments, including high school exams, Part I grade 3-8 tests, and any completed Part II grade 3-8 exams.

Now, to be clear, the “emergency” is that some students completed tests that weren’t graded and won’t be graded by Measurement Inc. because they were fired.

What about the fact that some tests were completed online and others were completed on paper? Never fear, the state’s data team has a plan:

Measurement Inc. already has scored high school exams completed online last fall for students who are on block schedules. Assistant Education Commissioner Nakia Townes said the state will use a formula to ensure that those scores are comparable to the scores of tests completed on paper, and to be graded by Pearson, this spring.

So, as a result of this new contract, there will be two different vendors grading the same test as well as some tests completed in an online format and some on pencil and paper.

Oh, and the results are due back in December. Well past time to have much value to inform instruction or help parents or students understand areas of deficiency.

Instead of spending $18.5 million on grading these tests which will have limited usefulness, the state could use that money to fully develop and pay for portfolio assessment at the district level for related arts and other non-tested teachers.

It could also use some of that money to support the unfunded mandate of RTI2.

Or, it could spend a portion of that money on developing an alternative assessment regimen — perhaps incorporating project-based assessment and reducing the reliance on standardized testing. Maybe even finding ways to reduce total testing time. Or, develop an assessment waiver as allowed under the new ESSA.

Out of crisis can come opportunity – and we have an opportunity and some unspent funds that could be used to develop better, more student-focused solutions going forward.

Instead, we’re handing money to Pearson and trying to get back to business as usual as soon as possible.

Rest assured:

…the department plans to select a new vendor in June to develop and administer next year’s state assessment.

For more on education politics and policy in Tennessee, follow @TNEdReport

Pearson: We’re Ready to Grade

While the Tennessee General Assembly forced a move away from Pearson as the state’s testing vendor in 2014, the familiar company (who delivered and scored TCAP for many years) is now back and will provide the grading for the TNReady high school tests.

Jason Gonzalez at the Tennessean notes:

The Tennessee Department of Education has contracted with its previous test vendor Pearson Education in an emergency maneuver to score TNReady high school tests.

The contract with Pearson is only for scoring and reporting of 2015-2016 assessments, according to Education Commissioner Candice McQueen in a letter Monday to school directors statewide.

The new contract is necessary because the Department of Education the state’s testing vendor, Measurement Inc. due to a failure to deliver the TNReady test as desired.

While high school tests will be scored, it’s not clear there will be any scores provided to students in grades 3-8 who completed Part I of the new assessment.

WATE in Knoxville reports:

So, the blank tests remain in the Durham warehouse. In another MI warehouse about 15 minutes away, there are thousands more boxes; they’re filled with Part I of the TNReady test. If your child took that test, chances are good that it’s just sitting in that warehouse. Scherich said those have all been scanned into MI’s system, but because the DOE cancelled the contract and, according to Scherich, never paid six months of invoices, it’s possible they’ll never be scored.

The state is currently seeking a vendor to both deliver and score the TNReady assessment in 2016-17.

For more on education politics and policy in Tennessee, follow @TNEdReport

 

 

Why TNReady Wasn’t

Grace Tatter over at Chalkbeat has an informative interview with the President of Measurement Inc., the company charged with delivering TNReady this year.

As I read the interview, a couple of items stood out. First, the company had never delivered an entire state’s online testing program. Tatter notes:

It was also an unprecedented task for Measurement Inc., which had never before developed and delivered a state’s entire online testing program.

Despite this, they somehow won the bid to deliver Tennessee’s program.

Second, the magnitude of the failure. Tatter:

About 48,000 students logged on that day, and about 18,000 submitted assessments. It’s unknown the number of students who weren’t having troubles with the test, but stopped after McQueen sent an email instructing districts to halt testing.

“It was a failure in some respects because we were supposed to design a system that would take 100,000 students in at one time… We had a problem with 48,000,” Scherich said.

Read that again. Measurement Inc. was tasked with developing an online platform that could handle 100,000 students taking a test at the same time. The system they developed couldn’t handle 48,000 students. They didn’t even develop a system that could handle HALF of what they were contracted to provide.

The company president goes on to detail the challenges of printing the tests in a short timeframe. However, back in February, Education Commissioner Candice McQueen expressed confidence in the printed tests:

“I want to stress to you that the paper version of TNReady is still TNReady,” McQueen wrote of the new test aligned to the state’s current Common Core academic standards.

She said the paper tests are being shipped to each district at no additional taxpayer cost.

Unfortunately, it wasn’t. Phase I tests did arrive, albeit quite late. And Phase II tests were not delivered in time to be administered this year.

Now, the state is seeking another vendor who can deliver the test in the 2016-17 academic year.


 

 

For more on education politics and policy in Tennessee, follow @TNEdReport

 

Ready for What’s Next?

Following yesterday’s announcement that Measurement Inc. had been fired and TNReady testing suspended, Commissioner of Education Candice McQueen sent an email to teachers explaining the decision and providing information for what’s next.

The email included a line with a bit of an apology: You were ready, and I am sorry that we were unable to provide all students with a consistent and complete testing experience this year.

Because, of course, all students want a consistent and complete testing experience.

Here’s the email in it’s entirety:

Earlier today, I announced that the department is terminating its statewide assessment contract with Measurement Inc., effective immediately. The state awarded Measurement Inc. a contract in November 2014 to provide an online testing platform with a paper and pencil backup. Not only did Measurement Inc. fail to deliver a reliable online platform for students across the state, it has also failed to print and ship paper tests by deadlines the company had set. Terminating our contract with Measurement Inc. was a challenging decision because we’ve been working to honor the effort and investment of Tennessee teachers and students, but we’ve exhausted every option in problem solving with this vendor to get these tests delivered. TNReady was designed to provide Tennessee students, teachers, and families with better information about what students know and understand, and the failure of this vendor has let down the educators and students of our state.

As a result of repeated failures from this vendor, we are suspending Part II of TNReady for grades 3-8 this year. However, because districts have already received all high school testing materials, high school testing will move forward as planned.

While Measurement Inc. had previously assured us it would have all grade 3-8 materials delivered by April 27, the third deadline the company had set and missed this month, 100 percent of districts are still waiting on some 3-8 materials to arrive, and few districts have complete sets of tests for any grade or subject. Last week, the department told districts that we would not ask them to continue to change schedules and extend the testing window beyond May 10. We know the transition to TNReady has presented unexpected challenges for educators, schools, and districts, and we will not ask you to further disrupt your end of year schedule.

Many of you have shared with me that, despite the challenges with implementation this year, you were excited for your students to show what they were able to do on a new assessment. I understand that many of you will share in my disappointment that we won’t have detailed score reports from this year’s assessment for students in grades 3-8. We will provide as much information to schools and students as possible related to results from Part I testing for grades 3-8. This will be used for informational purposes only, and no score will be associated with Part I for grades 3-8. High school tests will be scored, and these results will be shared in the fall.

The transition to a new assessment this year has been challenging, but aligned assessments are critical to ensure that all students are making progress on their path to postsecondary opportunities and the workforce. I have seen first-hand how hard you have worked to align your instruction to our newer, more rigorous standards. You and your students have risen to higher expectations, and I hope you are encouraged by the growth in critical thinking and problem solving that you have seen in your classrooms every day. You have been flexible and patient beyond what we should expect, especially as you planned for Part II amid important field trips and end-of-year celebrations. You were ready, and I am sorry that we were unable to provide all students with a consistent and complete testing experience this year.

I recognize that our apology is not enough. You also deserve clear guidance as we look ahead to next year. We know teachers’ jobs don’t stop when the final bell rings in May. You spend a great deal of time in the summer months planning for the following year. We’ll share updated math and ELA blueprints in May, which will help you plan for the coming year.

While navigating the challenges of this year’s administration of TNReady, we’ve also been working to improve our assessments for next year. Earlier this month, we shared with districts several changes to TNReady for next year. We’re eliminating Part I for the math assessment, and we will include one to three math problems on Part II called integrated tasks that will maintain the rigor of the performance tasks. As a result of eliminating Part I for math, we will be able to reduce overall testing time for math in grades 3-11. Additionally, we’re adding multiple choice items to ELA Part I, which will encourage students to closely engage with reading passages before they write their constructed response. This will reduce the number of items on ELA Part II and decrease the overall testing time for ELA. These changes were made in response to feedback we received from teachers, as well as school and district leaders, and we’re dedicated to continuously improving our assessments as we move forward with a new assessment vendor.

The department is currently working with the state’s Central Procurement Office to expedite the selection of a vendor for both the scoring of this year’s high school assessment and the administration of next year’s test. In the meantime, I encourage you to read our new frequently asked questions document (here), which offers more detailed information about how this latest announcement will affect you, your schools, and your districts. We will continue to update this resource. It’s also important to note that today’s announcement doesn’t change the flexibility that has already been provided on teacher evaluation. If a teacher has 2015-16 TNReady data, in this case high school teachers, TNReady data will only be used if it helps the teacher.

Thank you for your hard work and dedication to your students. I appreciate your patience and understanding as we look ahead to the 2016-17 school year.


 

 

For more on education politics and policy in Tennessee, follow @TNEdReport

Not Ready at All

It turns out TNReady wasn’t ready at all. From a technological failure on day one to a shipping delay in March to all out chaos in April, the transition to the new test has been, in the words of one Director of Schools, an “unmitigated disaster.”

As of today, in the face of yet another delay, the Tennessee Department of Education has terminated the contract with Measurement, Inc. and suspended further testing in grades 3-8 for this year.

Here’s the full press release from the Department of Education:

Education Commissioner Candice McQueen announced today the department will terminate its statewide testing contract with Measurement Inc., effective immediately. While high school testing will continue as planned, the state will suspend grade 3-8 testing during the 2015-16 school year due to Measurement Inc.’s inability to
deliver all testing materials.
After revising their shipping schedule for a third time this month, the state’s testing vendor, Measurement Inc., failed to meet its most recent deadline. As of this morning, all districts were still waiting on some grade 3-8 materials to arrive with a total of two million documents yet to be shipped. In February, the department was forced to move from the originally planned online assessment delivery to a paper-based format due to the failure of the vendor’s online platform.
“Measurement Inc.’s performance is deeply disappointing. We’ve exhausted every option in problem solving with this vendor to assist them in getting these tests delivered,” Commissioner Candice McQueen said. “Districts have exceeded their responsibility and obligation to wait for grade 3-8 materials, and we will not ask districts to continue waiting on a vendor that has repeatedly failed us.”
If districts have received materials for a complete grade or subject in grades 3-8 (i.e. fifth-grade math), they will have the option to administer that specific grade or subject level; however, the department will only be able to deliver limited student performance information for these particular grades and subjects. High school tests will be fully scored, and these results will be delivered later this fall.
“Challenges with this test vendor have not diverted us from our goals as a state. Tennessee has made historic and tremendous growth over the past several years. Higher standards and increased accountability have been a key part of this progress,” Commissioner McQueen said. “Our work toward an aligned assessment plays a critical role in ensuring that all students are continuing to meet our high expectations and are making progress on their path to postsecondary
and the workforce.”
Flexibility that has already been provided for teacher evaluation through recent legislation will remain. If a teacher has TNReady data, in this case high school teachers, TNReady data will only be used if it helps the teacher. If a teacher does not have TNReady data, their evaluation will rely on data from prior years.
The department is currently working with the state’s Central Procurement Office to expedite the selection of a vendor
for both the scoring of this year’s high school assessment and the development of next year’s test.  The department has also been in close contact with the United States Department of Education to ensure that Tennessee is in compliance with federal requirements and will continue to work with them on this issue.
TNReady, the state’s new assessment in math and English language arts in grades 3-11, was designed to be administered in two parts. Part I was given in late February and early March, and Part II was scheduled to begin on April 25.
Additional resources can be found in this frequently asked questions document:
https://tn.gov/assets/entities/education/attachments/tnready_faq_suspension.pdf.

MORE on Measurement, Inc.:

Replacing TCAP

TNReady Already?

Ready to Grade?

For more on education politics and policy in Tennessee, follow @TNEdReport

Seriously?

Will TNReady testing resume this year? For some students, maybe not.

The President of Measurement, Inc. said yesterday that there was no guarantee his company would make the testing window.

The Memphis Daily News reports:

The president of a North Carolina-based testing company said Monday that he can’t guarantee all students in Tennessee will receive the test on time.

Measurement Inc. president and CEO Henry “Hank” Scherich said his company is working furiously to get the new TNReady materials to students.

“I wish I could promise them,” Scherich said. He added they were doing everything humanly possible to get the tests to the students on time.

All of the students have at least some of the testing materials, he said, but the company has found itself scrambling to print and ship 5 million test booklets for Tennessee.

This follows last week’s  that a Friday deadline would be missed.

That event caused some lawmakers to call for this year’s testing to be cancelled. The Department of Education has still not agreed to that solution.

For more on education politics and policy in Tennessee, follow @TNEdReport

 

 

Not Our Fault

Measurement, Inc., the state’s vendor for the TNReady tests is saying it’s not their fault that for the third time in a row, the company has failed to deliver a testing product.

The failure has lawmakers and other critics calling for the test to be stopped and for Measurement, Inc. to be fired.

The Department of Education said:

“We share our districts’ frustration that we do not know specific delivery timelines due to [Measurement Inc’s] failure to provide shipping projections and find this lack of information extremely unsatisfactory,” spokesperson Ashley Ball said in a statement.

But the company’s president responded:

“You just can’t take the test off line and put it on a printing press,” President Henry Sherich said by phone Friday. “We’re not failing to deliver. We are delivering as fast as possible.”

Sherich revealed his company is only working with one printer as other printers they work with are booked. This after a delay in delivering Phase I of the tests in March.

Sherich didn’t offer an apology or express concern for the students, parents, and teachers who have suffered as a result of this delay.

While the Department of Education has said it will be flexible with districts as they respond to this new delay, they have not yet said they plan to fire Measurement, Inc.

For more on education politics and policy in Tennessee, follow @TNEdReport

 

A Modification

As we reported yesterday, Phase II of TNReady is not so ready. In fact, Grace Tatter reports that the problem is statewide, impacting grades 3-8.

Tatter cites an email from the Department of Education indicating the state is not sure when the Phase II tests will be delivered to districts.

The email also says:

“…Districts may modify their testing schedules as needed, without any prior approval or notice to the (state),”

The thing is, some districts have already been trying to modify their schedules by not giving the test at all. The idea of refusing to administer Phase II surfaced in Murfreesboro in late March and early April. The state responded by issuing a vague threat regarding withholding BEP funds.

Tullahoma City Schools on Monday approved a resolution unanimously calling on the state to cancel testing for the remainder of this year.

All of this was before the realization that Phase II tests would not make it to Tennessee districts on time. Now, though, the Department of Education’s own words suggest that districts may modify as they see fit without consulting the state. One possible modification would be to not administer the test at all. Another would be to schedule it for a time in June when students aren’t in school. Districts could say they offered the test, but no one showed up to take it.

The state has also made a big fuss about what happens to students/districts if students simply refuse to take the test. Trouble is, the state’s memo is based on some pretty fuzzy reasoning.

As this piece was being written, the Department of Education announced it will not ask districts to reschedule tests beyond the current testing window, which expires on May 10th. That means if materials are not received in time for administration by that date, districts don’t have to administer the tests. The Department also indicated it would provide additional flexibility to districts.

From Jason Gonzales:

The Tennessee Department of Education announced to districts Friday it won’t reschedule the TNReady testing window again this year and for those districts that don’t receive tests on time, will provide flexibility.

“We will not ask districts to reschedule again beyond what has been communicated to date, and we will not extend the testing window beyond May 10,” according to a statement sent to districts Friday.

So, what’s next? Will the state cancel the contract with testing vendor Measurement, Inc.? Will Commissioner McQueen assume responsibility for the failed transition to a new test?

Only time will tell, and there’s not much time left.

 

For more on education politics and policy in Tennessee, follow @TNEdReport