If this is what success looks like…

In a story about the Tennessee Department of Education scaling back the requirements for online testing next year in light of this year’s testing challenges, this caught my attention:

Even with the problems this year, it was one of the most successful online administrations for the state to date. More than 2.5 million TNReady tests were administered this spring, with about 300,000 students taking the test online. Only high school students were required to take the online version this year.

What does the word “success” mean? Because my recollection of this year’s TNReady administration is that it was a debacle.

I’m not the only one. As I noted last week:

While lots of states are moving to online testing, one expert says Tennessee is unique:

“I’m not aware of a state that has had a more troubled transition” to online testing, said Douglas A. Levin of the consulting group EdTech Strategies.

And there’s this helpful explainer:

Why is Tennessee in the unique position of having the worst online testing transition in the country?

The reality is that Tennessee’s online-testing mess has left everyone in a difficult position, said Chad Aldeman, a principal at Bellwether Education Partners, a consulting organization.

“The state has not [made] stability a key priority in their testing vendors,” Aldeman said.

Nevertheless, Commissioner of Education Candice McQueen says:

The state will put out a request for contract proposals in the fall, with a new vendor to be identified in the spring. Questar Assessment could again win the contract, but McQueen said who wins the proposal will have to show the ability and history of seamlessly administering an online test.

“We look for a company with a track record of success in administering online testing and who can manage our test well.”

Haven’t we heard that before?

For more on education politics and policy, follow @TNEdReport

Keep the education news coming!


 

Definitely Something Wrong

The Texas Tribune reports:

A couple of weeks after Texas penalized its main testing vendor over glitches with thousands of standardized tests, another potential testing mishap is under investigation after more than 100 students in a high-performing Houston-area high school received zeros on their English essays.

Valerie Vogt, chief academic officer at Lamar Consolidated Independent School District, said she was confused this spring when about 157 students at George Ranch High School, which generally performs higher than state average on standardized tests, received zeros on their English 1 and English 2 essays. In the other four high schools in the district, just 10 or fewer students received zeros on the essays.

“There’s definitely something wrong,” she said.

The testing vendor responsible is Educational Testing Services (ETS), which owns Tennessee’s testing vendor, Questar. This is the latest in a series of problems with ETS in Texas:

Last month, the TEA levied a $100,000 penalty against ETS after tens of thousands of Texas students were kicked out of the testing software or encountered connection problems when taking computerized tests in April and May. The agency also announced it would throw out the scores of students who experienced those glitches and reduce their effect on state accountability ratings for schools and districts.

Tennessee’s Department of Education announced recently ETS would be taking over more responsibility for TNReady after Questar’s administration of the testing this year was plagued by hackers and dump trucks.

Of course, ETS is not without a history of test administration problems. Edsurge.com notes:

The changes highlight a possible strategic shift for ETS whose reputation came under fire last year when the nonprofit had to pay $20.7 million dollars in damages and upgrades after multiple testing problems in Texas.

A recent analysis of the transition to online testing in the states indicates it is going well in most places, with Tennessee being the one glaring exception.

So, of course Tennessee hires the parent company of Questar — a company that has experienced consecutive years of testing problems in Texas — to come in and … make things right?

Yep, there’s definitely something wrong.

For more on education politics and policy in Tennessee, follow @TNEdReport

Your support keeps the education news coming!


 

You’re Fired….uh, Hired!

The Chattanooga Times Free Press notes that Governor Bill Haslam and Education Commissioner Candice McQueen are considering ending the state’s relationship with Questar:

Gov. Bill Haslam said the state is conducting an independent review of its current contractor running the problem-plagued TNReady student testing system and, depending on its findings, the company could be out of the picture once its current contract ends in November.

The likely replacement for Questar is Education Testing Service (ETS):

McQueen said that in addition to the state’s third-party review of Questar’s operations, the state is already going to move “all of our test development and design” to Educational Testing Services, which she said has a “reputation for very high quality work.”

Sounds great, right? Firing the vendor that was baffled by hackers and dump trucks and replacing them with a group with a solid reputation.

Except for just one thing:

Education Testing Services, the global billion dollar nonprofit that administers more than 50 million tests (including the GRE and TOEFL) across the world, recently sealed an agreement to acquire Questar, a Minnesota-based for-profit testing service, for $127.5 million. According to the press release, Questar will become a separate for-profit subsidiary of ETS.

Questar offers what they describe as a “fresh and innovative” method of testing for grades 3-8—providing states with summative assessments, design support, scalable technological innovation, administrative help, scoring and reporting services.

Ok, so maybe ETS will step in and give its baby brother Questar some guidance going forward? Well:

The changes highlight a possible strategic shift for ETS whose reputation came under fire last year when the nonprofit had to pay $20.7 million dollars in damages and upgrades after multiple testing problems in Texas.

Let’s get this straight: Governor Haslam and Commissioner McQueen think no one in Tennessee understands Google? They are “firing” the company that messed up this year’s testing and hiring a new company that owns the old one and that also has a reputation for messing up statewide testing.

Solid move.

For more on education politics and policy in Tennessee, follow @TNEdReport

Keep the education news coming!


 

YOU’RE FIRED Rubber Stamp over a white background.

WRONG: Another TNReady Story

Today’s TNReady story is a tale of the wrong test.

Reports from multiple districts indicate that students have been given tests for the wrong grade level today, particularly in ELA.

This was reported some last week, as well, with the biggest instance noted in Anderson County.

In an update to parents last week, Williamson County noted this type of testing mix-up had happened there as well.

Even when there’s not a widespread outage, as has happened due to dump trucks and hackers, there are problems with the execution of this test.

As Chalkbeat reports, the problem with yesterday’s test (not blamed on aliens, surprisingly) means the text-to-speech feature is turned off today. That means students requiring this feature will not be able to test:

Districts were told late in the day that the culprit was traced to a feature allowing text to be turned into speech for students needing audible commands.

“The problems presented by this feature impact the system for districts across the state, regardless of how many of your students are using text-to-speech,” said an email to superintendents from the state Department of Education.

No word on when students needing this feature will be able to test.

As I said yesterday, it’s amazing to me that this test has not been stopped. When will our Department of Education put students first?

 

For more on education politics and policy in Tennessee, follow @TNEdReport

Keep the education news coming!


 

 

Enough Already

Today, amid another round of testing problems, Williamson County Director of Schools Mike Looney tweeted:

Testing Update: TnReady testing challenges persist this morning. This has been the worst state testing process I have ever seen and it’s beyond ridiculous! Nevertheless, I am proud of WCS students and teachers for handling this with grace.

We’re now in the third week of the TNReady testing window and we’ve seen problems of some sort on a majority of those days. In fact, last week, Williamson County posted a list of TNReady problems by day:

Monday, April 16: Login problems affecting approximately 15,000 students.

Tuesday, April 17: Login problems affecting approximately 8,000 students.

Wednesday, April 18: WCS suspended testing to give the TDOE time to correct problems.

Thursday, April 19: Login problems affecting approximately 1,000 students.

Friday, April 20: No significant issues reported.

Monday, April 23: No significant issues reported.

Tuesday, April 24: System defaults caused 100+ students to take the wrong grade level test.

Wednesday, April 25: Delays and canceled testing affecting approximately 8,000 students.

Thursday, April 26: System lockout affecting approximately 15,000 students.

Friday, April 27: No significant issues reported.

That’s just one district, and the problems have been reported by a number of districts large and small across the state.

The Department of Education has blamed the problems on mysterious forces such as hackers and dump trucks, but it seems clear testing vendor Questar is not quite prepared for the job Tennessee is paying them $30 million this year to do.

Oh, and sometimes students get the wrong test.

All of this has caused an outcry among students, parents, and teachers. While one legislator says all the “whiners” should just suck it up, the TNReady trouble this year has caused legislators to take matters into their own hands, passing bills on “holding harmless” and “adverse actions” in order to make clear these tests should not negatively impact teachers, students, or schools.

One might recall that before our state’s testing window started, there was a bit of a warning that trouble might be headed our way. Despite the signs of potential trouble, Questar and the TN DOE expressed confidence in TNReady:

State officials said Thursday they are confident the new digital platform will work under heavy traffic, even as their new testing vendor, Questar, had headaches administering computer-based tests in New York on Wednesday. Some students there struggled to log on and submit their exam responses — issues that Questar leaders blamed on a separate company providing the computer infrastructure that hosts the tests.

It seems that confidence was misplaced.

I’ve talked with testing coordinators who tell me districts will be testing all the way up until graduation. I heard today that even when the login and submission problems were “resolved,” some students returned to their computers only to be issued a test for a subject other than the one they had started earlier in the day.

Student answers have been deleted or lost. Because of the legislation passed at the end of legislative session, TNReady will likely not count in many student’s grades. Teachers and administrators report that whether the scores count or not, students have no confidence in the system and no longer take the test seriously.

Even today, as I began seeing reports of issues around the state, I realized that TNReady being down is no longer news, it’s the norm.

Of course, Tennessee has had some sort of problem with testing or test results for five years now, dating back to the last administration of TCAP.

Here’s what else I realized: This test will just keep going. No one will stop it. Governor Haslam has yet to seriously weigh-in and appears to be fully behind Commissioner McQueen despite years of testing failures. While Directors of Schools complain about the ridiculous excuses from DOE and poor execution from Questar, so far, no district has permanently suspended testing.

Representatives from the Department of Education told lawmakers last week that there will be some valid data from this administration of TNReady. They even said it with a straight face.

So, why won’t this stop? Will any district refuse to subject students to further testing in this environment? Who will finally stand up and say “Enough?”

For more on education politics and policy in Tennessee, follow @TNEdReport

Keep the education news coming by becoming a Patron!


 

 

BREAKING: Another Dump Truck Hits TNReady?!

Tennessee’s failed testing system, TNReady, is experiencing problems yet again. Last week, the Department of Education alleged a dump truck took out the test.

That was the latest in a line of testing challenges during this year’s administration of TNReady.

This morning, eleven systems so far are reporting login and/or submission problems with TNReady. Knox, Putnam, Campbell, Sequatchie, Rutherford,  Robertson, Williamson, Montgomery, Collierville, Sumner, and Cumberland have reported issues with the test today so far.

UPDATE: 9:59 AM — It appears the TNReady problems are widespread across the state. No word yet on what caused today’s outage.

STAY TUNED for more on this story.

 

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Refused

One parent in Knox County has had enough and raises concerns over student privacy issues as the Tennessee Department of Education and Questar allege hacking of the state’s TNReady testing system.

Here’s the letter she sent to Knox County School leaders today:

TNReady has failed again. This time, there is serious concern about the safety of our students’ data.  If Questar has truly been hacked in the last 24 hours, then there is no way they can assure me that my student’s data is safe. To continue testing under these circumstances is irresponsible and possibly open to lawsuits re FERPA violations.

 

Until I can be assured that everything is safe and secure, my son, xxxxxx, simply cannot participate in these tests. Because he is in high school, and because GPA matters, I insist that he be given an opportunity to make up any tests once the above criteria are met.

 

I anticipate a certain amount of push-back on my decision and requirements for make-up.  I realize every one of you, the BOA, the principals, KCS admin, and the TN Legislature are scrambling today to figure out the ramifications and make decisions on future actions. I also see zero accountability or ownership of this problem by TNDOE.  However, since the testing resumes at 1pm for him, and no answers will be forthcoming by that time, I, as his parent, have to make this difficult decision until there is consensus among you.

 

I’m sure there will be no time for individual responses at this late time and on this busy day, however, rest assured, I will fight like a momma bear to make sure this fiasco doesn’t impact my son’s grades.

 

Thank you for your attention and a special thanks to those of you who are working so hard today to make sure we do what is best for our children.

 

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Danger Ahead?

As Tennessee prepares to test more students than ever via an online platform, there are some signs of potential trouble.

Chalkbeat reports:

State officials said Thursday they are confident the new digital platform will work under heavy traffic, even as their new testing vendor, Questar, had headaches administering computer-based tests in New York on Wednesday. Some students there struggled to log on and submit their exam responses — issues that Questar leaders blamed on a separate company providing the computer infrastructure that hosts the tests.

 

Tennessee officials say they are working with Questar to ensure similar problems don’t occur in Tennessee. They also point out our online testing infrastructure is different and that Questar will have troubleshooting staff in the state during the test administration.

Here’s the problem: Across multiple testing vendors and dating back to TCAP, Tennessee has had problems with testing. This includes the now perennial issue of not being able to deliver scores back to districts in a timely manner. In fact, in December, districts were told scores might not be back in a timely fashion this year, either.

It’s possible the state and Questar have all the issues worked out and this year’s test administration will be nearly flawless. However, the record over the past few years is not encouraging.

Then, there’s the issue of what happens with the results. If they are available for factoring into student scores, it is up to districts to choose the method. I’ve written before about why that’s problematic. Here’s a quick summary:

The cube root method yielded on average a quick score, the score that goes for a grade, of 4.46 points higher. In other words, a studentscoring basic with a raw score of 30 or higher would, on average, receive an extra 4.46% on their final quick score grade, which goes on their report card. A student who scored a 70 last year could expect to receive a 74 under the new quick score calculation.

The additional points do drop as one goes up the raw score scale, however. For the average basic student grades 3-8 with a raw score between 30 and 47, they would receive an extra 5.41 extra points under the new method.

The average proficient student grades 3-8 with a raw score between 48 and 60 would get 4.32 extra points under the new method.

The average advanced student grades 3-8 with a raw score of between 61 and 67 would receive an extra 1.97 extra points under the new method.

The difference varies much more widely for below basic students, but the difference can be as much as 25 points in some cases.

So, for those districts using quick scores in report cards, there could be a wide variance across districts depending on the method chosen. It seems to me, districts should have already communicated to families how they will calculate quick scores with some justification for that choice. Alternatively, the state could have (should have?) mandated a method so that there is score consistency across the state.

Then, of course, there’s the issue of using these scores in teacher evaluation. Let’s say testing goes well this year. This would be the first year of a test without problems. If that happens, this should serve as the baseline for any test-based teacher evaluation. Yes, I think using value-added scores is a misguided approach, but if Tennessee is going to go this route, the state ought to take steps to ensure the data is as accurate as possible. That would require at least three years of successful test administration. So far, we have zero.

If TNReady is a great test that has the potential to offer us useful insight into student learning, it’s worth taking the time to get it right. So far, it seems Tennessee has yet to learn the lesson of the NAEP outlier — we don’t need rapid acceleration, we need to be patient, take our time, and focus.

 

For more on education politics and policy in Tennessee, follow @TNEdReport


 

We Paid $60 Million for That?

Guess what? Tennessee taxpayers are on the hook for some $60 million to testing vendor Questar for new TNReady tests.

Guess what else? Those tests aren’t exactly helpful.

At least that’s the word from the education professionals in classrooms.

The Tennessean reports this year’s survey of Tennessee teachers indicates:

Sixty-five percent of educators surveyed said standardized exams aren’t worth the investment of time and effort. The same percentage of teachers said the exams don’t help refine teaching practices.

And 60 percent said the test doesn’t help them understand whether students gain the knowledge necessary to meet state standards.

In response, Education Commissioner Candice McQueen notes:

And teachers haven’t yet recieved meaningful data from the change to TNReady to help guide instruction due to the transition, Tennessee Education Commissioner Candice McQueen said. The state has and will continue to work to improve the usefulness of the test, she said.

Except the attitude about the previous tests wasn’t much better:

The attitudes toward testing also aren’t necessarily new.

In the 2015 survey, educators said testing was a burden, with teachers reporting they spent too much time preparing students for exams and taking tests.

Now, though, our students spend even more time testing and preparing for tests. Tests that educators don’t find useful. In fact, Chalkbeat notes:

By the time that Tennessee’s testing period wrapped up last week, the state’s elementary and middle school students had undergone about eight hours of end-of-year testing.

That’s more than double the testing minutes in 2012.

Why isn’t the testing useful? For one, the results don’t come back in time. Even with the switch to a new testing vendor this year following the debacle of the first year of TNReady, quick score results weren’t back very quickly and final results will be delivered later this year.

It’s worth noting, though, that even before the transition to TNReady, teachers found the testing regime burdensome and unhelpful. It’s almost like the state is surveying teachers but not actually paying attention to the results.

Why are educators frustrated, exactly? Teacher Mike Stein offers this:

Meanwhile, teachers’ performance bonuses and even their jobs are on the line. Though they wouldn’t assert themselves into the discussion, principals and directors of schools also rely heavily upon the state to administer a test that measures what it says it will measure and to provide timely results that can be acted upon. As long as both of these things remain in question, I must question both the importance of TNReady and the competence of those who insist upon any standardized test as a means of determining whether or not educators are doing their jobs.

Taxpayers are spending money on a test that day-to-day practitioners find unhelpful. In the case of evaluations, they also find it unfair. Because, well, it’s just not a valid indicator of teacher performance.

Perhaps state policymakers should take a closer look at the teacher survey results. Teachers have no problem being evaluated, and in fact, most say the current system provides them with useful feedback. The observation rubric is robust and with proper implementation and meaningful building-level support, can be helpful as a means of improving teaching practice.

What’s not especially helpful is a test that takes up significant instructional time and doesn’t yield information in a timely or useful manner.

Taking a step back and removing the high stakes associated with a single test could be an important first step toward right-sizing our state’s approach to student assessment.

 

 

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Testing Resolve

The Tullahoma City Schools Board of Education will vote September 19th on a resolution related to state standardized tests. Specifically, the resolution calls for a shift to the use of ACT/SAT assessments and a significant reduction in the amount of time students spend taking tests.

A similar resolution was passed by the Board last year.

Here is the resolution:

A RESOLUTION OF THE TULLAHOMA CITY BOARD OF EDUCATION
IN SUPPORT OF ADMINISTRATION OF THE ACT OR SAT SUITE OF ASSESSMENTS TO MEET TCAP AND “EVERY STUDENT SUCCEEDS ACT” REQUIREMENTS IN OUR END OF COURSE ASSESSMENTS AT THE HIGH SCHOOL LEVEL AND AT THE 3-8 GRADE LEVELS

WHEREAS, the Tullahoma City Board of Education is the local governmental body responsible for providing a public education to the students and families of Tullahoma City, Tennessee; and

WHEREAS, the State of Tennessee through the work of the Tennessee General Assembly, the Tennessee Department of Education, the Tennessee Board of Education, and local boards of education has established nationally recognized standards and measures for accountability in public education; and

WHEREAS, the Tennessee Department of Education is currently working to implement a replacement to the former Tennessee Comprehensive Assessment Program (TCAP) for the 2016-2017 school year; and

WHEREAS, these new assessments are called TNReady for the areas of English/language arts and math, grades 3 – 8 and TCAP Social Studies Achievement and U.S. History End of Course (EOC) exams; and

WHEREAS, during the assessment cycle of the 2015-16 school year an attempt to administer the assessments was deemed by the Tennessee Department of Education to be a “No-Go;” and

WHEREAS, the Tennessee Department of Education terminated the contract with Measurements, Incorporated and has secured the services of Questar to provide assessment services for Tennessee; and

WHEREAS, ACT and SAT have been used for decades as standard measures of college readiness and that all universities and colleges in Tennessee and the United States utilize the ACT as an admission assessment; and

WHEREAS, Pursuant to T.C.A. § 49-6-6001, all public school students must participate in a postsecondary readiness assessment such as the ACT or SAT. Districts may choose to administer the ACT or the SAT. Districts can also provide both assessments and allow their students to choose the assessment that is right for them; and

WHEREAS, one of the strategic goals of the Tennessee Department of Education is an increase of the average composite score to 21, and the benchmark for college readiness is a composite score of 21. The ACT has further broken down the benchmarks into an 18 for English, 22 for Math, 22 for Reading, and 23 for Science. If a student is able to score at, or above, these important benchmarks, they have a high probability of success in credit-bearing college courses. School districts are distinguished by the percentage of students meeting college readiness benchmarks; and

WHEREAS, both the ACT and the SAT are designed to assist colleges, universities, employers, and policy makers in the determination of college and career ready students, and experts in education administration, child development, and child psychology endorse standardized testing as a limited measure of progress and effectiveness in the important task of learning;

NOW, THEREFORE, BE IT RESOLVED

The Tullahoma City Board of Education implores the Tennessee General Assembly and the Tennessee Department of Education to allow school districts the opportunity to select either the math and English language arts assessments provided by the State of Tennessee or an English or math test that is part of the suites of standardized assessments available from either ACT or SAT.

BE IT FURTHER RESOLVED,

The Tullahoma City Board of Education implores the Tennessee General Assembly and the Tennessee Department of Education to direct psychometricians, contractors, and developers to construct assessments designed to inform instructional practice and to provide accountability that would not require for administration a period of time in hours greater in aggregate than the specific grade level of the said child, and not to exceed eight hours in length per academic year.

More on testing:

Still Too Much Testing?

Testing Time Reductions Announced

Questar’s Challenge

For more on education politics and policy in Tennessee, follow @TNEdReport