Posted on by

BREAKING: Another Dump Truck Hits TNReady?!

Tennessee’s failed testing system, TNReady, is experiencing problems yet again. Last week, the Department of Education alleged a dump truck took out the test.

That was the latest in a line of testing challenges during this year’s administration of TNReady.

This morning, eleven systems so far are reporting login and/or submission problems with TNReady. Knox, Putnam, Campbell, Sequatchie, Rutherford,  Robertson, Williamson, Montgomery, Collierville, Sumner, and Cumberland have reported issues with the test today so far.

UPDATE: 9:59 AM — It appears the TNReady problems are widespread across the state. No word yet on what caused today’s outage.

STAY TUNED for more on this story.

 

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Posted on by

Bump in the Road

While one Tennessee legislator refers to the TNReady trouble as a mere “Bump in the Road,” school districts around the state are stuck trying to pick up the pieces and move forward. Here’s an excerpt from a message to parents provided by Williamson County Schools with an outline of the challenges faced during TNReady testing so far:

On behalf of WCS, I want to apologize to you and your children for having to endure the State’s failed online testing program these past two weeks. We know that this has been difficult for everyone involved, and for that we are sorry.

State standardized testing has been required for decades in Tennessee. What has changed in the past few years is that the Tennessee Department of Education has been trying to transition to online testing. As many of you know, they have had massive difficulties with online administration of TNReady and high school End of Course exams (EOCs). These online problems have affected our grades 5-11.

Here’s a quick timeline of what our students and staff have undergone since State testing began last week:

Monday, April 16: Login problems affecting approximately 15,000 students.

Tuesday, April 17: Login problems affecting approximately 8,000 students.

Wednesday, April 18: WCS suspended testing to give the TDOE time to correct problems.

Thursday, April 19: Login problems affecting approximately 1,000 students.

Friday, April 20: No significant issues reported.

Monday, April 23: No significant issues reported.

Tuesday, April 24: System defaults caused 100+ students to take the wrong grade level test.

Wednesday, April 25: Delays and canceled testing affecting approximately 8,000 students.

Thursday, April 26: System lockout affecting approximately 15,000 students.

Friday, April 27: No significant issues reported.

That’s six days of problems over a two week testing period. Williamson County is now the third district (joining Knox and Anderson) to report students being given the wrong test.

This doesn’t look like a bump in the road, it looks like a huge mess. Thousands of students in just this one district have been impacted. Districts are now scheduling meetings to determine how to move forward in light of “hold harmless” and “adverse action” legislation.

Still, the Tennessee Department of Education insists that testing must keep going. A TDOE representative told House and Senate members this week that it was entirely possible to obtain valid data from this test administration.

We’ve supposedly had hackers and dump trucks running around trying to stop the test.

Still, Candice McQueen and her Department, backed by Governor Haslam, insist we simply MUST keep testing.

Why?

Whenever the idea of stopping testing is brought up, the state says we will lose federal money. Even this week, amid legislative wrangling on the issue, the final proposal was adopted as means of preserving compliance with federal law.

Here’s how Chalkbeat reported it:

The language in both bills seeks to keep Tennessee’s school accountability plan in compliance with a federal education law that requires states to include student performance in their teacher evaluation model — or risk losing federal funding for schools. Lawmakers also cited the state’s tenure rules in preserving the data.

So, what’s the real risk?

There isn’t one. If Tennessee stops testing this year and doesn’t include the data at all in teacher evaluation, we’d only be violating the plan we wrote, not some federal mandate. It was Tennessee’s ESSA plan that spelled out how our state planned to use data from testing. Certainly, a case can be made that testing didn’t go as planned this year, so we won’t use this year’s data.

Still, could we lose money?

No.

I mean, it’s not exactly THAT clear, but pretty much.

Here’s what I wrote on this topic back in 2016 (yes, we have testing problems all the time — as one person noted on Twitter, we’ve become the Cleveland Browns of state testing):

There’s just one problem: The federal government has not (yet) penalized a single district for failing to hit the 95% benchmark. In fact, in the face of significant opt-outs in New York last year (including one district where 89% of students opted-out), the U.S. Department of Education communicated a clear message to New York state education leaders:  Districts and states will not suffer a loss of federal dollars due to high test refusal rates. The USDOE left it up to New York to decide whether or not to penalize districts financially.

And here’s more on how the federal Department of Education rarely withholds funds from states over testing or accountability issues:

  1. In 2015 more than 600,000 students opted out of state tests around the country, including 20% of all students in NYS, and 100,000 students in New Jersey and Colorado. Here in Illinois, more than 40,000 students opted out, including 10% of all CPS students eligible for the test. In 2016, as a district, CPS still did not make 95% participation, and more than 160 individual CPS schools also had <95% participation on PARCC last year. And in New York State in 2016, more than 9 out of 10 school districts had less than 95% participation. No state, district or school lost a single penny—despite threats throughout testing season every year since mass opt out began.  In fact, as mentioned above, no state or local educational agency has lost any funding for participation rates ever. And states have had participation below 95% in the past (particularly in demographic subgroups), even before the era of mass opt out campaigns.
  2. In 2015 the IL State Board of Education (ISBE) opted out the entire state from science testing. States must administer science testing by grade-span (i.e. once in 3-5th, once in 6-8th, once in high school). There was a 0% participation rate. No funding was lost. The US Department of Education (USED)’s response was described by the Chicago Tribune as a ‘crackdown‘. In fact, the ‘crackdown’ was a stern letter, informing the state that they needed to administer a science test the next year.

So, will we lose money because we tried to administer a test but experienced a series of unfortunate events?

No.

No we will not.

Still, the Tennessee Department of Education insists that our students, teachers, and schools persist.

Still, TDOE insists the data is somehow valid and useful.

The fact is, TNReady has not been. Not in 2016, not this year, not yet.

For more on education politics and policy in Tennessee, follow @TNEdReport

Keep the education news and analysis coming!


 

Posted on by

Whiners

Rep. Sabi Kumar has a message for students, parents, and teachers about the TNReady problems: Stop whining!

Yes, that’s what the Robertson County State Representative said in an email to an educator. The full email response is pictured below. Here’s the text where he responds directly about the testing issues this year:

I also ask a favor of you.

I ask that you and your colleagues rise to your calling as Educators and use this TN Ready Software Failure, that is being labeled a “Debacle” and a “Spectacular Failure”, as a Teaching Moment for your students.

I ask that you teach your students to treat this as a Bump in the Road and teach them how to handle Software Failures at home, or later, in their work space and in Life!

We do not want a child to learn that if they complain loud enough or cry in response to stress, life will get better. It is more important that they learn how to overcome stress and find solutions to problems in the Software of Life.

We want our students to be Winners Who Overcome Circumstances in Life and not whiners who complain. I do agree that each child has different needs and responses but our goal should be the same.

I admire you and your fellow Educators because you do so much for our children.

There is no successful adult who does not owe gratitude to that special teacher, or teachers, who helped shape us towards success in life.

So, Rep. Kumar ends his letter with kind-sounding words about teachers after suggesting that raising concerns about a testing system that has repeatedly failed students makes them “whiners.”

Kumar speaks of teaching children to find solutions to problems while simultaneously refusing to hold the Tennessee Department of Education accountable. He fails to mention that the voters of Robertson County send him to Nashville to help find solutions. Instead, he allows a multi-million dollar, multi-year testing failure to continue and implores those pointing that out to stop whining.

To students who have had schedules disrupted, test answers erased, and instructional time lost, Mr. Kumar says: stop complaining and don’t cry.

One thing he says is true: So far, in spite of the outcry from teachers, parents, and students, life in schools during TNReady is not getting any better. The testing marches on, the window is extended, the problems continue and those responsible are not held accountable.

Here’s an image of the email sent by Kumar:

Here’s how to contact Rep. Kumar.

For more on education politics and policy in Tennessee, follow @TNEdReport

Your support keeps the education news coming!


 

Posted on by

Weird Science

Last week, some parents suggested there was a problem with TNReady tests their children were taking. The students reported questions on material not covered or material not in their standards.

Now, it seems there may be an answer — at least when it comes to Science and Social Studies.

WBIR reports:

Over 900 TNReady Science and Social Studies test given to Anderson County student were from the wrong grade level, according to Director of Schools Tim Parrott.

Here’s the response from the Tennessee Department of Education:

“There was a poorly designed feature of the online testing system that contributed to some users accidentally administering a test to students that was below their grade level, including those at Norris Middle School. We’ve provided guidance to the district staff and the building testing coordinator to invalidate these tests. Students are not required to re-test, and their tests will not be scored. This means they will not count toward an educator’s evaluation nor will they factor into the scores we report for Norris Middle School. They will also not hurt the district’s or school’s participation rate.” – Tennessee Department of Education.

I’d suggest that the entirety of this year’s TNReady test has been a “poorly designed feature” of the Tennessee Department of Education and testing vendor Questar.

I’m interested in learning if this problem has popped up in other districts around the state. If you have something to report on this, email me: andy@tnedreport.com

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Posted on by

What Is Normal?

It’s probably difficult to imagine a normal testing environment given the trouble Tennessee has had this year and in 2016. However, an administrator at a middle school offers some insight into what a “normal” testing schedule looks like:

The schedule includes seven different testing sessions to accommodate the eleven subtests required of middle school students.

With an average time on test of 45 minutes per subtest and an additional 40 minutes for ELA part 1, students are spending 8 hours and 55 minutes online.

Seven testing sessions means seven class transitions to the computer lab at five minutes each, plus at least five minutes to log on and go through directions. That is an additional hour and ten minutes lost teaching time.

So, in total middle school students are spending ten hours and five minutes on an online test that the state now says won’t count.  The school day is seven hours. Therefore, the state has robbed students of a day and a half of potential learning in the name of “testing what they know.”

This is what happens under “normal” circumstances. Of course, now there are the added factors of testing delays, suspension of test administration, and extended testing windows.

How long will the state continue a system that robs students of a day and a half of learning?

 

A couple of interesting comments:

Considering that many schools are only giving one subtest per day…some shools are taking nearly two and a half weeks to complete assessments. Also, if you take into account that a school day (7 hours in most places now) has other things like lunch, related arts, and class changes…instruction time is really in the five hour range. So, at minimum the assessment is taking two complete days of instruction. But even that is not the story. Where I live students in middle school are taking eight days over a three week period(T, W, TR) to complete assessments. Again, one subpart at a time. Due to the amount of time that it takes to set-up the lab or ready paper materials(counting every single item in the test admin room as it leaves, distributing in the classroom, counting every test item as it leaves the classroom, and then counting it again as it reaches the test admin office)…the process is much, much longer. Items that must be counted are state issued rulers, calculators, answer sheets, test booklets, scrap paper. It takes forever. Most schools begin w TNReady to start the day. So, basically it is like running school on a two hour delay snow day. Anyone know how much work gets done on snow delay days or early dismissals? Not much. Sorry for the long post. Now ask teachers how much time is spent on benchmarking and quarterly assessments, basically now testing to prepare for the test. I am willing to suggest that the number of days(not actual hours….just days where the instructional environment is disrupted) where teachers are having to give district, state, and federal assessments(think NAEP) is roughly 25-30 days per school year. Over 12 years, that is 360 days or roughly two years of a child’s instructional time from grades 1-12 where a local/state/federal assessment disrupts some portion of a student’s day. Now imagine how much more our children would learn if we got back all of that time accrued over 12 years that was lost to assessments.

AND:

At the high school where I work, we have testing the in AM and PM spread out over THREE WEEKS. That’s 3 weeks of altered daily schedules, 3 weeks of some but not all students missing a class because of testing, 3 weeks of students coming to my class surprised that we are actually doing work that day (because they are equating test taking time as a time for rest in all classes)(I can’t say I really blame them), and 3 weeks of students wondering when it will be over. Three weeks of this energy-sapping, soul sucking testing nonsense that doesn’t count for anything. Try making teenagers take it seriously. It’s a joke and they know it.

For more on education politics and policy in Tennessee, follow @TNEdReport

Your support keeps Tennessee Education Report going! Thank you!


 

Posted on by

Down Down Down

Another day of TNReady testing, another set of issues.

Reports are coming in from systems across the state that students are having login and submission problems.

Reports are in from Hardin, Chester, Knox, Shelby, Sumner, Wilson, Washington, Montgomery, Roane, Rutherford so far.

Some reports suggest that a cut fiber line between Georgia and Tennessee is to blame.

Stay tuned …

UPDATE: 5:20 PM

The Tennessee Department of Education is blaming a dump truck for cutting a fiber optic line that provided internet connectivity.

Here’s what Joey Hassell, Director of Schools in Haywood County had to say about that on Twitter:

I don’t understand. NEVER lost internet connectivity. Our schools could access the internet & phone service w/ worked fine. It was business as usual except for in our schools! So…the dump truck just messed up testing?!?

 

 

For more on education politics and policy in Tennessee, follow @TNEdReport

Help keep TNEdReport on the beat!


 

Posted on by

No Adverse Action

After much wrangling in a day that saw the Tennessee House of Representatives hold up proceedings in order to move forward with an effort to truly hold students, teachers, and schools harmless in light of this year’s TNReady trouble, it appears a compromise of sorts has been reached.

Here’s the language just adopted by the Senate and subsequently passed by the House:

SECTION 1. Tennessee Code Annotated, Title 49, Chapter 6, Part 60, is amended by adding the following language as a new section: Notwithstanding any law to the contrary, no adverse action may be taken against any student, teacher, school, or LEA based, in whole or in part, on student achievement data generated from the 2017-2018 TNReady assessments. For purposes of this section, “adverse action” includes, but is not limited to, the identification of a school as a priority school and the assignment of a school to the achievement school district.

This language does not explicitly address the issue of using TNReady for TVAAS, but it has an effect similar to legislation passed in 2016 during that year’s TNReady trouble. Yes, it seems problems with testing in Tennessee are the norm rather than the exception.

Here’s what this should mean for teachers: Yes, a TVAAS score will be calculated based on this year’s TNReady. But, if that TVAAS score lowers your overall TEAM score, it will be excluded — lowering your TEAM score would be an “adverse action.”

While not perfect, this compromise is a victory — the TNReady data from a messed up test will not harm grades or be used in the state’s A-F report card for schools or be used to give a negative growth score to a teacher via TVAAS.

Yes, TVAAS is still suspect, but there’s an election in November and a new Commissioner of Education coming after that. Heading into the November election is a great time to talk with candidates for the legislature and for Governor about the importance of evaluations that are fair and not based on voodoo math like TVAAS. Remember, even under the best of circumstances, TVAAS would not have yielded valid results this year.

While it is disappointing that Senators did not want to follow the lead of their House counterparts and explicitly deal with the TVAAS issue, there’s no doubt that persistent outreach by constituents moved the needle on this issue.

For more on education politics and policy in Tennessee, follow @TNEdReport

If you enjoy the education news provided here, consider becoming a patron!


 

Posted on by

TN PTA Takes on TNReady

As the legislature grinds to a halt over the TNReady testing issue, the Tennessee PTA weighs-in.

Here’s their statement:

Tennessee PTA supports legislative solutions that can be passed by the Tennessee General Assembly before this session ends to respond to the TNReady testing issues quickly.

Tennessee PTA has the following concerns about the recent TNReady tests:

  • the inability for some students to submit answers and essays;
  • the inability to submit tests in a timely manner;
  • the electronic component of the recall test feature activated the next day has the potential for students to change answers;
  • typos on paper tests that may cause students to pick the wrong answers; and;
  • TVAAS scoring data for a three-year teacher evaluation based on this year’s test scores.

We believe these concerns justify invalidating all reporting for this school year on all TNReady test scores and invalidating TVAAS scoring to be included in teacher three-year testing evaluation.

And more importantly, our youth have been subjected to online cyber-attacks and still enduring test anxiety for scores that will count for nothing.

The Tennessee PTA believes in testing accountability, but the class time missed and the lack of new material not introduced during this time should be considered a deterrent to student achievement and to the social emotional well-being of students. We continue to support and educate parents to advocate for what their children need to be successful in school and in life.

For more on education politics and policy in Tennessee, follow @TNEdReport

Consider becoming a Patron and help keep the education news coming!


 

Posted on by

SUSPENDED: A TNReady Story

As the 2018 TNReady saga continues, there are scattered reports today of testing issues.

Testing has been suspended at Collierville High School due to issues with the vendor, Questar. Likewise, problems have been reported in Cookeville.

Various login and submission issues have been reported from some schools in Nashville.

Reports from multiple districts indicate an upgrade to the Nextera testing platform used by Questar wiped out class rosters. Guidance from Questar initially suggested that testing coordinators could manually upload the rosters.

It is not yet clear how much of an impact issues related to the Nextera upgrade have had across the state. Some schools report the rosters eventually reappeared.

UPDATE: Reports at the end of the day of testing issues (login problems, rosters disappearing/reappearing, wrong tests loaded and replaced, submission delays) from Washington County, Knox County, Wilson County, and Williamson County in addition to the problems reported earlier from Cookeville and Collierville.

If you know more about this issue or if your district or school has had testing issues today, please email me at andy@tnedreport.com 

Stay tuned for more on TNReady and testing in Tennessee.

For more on education politics and policy in Tennessee, follow @TNEdReport

Your support helps keep TNEdReport going strong!


 

Posted on by

Would You Eat This Pie?

After last week’s TNReady failure, the Tennessee General Assembly took some action to mitigate the impact the test would have on students and teachers.

I wrote at the time that the legislature’s action was a good step, but not quite enough:

  1. The law does say that districts and schools will not receive an “A-F” score based on the results of this year’s test. It also says schools can’t be placed on the state’s priority list based on the scores. That’s good news.

  2. The law gives districts the option of not counting this year’s scores in student grades. Some districts had already said they wouldn’t count the test due to the likelihood the scores would arrive late. Now, all districts can take this action if they choose.

  3. The law says any score generated for teachers based on this year’s test cannot be used in employment/compensation decisions.

Here’s what the law didn’t say: There will be NO TVAAS scores for teachers this year based on this data.

In other words, this year’s TNReady test WILL factor into a teacher’s evaluation.

The Department of Education took some steps to clarify what that means for teachers and offered a handy pie chart to explain the evaluation process:

First, this chart makes clear that this year’s TNReady scores WILL factor into a teacher’s overall evaluation.

Second, this chart is crazy. A teacher’s growth score is factored on tests from three different years and three types of tests.

15% of the growth score comes from the old TCAP (the test given in 2014-15, b/c the 2015-16 test had some problems). Then, 10% comes from last year’s TNReady, which was given on paper and pencil. Last year was the first year of a full administration of TNReady, and there were a few problems with the data calculation. A final 10% comes from this year’s TNReady, given online.

So, you have data from the old test, a skipped year, data from last year’s test (the first time TNReady had truly been administered), and data from this year’s messed up test.

There is no way this creates any kind of valid score related to teacher performance. At all.

In fact, transitioning to a new type of test creates validity issues. The way to address that is to gather three or more years of data and then build on that.

Here’s what I noted from statisticians who study the use of value-added to assess teacher performance:

Researchers studying the validity of value-added measures asked whether value-added gave different results depending on the type of question asked. Particularly relevant now because Tennessee is shifting to a new test with different types of questions.

Here’s what Lockwood and McCaffrey (2007) had to say in the Journal of Educational Measurement:

We find that the variation in estimated effects resulting from the different mathematics achievement measures is large relative to variation resulting from choices about model specification, and that the variation within teachers across achievement measures is larger than the variation across teachers. These results suggest that conclusions about individual teachers’ performance based on value-added models can be sensitive to the ways in which student achievement is measured.
These findings align with similar findings by Martineau (2006) and Schmidt et al (2005)
You get different results depending on the type of question you’re measuring.

The researchers tested various VAM models (including the type used in TVAAS) and found that teacher effect estimates changed significantly based on both what was being measured AND how it was measured.

And they concluded:

Our results provide a clear example that caution is needed when interpreting estimated teacher effects because there is the potential for teacher performance to depend on the skills that are measured by the achievement tests.

If you measure different skills, you get different results. That decreases (or eliminates) the reliability of those results. TNReady is measuring different skills in a different format than TCAP. It’s BOTH a different type of test AND a test on different standards. Any value-added comparison between the two tests is statistically suspect, at best. In the first year, such a comparison is invalid and unreliable. As more years of data become available, it may be possible to make some correlation between past TCAP results and TNReady scores.

I’ve written before about the shift to TNReady and any comparisons to prior tests being like comparing apples and oranges.

Here’s what the TN Department of Education’s pie chart does: It compares an apple to nothing to an orange to a banana.

Year 1: Apple (which counts 15%)

Year 2: Nothing, test was so messed up it was cancelled

Year 3: Orange – first year of TNReady (on pencil and paper)

Year 4: Banana – Online TNReady is a mess, students experience login, submission problems across the state.

From these four events, the state is suggesting that somehow, a valid score representing a teacher’s impact on student growth can be obtained. The representative from the Department of Education at today’s House Education Instruction and Programs Committee hearing said the issue was not that important, because this year’s test only counted for 10% of the overall growth score for a teacher. Some teachers disagree.

Also, look at that chart again. Too far up? Too confusing? Don’t worry, I’ve made a simpler version:

For more on education politics and policy in Tennessee, follow @TNEdReport


 

 

Your support keeps Tennessee Education Report going strong — thank you!