Weird Science

Last week, some parents suggested there was a problem with TNReady tests their children were taking. The students reported questions on material not covered or material not in their standards.

Now, it seems there may be an answer — at least when it comes to Science and Social Studies.

WBIR reports:

Over 900 TNReady Science and Social Studies test given to Anderson County student were from the wrong grade level, according to Director of Schools Tim Parrott.

Here’s the response from the Tennessee Department of Education:

“There was a poorly designed feature of the online testing system that contributed to some users accidentally administering a test to students that was below their grade level, including those at Norris Middle School. We’ve provided guidance to the district staff and the building testing coordinator to invalidate these tests. Students are not required to re-test, and their tests will not be scored. This means they will not count toward an educator’s evaluation nor will they factor into the scores we report for Norris Middle School. They will also not hurt the district’s or school’s participation rate.” – Tennessee Department of Education.

I’d suggest that the entirety of this year’s TNReady test has been a “poorly designed feature” of the Tennessee Department of Education and testing vendor Questar.

I’m interested in learning if this problem has popped up in other districts around the state. If you have something to report on this, email me: andy@tnedreport.com

For more on education politics and policy in Tennessee, follow @TNEdReport


 

What Is Normal?

It’s probably difficult to imagine a normal testing environment given the trouble Tennessee has had this year and in 2016. However, an administrator at a middle school offers some insight into what a “normal” testing schedule looks like:

The schedule includes seven different testing sessions to accommodate the eleven subtests required of middle school students.

With an average time on test of 45 minutes per subtest and an additional 40 minutes for ELA part 1, students are spending 8 hours and 55 minutes online.

Seven testing sessions means seven class transitions to the computer lab at five minutes each, plus at least five minutes to log on and go through directions. That is an additional hour and ten minutes lost teaching time.

So, in total middle school students are spending ten hours and five minutes on an online test that the state now says won’t count.  The school day is seven hours. Therefore, the state has robbed students of a day and a half of potential learning in the name of “testing what they know.”

This is what happens under “normal” circumstances. Of course, now there are the added factors of testing delays, suspension of test administration, and extended testing windows.

How long will the state continue a system that robs students of a day and a half of learning?

 

A couple of interesting comments:

Considering that many schools are only giving one subtest per day…some shools are taking nearly two and a half weeks to complete assessments. Also, if you take into account that a school day (7 hours in most places now) has other things like lunch, related arts, and class changes…instruction time is really in the five hour range. So, at minimum the assessment is taking two complete days of instruction. But even that is not the story. Where I live students in middle school are taking eight days over a three week period(T, W, TR) to complete assessments. Again, one subpart at a time. Due to the amount of time that it takes to set-up the lab or ready paper materials(counting every single item in the test admin room as it leaves, distributing in the classroom, counting every test item as it leaves the classroom, and then counting it again as it reaches the test admin office)…the process is much, much longer. Items that must be counted are state issued rulers, calculators, answer sheets, test booklets, scrap paper. It takes forever. Most schools begin w TNReady to start the day. So, basically it is like running school on a two hour delay snow day. Anyone know how much work gets done on snow delay days or early dismissals? Not much. Sorry for the long post. Now ask teachers how much time is spent on benchmarking and quarterly assessments, basically now testing to prepare for the test. I am willing to suggest that the number of days(not actual hours….just days where the instructional environment is disrupted) where teachers are having to give district, state, and federal assessments(think NAEP) is roughly 25-30 days per school year. Over 12 years, that is 360 days or roughly two years of a child’s instructional time from grades 1-12 where a local/state/federal assessment disrupts some portion of a student’s day. Now imagine how much more our children would learn if we got back all of that time accrued over 12 years that was lost to assessments.

AND:

At the high school where I work, we have testing the in AM and PM spread out over THREE WEEKS. That’s 3 weeks of altered daily schedules, 3 weeks of some but not all students missing a class because of testing, 3 weeks of students coming to my class surprised that we are actually doing work that day (because they are equating test taking time as a time for rest in all classes)(I can’t say I really blame them), and 3 weeks of students wondering when it will be over. Three weeks of this energy-sapping, soul sucking testing nonsense that doesn’t count for anything. Try making teenagers take it seriously. It’s a joke and they know it.

For more on education politics and policy in Tennessee, follow @TNEdReport

Your support keeps Tennessee Education Report going! Thank you!


 

Down Down Down

Another day of TNReady testing, another set of issues.

Reports are coming in from systems across the state that students are having login and submission problems.

Reports are in from Hardin, Chester, Knox, Shelby, Sumner, Wilson, Washington, Montgomery, Roane, Rutherford so far.

Some reports suggest that a cut fiber line between Georgia and Tennessee is to blame.

Stay tuned …

UPDATE: 5:20 PM

The Tennessee Department of Education is blaming a dump truck for cutting a fiber optic line that provided internet connectivity.

Here’s what Joey Hassell, Director of Schools in Haywood County had to say about that on Twitter:

I don’t understand. NEVER lost internet connectivity. Our schools could access the internet & phone service w/ worked fine. It was business as usual except for in our schools! So…the dump truck just messed up testing?!?

 

 

For more on education politics and policy in Tennessee, follow @TNEdReport

Help keep TNEdReport on the beat!


 

Would You Eat This Pie?

After last week’s TNReady failure, the Tennessee General Assembly took some action to mitigate the impact the test would have on students and teachers.

I wrote at the time that the legislature’s action was a good step, but not quite enough:

  1. The law does say that districts and schools will not receive an “A-F” score based on the results of this year’s test. It also says schools can’t be placed on the state’s priority list based on the scores. That’s good news.

  2. The law gives districts the option of not counting this year’s scores in student grades. Some districts had already said they wouldn’t count the test due to the likelihood the scores would arrive late. Now, all districts can take this action if they choose.

  3. The law says any score generated for teachers based on this year’s test cannot be used in employment/compensation decisions.

Here’s what the law didn’t say: There will be NO TVAAS scores for teachers this year based on this data.

In other words, this year’s TNReady test WILL factor into a teacher’s evaluation.

The Department of Education took some steps to clarify what that means for teachers and offered a handy pie chart to explain the evaluation process:

First, this chart makes clear that this year’s TNReady scores WILL factor into a teacher’s overall evaluation.

Second, this chart is crazy. A teacher’s growth score is factored on tests from three different years and three types of tests.

15% of the growth score comes from the old TCAP (the test given in 2014-15, b/c the 2015-16 test had some problems). Then, 10% comes from last year’s TNReady, which was given on paper and pencil. Last year was the first year of a full administration of TNReady, and there were a few problems with the data calculation. A final 10% comes from this year’s TNReady, given online.

So, you have data from the old test, a skipped year, data from last year’s test (the first time TNReady had truly been administered), and data from this year’s messed up test.

There is no way this creates any kind of valid score related to teacher performance. At all.

In fact, transitioning to a new type of test creates validity issues. The way to address that is to gather three or more years of data and then build on that.

Here’s what I noted from statisticians who study the use of value-added to assess teacher performance:

Researchers studying the validity of value-added measures asked whether value-added gave different results depending on the type of question asked. Particularly relevant now because Tennessee is shifting to a new test with different types of questions.

Here’s what Lockwood and McCaffrey (2007) had to say in the Journal of Educational Measurement:

We find that the variation in estimated effects resulting from the different mathematics achievement measures is large relative to variation resulting from choices about model specification, and that the variation within teachers across achievement measures is larger than the variation across teachers. These results suggest that conclusions about individual teachers’ performance based on value-added models can be sensitive to the ways in which student achievement is measured.
These findings align with similar findings by Martineau (2006) and Schmidt et al (2005)
You get different results depending on the type of question you’re measuring.

The researchers tested various VAM models (including the type used in TVAAS) and found that teacher effect estimates changed significantly based on both what was being measured AND how it was measured.

And they concluded:

Our results provide a clear example that caution is needed when interpreting estimated teacher effects because there is the potential for teacher performance to depend on the skills that are measured by the achievement tests.

If you measure different skills, you get different results. That decreases (or eliminates) the reliability of those results. TNReady is measuring different skills in a different format than TCAP. It’s BOTH a different type of test AND a test on different standards. Any value-added comparison between the two tests is statistically suspect, at best. In the first year, such a comparison is invalid and unreliable. As more years of data become available, it may be possible to make some correlation between past TCAP results and TNReady scores.

I’ve written before about the shift to TNReady and any comparisons to prior tests being like comparing apples and oranges.

Here’s what the TN Department of Education’s pie chart does: It compares an apple to nothing to an orange to a banana.

Year 1: Apple (which counts 15%)

Year 2: Nothing, test was so messed up it was cancelled

Year 3: Orange – first year of TNReady (on pencil and paper)

Year 4: Banana – Online TNReady is a mess, students experience login, submission problems across the state.

From these four events, the state is suggesting that somehow, a valid score representing a teacher’s impact on student growth can be obtained. The representative from the Department of Education at today’s House Education Instruction and Programs Committee hearing said the issue was not that important, because this year’s test only counted for 10% of the overall growth score for a teacher. Some teachers disagree.

Also, look at that chart again. Too far up? Too confusing? Don’t worry, I’ve made a simpler version:

For more on education politics and policy in Tennessee, follow @TNEdReport


 

 

Your support keeps Tennessee Education Report going strong — thank you!

A TNReady Letter

An educator in Campbell County sent this letter to legislators about Tennessee’s TNReady trouble:

I am an educator. My husband is an educator as well. We have each been teaching for 17 years and hold master’s degrees in our fields. We are also both history teachers who uphold democratic principles and stress the importance of fulfilling our civic duty.

I am contacting you regarding the issues educators and students are dealing with when it comes to testing and the education system in general. First and foremost, why are we expected to give our kids a standardized test when our students are not standardized kids? We differentiate our instruction every day, we change and adapt to our students needs, we support and scaffold and encourage, but these tests leave no room for that.

As the only social studies teacher in my school at my grade level, I see 165 students a day. I get them for 45 minutes. I teach from the highest high to the lowest low sometimes in the same class period. I have students reading at a 12th grade level with students who literally can spell 2 words due to cognitive delays, yet every kid takes the same test. They may have accommodations such as read aloud or extended time, but someone who is functioning on a third grade level really shouldn’t be expected to take an 8th grade test. That makes that student feel like a failure. No kid should feel like a failure.

I understand the need for assessment, but it should only be used as a measurement tool to gauge growth of the individual students. It should never be used as a weapon to punish the child or the teacher. I don’t like the term accountability because it turns into blame. I promise you that on any given day, you can come into my class and my students are engaged in high order thought processes. We have deep intense discussions about the subject matter, we hold round table talks as historical figures, we participate in congressional hearings where a guest panel fires questions at them, we have simulations, we have csi cases, we examine historical evidence to make a determination, we really dig into history. I teach my butt off. Every day. I love what I do and I am passionate about it, but I am also frustrated because what if I didn’t cover tested material and I look ineffective on paper.

This brings me to my next point. The standards are impossible to truly teach in the timeframe. I don’t believe education comes from doing vocabulary or listening to a teacher lecture. I think true understanding comes from discovery and having the time to explore the topics. In 8th grade, I am responsible for 98 separate standards. There will be a few less in 2020, but right now I have 98 separate standards. Some of those standards only cover one subtopic, but those are few and far between. I put a standard on the board today that included 18 different subtopics. I have counted my subtopics. There are 582 of them. 582 new terms and phrases and concepts. It is impossible to teach all of those well. So I focus on what is most important: Settlement, slavery, conflict, government, native Americans, foreign relations. I would like to invite any legislators to come into our school and sit in our classes and take the 8th grade test that our students, our 13-14 year old children are expected to take. It would prove to be very difficult.

These standards are not age appropriate. I understand why legislators have latched onto the word rigor. It sounds like something is being done. The only thing that has happened is we are setting these kids up for failure. We have jumped on board with this terminology and thrown out the buzzwords, but everyone has lost their common sense. We need to ask ourselves, does an 8th grader, 7th, 6th…etc. really need to know this? Why would someone besides a historian need this? Where are the geography and map reading skills? Why are we trying to push these kids beyond what they are capable of understanding at their age? It’s insanity and it is getting worse with every new change.

The testing debacle has been at the forefront the past few days. TN ready has consistently proven to be not ready. Every year a plague of problems hits the news circuit concerning the system. Why don’t we just let it go? Too much of our tax money has gone into this program. If you ask educators, most will tell you these tests do not accurately measure student growth or achievement. There are too many variables. Why can’t we change the testing structure? It would make sense to test our students on all grade level skills upon entrance in the fall to gain a baseline, test again in the winter to determine growth, and test a final time the last week of school to see what the student did that year. The standards and the tests should be created by current educators. No one knows better than the teachers how to help the students.

Teachers are not lazy. We spend years becoming experts in our fields. We plan lessons, spend money, give our time for free, worry and counsel these kids to make sure they make it. For the majority of us, this isn’t an 8-3 job. From August 1st until June 1st, we are 100% devoted to our schools and our students. Many of us do extra training in the summer to stay current. When we voice concerns, it isn’t because we want our jobs to be easier. It’s because the system is broken, and more times than not we are treated like the villain. We just want professional courtesy.

Please vote to keep tests from counting against our teachers and our students. But do even better. Try to find a solution so our students get the quality of education they deserve. Think what could happen if we funneled some of the millions away from testing and test prep, and sent it directly into the classes. We could hire more teachers and get rid of overcrowding, we could finance field trips so the kids could experience things first hand, we could have materials for science experiments for every kid. Learning could be something kids looked forward to again. It would not be drill and kill test prep.

I get passionate about this subject. Our kids are too important to not get passionate about. I truly want education to be better. I want to see big changes. Get out and talk to teachers. Talk to students. Talk to parents.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Your support keeps TNEdReport going strong!

Moratorium?

Echoing a call made earlier this week by teachers in Shelby County, a group of House Democrats including gubernatorial candidate and Democratic leader Craig Fitzhugh, said yesterday the state should place a three year moratorium on using TNReady scores in student grades and teacher evaluations.

The move comes amid the latest round of troubles for the state’s standardized test, known as TNReady.

Jason Gonzales of the Tennessean reports:

The call for a three-year stay on accountability comes after another round of TNReady issues, the state’s standardized assessment. This is the second year in a row that Tennessee House Democrats have called for such a moratorium.

“Right now, as it stands, Tennessee isn’t proficient in getting students assessed,” said Rep. Antonio Parkinson, D-Memphis. “We encourage you (the Tennessee Department of Education) to put your pride aside … and give TNReady three years to be perfected.”

The group did not call for scrapping the test, and in fact, under their proposal, the test would still be administered. However, by not including the results in student grades or teacher evaluations, problems such as the missed deadlines that impacted report cards at the end of last school year and the missing students that are now impacting teacher TVAAS scores, could be avoided.

Additionally, taking three years to build a reliable base of data would help add to the validity of any accountability measure based on those scores going forward.

It’s not clear if there’s more momentum for a moratorium this year. What is clear is that the House will hold hearings on the latest TNReady problems next week. Those hearings may well indicate what the future holds for Tennessee’s troubled testing system.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

 

Seriously?

Will TNReady testing resume this year? For some students, maybe not.

The President of Measurement, Inc. said yesterday that there was no guarantee his company would make the testing window.

The Memphis Daily News reports:

The president of a North Carolina-based testing company said Monday that he can’t guarantee all students in Tennessee will receive the test on time.

Measurement Inc. president and CEO Henry “Hank” Scherich said his company is working furiously to get the new TNReady materials to students.

“I wish I could promise them,” Scherich said. He added they were doing everything humanly possible to get the tests to the students on time.

All of the students have at least some of the testing materials, he said, but the company has found itself scrambling to print and ship 5 million test booklets for Tennessee.

This follows last week’s  that a Friday deadline would be missed.

That event caused some lawmakers to call for this year’s testing to be cancelled. The Department of Education has still not agreed to that solution.

For more on education politics and policy in Tennessee, follow @TNEdReport