Rhetoric vs. Reality: TNReady 2017 Edition

Here’s what Commissioner of Education Candice McQueen had to say about this year’s TNReady tests in an email sent to educators on May 16th:

This year’s administration of TNReady was a success, both online and on paper, in schools across Tennessee. Thank you for you partnership and for preparing students with strong instruction every day. Stay tuned as we continue to share updates and resources.

Since then, it’s become clear that TNReady results won’t actually be ready for most districts in a timely fashion — meaning they’ll either be excluded from student grades or report cards will be held until results are available.

WPLN’s Blake Farmer reports on the scope of the problem:

The state department of education says less than a quarter of districts finished testing in time to get the results by the end of May. For those that did wrap up early, they started getting results back this week.

Yes, that’s right. More than 75% of districts won’t have results back before the end of May.

That’s an odd definition of success.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

The NeverEnding Story

Another day, more stories of districts reporting to families that TNReady scores won’t be back in time to be factored into student grades. I first reported that Williamson County sent word that scores would not be back according to the original timeline. Next, it was MNPS telling parents that TNReady scores won’t be back until June, meaning they won’t be factored into report cards.

Now, two more middle Tennessee districts have sent notices about TNReady results not being ready in time.

Here’s the notice from Clarksville-Montgomery County:

The TNReady materials from CMCSS have been returned to Questar. Tennessee has noted that they will be unable to provide the district with the test results until after the end of May. Based on CMCSS Administrative Policy INS-A023, effective April 17, 2015 in alignment with HB 36 SB 285 Amendment (005744), Clarksville Montgomery County School System will not include students’ state assessment scores in their final spring semester grades if the state assessment scores are not received by the district at least five instructional days before the end of the academic year. As we will not be receiving the scores until the end of May the scores will not be included in students’ grades for this year. The second semester average for elementary and middle will be 50% 3rd 9 weeks and 50% 4th 9 weeks. The second semester grades for high school will be 40% 3rd 9 weeks, 40% 4th 9 weeks, and 20% final exam.

And one from Wilson County Schools:

Good Afternoon!

The end of a school year always brings about a flurry of activity and excitement, but I wanted to take a moment to update you on report cards for the spring semester.

A couple of weeks ago, we announced that report cards would be available, VIA Skyward, on Tuesday May 30th. Unfortunately, it’s unclear whether the district will be able to meet that date, due to a shipping delay that was beyond our control. While our district met all of the required deadlines to ensure that our raw scores would be returned by May 22nd, the state vendor responsible for picking up the completed materials arrived several days later than scheduled. This has affected a number of large districts across the state, including Wilson County.

The TN Department of Education is aware of these delays. They’ve assured us that they’re working with the vendor to “find a remedy” for the school districts impacted. Our hope is that a solution WILL be found, and our raw scores will be returned on time. Having said that, we thought it was important to make you aware of what’s happened, in the event that report cards have to be delayed for a week.

You may remember, TNReady scores came back later than expected for the fall semester, causing report cards to be delayed. While school districts have the authority to exclude TNReady scores that are returned more than 5 days late, it is the position of Wilson County Schools that the scores be included for this semester, as they were in the fall. This is not a decision that was taken lightly. Many conversations have taken place with teachers and principals about this issue, and the overwhelming consensus is that we include the scores on report cards. Students have worked incredibly hard all year to show of their skills, and we’re eager to see just how well they did!

Thank you for remaining patient, as we work through the process. We’ll keep you updated, as we receive additional information from the state. If you have any questions, feel free to submit those to “Let’s Talk” at the following link: http://www.k12insight.com/Lets-Talk/embed.aspx?k=WK9F4DLT. You can also reach out to me directly, using the information below.

Sincerely,
Jennifer Johnson

Here’s what the Department of Education has to say about the importance of state assessments:

Our state tests serve multiple objectives:

  • They provide feedback about students’ academic progress and how it aligns with grade-level expectations, providing parents and teachers a big-picture perspective about how a student is progressing compared to peers across the district and state, including a student’s strengths and growth opportunities.
  • This builds confidence and transparency about students’ readiness for college and the workforce among Tennessee universities and employers and holds us accountable to serving all students fairly.
  • Assessments help educators strengthen instruction and reflect on their practice, and allow us to highlight schools where students are excelling, so we can learn from those who are doing well.
  • State assessments also help inform decisions at the state level and help state and district leaders determine how to allocate resources, better invest in schools, and identify where we may need to offer additional support.

All of this sounds pretty important. But, not important enough to get it right. Last year, TNReady was a complete disaster. For the past four years, there have been problems with scores being either not available or not clearly communicated.

This year, the state is not providing quick scores to districts — those are the scores used to factor into a student’s final grade. Instead, the districts were to receive the raw data and choose a method of tabulating quick scores. An analysis of the various methods indicates a significant difference in scores depending on the calculation used:

The cube root method yielded on average a quick score, the score that goes for a grade, of 4.46 points higher. In other words, a student scoring basic with a raw score of 30 or higher would, on average, receive an extra 4.46% on their final quick score grade, which goes on their report card. A student who scored a 70 last year could expect to receive a 74 under the new quick score calculation.

The additional points do drop as one goes up the raw score scale, however. For the average basic student grades 3-8 with a raw score between 30 and 47, they would receive an extra 5.41 extra points under the new method.

The average proficient student grades 3-8 with a raw score between 48 and 60 would get 4.32 extra points under the new method.

The average advanced student grades 3-8 with a raw score of between 61 and 67 would receive an extra 1.97 extra points under the new method.

The difference varies much more widely for below basic students, but the difference can be as much as 25 points in some cases.

So, for those districts using quick scores in report cards, there could be a wide variance across districts depending on the method chosen. It seems to me, districts should have already communicated to families how they will calculate quick scores with some justification for that choice. Alternatively, the state could have (should have?) mandated a method so that there is score consistency across the state.

Of course, since a number of districts now won’t have data back in a timely fashion, there may not be many districts using quick scores at all this year.

Here’s the key point: Last year’s TNReady was a debacle. That means this year is really the first year we’ve done TNReady. Instead of jerking districts (and their students) around, the state should have waived use of TNReady scores to evaluate teachers and grade students this year. Doing so would have provided insight into the time it takes to get scores back to districts and allowed for possible changes in administration for next year. Instead, the plan was rushed with a new vendor. Now, we’re where we’ve been year after year: The school year is ending, and there’s a problem with test data.

One more thing: Despite this being the first year of a successful administration of a new test and despite the gap in test results — TCAP in 2015, no results in 2016, TNReady in 2017 — the scores from TNReady will still factor into teacher evaluation.

A word of caution to districts during the 2017-18 testing cycle: The state’s track record with deadlines and score results is not so great. Maybe when they promise you scores will be ready according to a certain timeline, you should be making plans for that timeline not being met.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Maybe Someday

I reported recently on Williamson County Schools posting information that indicated a delay in the return of TNReady scores for this year. That report indicated scores would not be returned on the agreed timeline and a delay in report cards would result.

Now, word comes from MNPS that scores will not be returned to them until June. This means TNReady and EOC scores will not be factored into student grades.

Here’s the text of an email sent home to parents at JT Moore Middle School in Nashville:

Dear JT Moore Families:
TCAP grades and EOC scores will not be back in time to be included on report cards.
TCAP quick scores arriving in June
The Tennessee Department of Education has confirmed that we will not receive quick scores from state assessments before the end of the school year. Thus they will not be factored into student grades.
Infinite Campus is set up to properly adjust the weighting of nine weeks grades in the event that no exam grade is entered. Each nine-week grade will count as 25 percent of the yearly average for grades 3-8 or 50 percent of the semester average for high school courses.
As a result of this the following will apply:
No grade will go in the TCAP column in ES (3-4) and MS (5-8)
For the following HS or HS for Credit that take an EOC course there will be NO EXAM grade at all.

The grade for these semester classes will calculate 50/50:
English I, II, and III
Algebra I, Algebra II, and Geometry
Integrated Math IB and Integrated Math IIB
US History
Biology

Once again, the state’s testing regime is creating chaos. In some districts, the scores may end up counting in student grades — resulting in delayed report cards. Other districts (like MNPS) will simply not factor the test scores into student grades.

Imagine studying for an exam, being prepared, and doing well — knowing your performance is a significant factor in your final grade. Then, being told that the people who mandate the test simply won’t get it back in time. That’s the level of consideration being shown to our students.

This marks the second year of problems with TNReady and the fourth consecutive year of testing trouble wreaking havoc on students and teachers.

Oh, and then there’s the matter of what these tests really tell us:

An analysis of TCAP performance over time indicates that those school systems with consistently high levels of poverty tend to have consistently low scores on TCAP. Likewise, those systems with the least amount of poverty tend to have consistently higher scores on TCAP.

Of course, while the scores may or may not count in student grades (depending on district), they WILL be factored into teacher evaluations this year. This despite the fact they won’t provide any valid information.

TNReady will be ready. Maybe. Someday.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Definitely Not Ready

Last year, I wrote about how Tennessee students, teachers, and parents have come to expect that there will be issues with state tests and timely distribution of results. Last year was the biggest testing disaster in recent memory, but it seems there are problems popping up again this year.

Williamson County Schools recently posted this message about TNReady results and report cards:

The State Department of Education has notified the district that results from this spring’s TNReady testing will be delayed so grades cannot be calculated by the timeline established. The district is in the planning stages of determining how to proceed.

“We are disappointed in this delay,” said Superintendent Dr. Mike Looney. “The school district is preparing an action plan so we can accurately communicate to parents how their student performed during the 2016-17 school year.”

This marks the fourth consecutive year the state has not been able to produce testing results as promised.

Interestingly, earlier this week, Commissioner of Education Candice McQueen declared TNReady a “success.”

Here’s what Chalkbeat reported:

Testing results won’t be in for months, but Education Commissioner Candice McQueen is giving this year’s TNReady run an A-plus.

TNReady’s second year — and first full year for grades 3-8 — was a success on both paper and online, McQueen told education stakeholders in an email on Monday.

While the administration of the test may have been uneventful, the release of scores is proving problematic. I wrote in December about the State Board of Education getting involved late in the game on whether and how much TNReady scores should count in student grades.

Now, with a delay in scores being released, local school boards will have to decide what to do with the results. Will they hold report cards until quick scores can be reported OR will they release report cards with grades that don’t include TNReady scores?

In either case, we’re now on year four of testing trouble in Tennessee. Will next year continue the cycle?

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Haslam to Kids: Be Ready, Even Though TN Hasn’t Been

In a letter sent home to students ahead of TNReady testing season, Governor Bill Haslam encourages them to do well and tells them, “Tennessee is behind you.”

Here’s the full text of the letter:

IMG_3182

These words of encouragement as well as a handy number 2 pencil were paid for by SCORE.

Here’s the thing: For the past few years, Tennessee hasn’t exactly been “behind” kids. Not in terms of delivering an annual test in an effective manner.

I wrote last year about the new “Rite of Spring.” Here’s what I said then:

Lately, this season has brought another ritual: The Tennessee Department of Education’s failure to deliver student test scores. Each of the last three years has seen TNDOE demonstrate it’s inability to get state testing right (nevermind the over-emphasis on testing to begin with).

Back in 2014, there was a delay in the release of the all-powerful “quick scores” used to help determine student grades. Ultimately, this failure led to an Assistant Commissioner losing her job.

Then, in 2015, the way “quick scores” were computed was changed, creating lots of confusion. The Department was quick to apologize, noting:

We regret this oversight, and we will continue to improve our processes such that we uphold our commitment to transparency, accuracy, and timeliness with regard to data returns, even as we experience changes in personnel.

The processes did not appear to be much improved at all as the 2016 testing cycle got into full swing, with a significant technical failure on Day One.

When it comes to actually getting test administration and subsequent details right, Tennessee hasn’t exactly been “behind” the kids taking the tests.

But this year, armed with a letter from the Governor and a new pencil, the kids are ready. Haslam wants them to do their best, even though the state has been letting them down.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Shelby County Passes TNReady Resolution

The Shelby County Commission last night unanimously passed a resolution calling on state lawmakers to suspend use of TNReady data for student grades and teacher evaluations this year.

Here’s what they had to say:

RESOLUTION URGING THE TENNESSEE COMMISSIONER OF EDUCATION, GOVERNOR BILL  HASLAM, AND THE GENERAL ASSEMBLY TO ELIMINATE THE TENNESSEE READY SCORES AS A COMPONENT OF TEACHER EVALUATIONS AND STUDENT SCORES.  SPONSORED BY COMMISIONER DAVID REAVES

 

WHEREAS, the State of Tennessee has invested heavily in the development of educational standards known as Tennessee (TN) Ready, and

WHEREAS, standardized testing has become the cornerstone of measuring mastery of the TN educational standards, and

WHEREAS, TN Ready should be meant to be diagnostic in nature and help teachers and administrators understand and develop an educational plan to help students close the achievement gap in proficiency; and

WHEREAS, the TN Ready test has become a final exam for children instead of a continual diagnostic view; and

WHEREAS, the TN General Assembly has chosen to hold teachers accountable by linking student performance on the TN Ready exam to a teacher’s evaluation; and

WHEREAS the unintended consequence of such action has led to teachers teaching children to score high on a test versus teaching real mastery of subject matter; and

WHEREAS, while giving off the appearance of a better education, this type of teaching to the test behavior actually limits the amount of quality content in deference to test taking strategies; and

WHEREAS, the TN General Assembly has now also tied student scores to the results of standardized testing creating an unfair playing field for students and their college scholarship prospects with private school students who do not count standardized tests as part of their grade point average (GPA); and

WHEREAS, parents, students, and teachers are all impacted by the State of TN placing so much emphasis on testing instead of instruction; and

WHEREAS, record numbers of quality teachers are leaving the teaching profession and school districts are struggling to recruit and retain quality teachers due to the TN standards imposed in regards to standardized testing.

NOW, THEREFORE, BE IT RESOLVED BY THE SHELBY COUNTY BOARD OF COMMISSIONERS, that we urge the TN General Assembly to suspend the use of TN Ready Results as part of the teacher evaluations and as part of the students’ GPAs.

MORE on TNReady:

Will TNReady Yield Valid Data for Teacher Evaluations?

Washington County Joins Waiver Wave

State Board Makes Late Call on TNReady Data

For more on education politics and policy in Tennessee, follow @TNEdReport


 

 

4 Bills Teachers Need To Know About

While the debate around vouchers is loud and needed, we must not forget about the other bills that are making their way through the legislative process. Here are four bills that teachers need to know that will change how teacher effectiveness and preparation are measured.

The proposed bills all look like they will pass and become law. Spread the word about these bills so teachers will have the most updated information.

SB114/HB695 By Senator Bo Watson & Rep. Ryan Williams

There is a consensus that we need to improve the preparation of future teachers. Teachers need the most updated information from faculty that still have connections to the classroom.

The amended version of the bill requires education preparation faculty, including education deans, to have direct personal involvement in a school annually. The state summarizes that bill as follows:

Requires full-time educator preparation program faculty members, including academic deans, to have direct personal involvement in public schools or local education agencies (LEAs) annually. Requires faculty involvement to include professional learning targeted to pre-K through grade 12 teachers; learning focused on LEA specific initiatives; direct instruction to pre- K through grade 12 students; district-level partnership; or observation of pre-K through grade 12 teachers.

The bill has passed the Senate and is waiting to be taken up in the House Finance committee this week.

SB575/HB626 by Senator Dolores Gresham & Rep. Sabi Kumar:

Right now, you are able to log on the Teacher Prep Report Card to find out information about how teacher preparation programs are doing in preparing teachers for the classroom. This bill will add teacher observation data into this mix.

The bill requires the department of education to provide all state board of education approved teacher training programs access to annual evaluation data for teachers and principals graduating from the programs for a minimum of five years following the completion of the program.

It’s not clear if the public will be able to see the evaluation data from the different preparation programs. Either way, I hope the programs will use the data to improve.

The bill has passed the Senate and is waiting for the House to take it up this weeks.

SB1196/HB309 by Senator Mark Norris and Rep. David Hawk

This bill deals with assessment data that are used in overall teacher evaluations. The bill makes permanent the flexibility to use the most recent year of TVAAS student growth, if it leads to a higher evaluation score for the teacher. I’ve heard that some superintendents like this bill because it could be used to reward a teacher for a large one year growth. The three year growth option will allow teacher flexibility to change schools, grade level, or move to support a higher need population.

And here’s the state’s summary:

Requires the student growth portion of teacher evaluations to account for 10 percent of the overall evaluation criteria in FY16-17 and 20 percent in FY17-18 and each year thereafter. Requires that the most recent year’s student growth evaluation composite account for 35 percent of growth data in a teacher’s evaluation, if such use results in a higher evaluation score. Authorizes the use of educational progress and evaluation data for research purposes at postsecondary institutions. Requires Tennessee Comprehensive Assessment Program (TCAP) subject-area scores to make up the following percentages of elementary and middle school students’ final spring semester grades in grades 3-8: 10 percent in FY16-17; 15 percent in FY17-18; and 15 to 25 percent in FY18-19 and subsequent years.

This bill has passed the House and is waiting to be passed in the Senate.

SB250/HB67 by Senator Jim Tracy & Rep. Eddie Smith

This bill is trying to solve the problem that arises with teachers who teach in non-tested subjects. The state summary is pretty clear in this case:

Requires local education agencies (LEAs), by the 2018-2019 academic school year, to adopt at least one appropriate alternative growth model approved by the State Board of Education in order to provide individual growth scores to teachers in non- tested grades and subjects. Requires the Department of Education (DOE) to develop valid and reliable alternative student growth models for non-tested grades and subjects currently without such models.

What do you think?

Teachers, what are thoughts on these four bills? Let us know in the comments.

For more on education politics and policy in Tennessee, follow @TNEdReport.


 

 

It May Be Ready, But is it Valid?

In today’s edition of Commissioner Candice McQueen’s Educator Update, she talks about pending legislation addressing teacher evaluation and TNReady.

Here’s what McQueen has to say about the issue:

As we continue to support students and educators in the transition to TNReady, the department has proposed legislation (HB 309) that lessens the impact of state test results on students’ grades and teachers’ evaluations this year.

In 2015, the Tennessee Teaching Evaluation Enhancement Act created a phase-in of TNReady in evaluation to acknowledge the state’s move to a new assessment that is fully aligned to Tennessee state standards with new types of test questions. Under the current law, TNReady data would be weighted at 20 percent for the 2016-17 year.

However, in the spirit of the original bill, the department’s new legislation resets the phase-in of growth scores from TNReady assessments as was originally proposed in the Tennessee Teaching Evaluation Enhancement Act. Additionally, moving forward, the most recent year’s growth score will be used for a teacher’s entire growth component if such use results in a higher evaluation score for the teacher.

We will update you as this bill moves through the legislative process, and if signed into law, we will share detailed guidance that includes the specific options available for educators this year. As we announced last year, if a teacher’s 2015-16 individual growth data ever negatively impacts his or her overall evaluation, it will be excluded. Additionally, as noted above, teachers will be able to use 2016-17 growth data as 35 percent of their evaluation if it results in a higher overall level of effectiveness.

And here’s a handy graphic that describes the change:

TNReady Graphic

 

 

Of course, there’s a problem with all of this: There’s not going to be valid data to use for TVAAS. Not this year. It’s bad enough that the state is transitioning from one type of test to another. That alone would call into question the validity of any comparison used to generate a value-added score. Now, there’s a gap in the data. As you might recall, there wasn’t a complete TNReady test last year. So, to generate a TVAAS score, the state will have to compare 2014-15 data from the old TCAP tests to 2016-17 data from what we hope is a sound administration of TNReady.

We really need at least three years of data from the new test to make anything approaching a valid comparison. Or, we should start over building a data-set with this year as the baseline. Better yet, we could go the way of Hawaii and Oklahoma and just scrap the use of value-added scores altogether.

Even in the best of scenarios — a smooth transition from TCAP to TNReady — data validity was going to be challenge.

As I noted when the issue of testing transition first came up:

Here’s what Lockwood and McCaffrey (2007) had to say in the Journal of Educational Measurement:

We find that the variation in estimated effects resulting from the different mathematics achievement measures is large relative to variation resulting from choices about model specification, and that the variation within teachers across achievement measures is larger than the variation across teachers. These results suggest that conclusions about individual teachers’ performance based on value-added models can be sensitive to the ways in which student achievement is measured.
These findings align with similar findings by Martineau (2006) and Schmidt et al (2005)
You get different results depending on the type of question you’re measuring.

The researchers tested various VAM models (including the type used in TVAAS) and found that teacher effect estimates changed significantly based on both what was being measured AND how it was measured.

And they concluded:

Our results provide a clear example that caution is needed when interpreting estimated teacher effects because there is the potential for teacher performance to depend on the skills that are measured by the achievement tests.

If you measure different skills, you get different results. That decreases (or eliminates) the reliability of those results. TNReady is measuring different skills in a different format than TCAP. It’s BOTH a different type of test AND a test on different standards. Any value-added comparison between the two tests is statistically suspect, at best. In the first year, such a comparison is invalid and unreliable.

So, we’re transitioning from TCAP to TNReady AND we have a gap in years of data. That’s especially problematic — but, not problematic enough to keep the Department of Education from plowing ahead (and patting themselves on the back) with a scheme that validates a result sure to be invalid.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

TNReady Again

TNReady is back again with online testing being phased-in this time.

Grace Tatter at Chalkbeat reports:

The Department of Education is staggering its transition to online testing — a lesson learned last year when most of the state tried to do it all at once and the online platform buckled on the first day. As a result, the department fired its testing company, derailing the state’s assessment program, and later hired  Questar as its new test maker.

Here are the districts opting to test high school students online this year:

  • Alvin C. York Institute
  • Bedford County
  • Bledsoe County
  • Blount County
  • Bristol City
  • Campbell County
  • Cannon County
  • Cheatham County
  • Clay County
  • Cocke County
  • Coffee County
  • Cumberland County
  • Grundy County
  • Hamilton County
  • Hancock County
  • Knox County
  • Jackson-Madison County
  • Moore County
  • Morgan County
  • Putnam County
  • Scott County
  • Sullivan County
  • Trousdale County
  • Washington County
  • Williamson County

For more on education politics and policy in Tennessee, follow @TNEdReport


 

 

Russo on Tennessee’s Alternative Graduation Facts

Much has been made about a state report seeming to indicate that one third of Tennessee’s high school graduates finished school without meeting state minimum requirements.

The initial release of the report caused alarm, of course. Then, the state walked back the numbers after admitting a data error.

Alexander Russo takes a look at the report and the surrounding news coverage and comes to this conclusion:

The state issued  a bad number without carefully considering its flaws or making them clear to reporters and board members, then belatedly realized its mistake and walked the initial figure back. But news outlets contributed to the problem by rushing to report the initial figure without questioning just how iffy it might be, unintentionally delivering inaccurate information to the public. The end result has been widespread confusion that will take a long time to clear up – if it ever is.

While Russo suggests media outlets should have done a better job of both raising questions and seeking clarifying information, he notes the state bears the brunt of the responsibility:

To be clear, the state department should have checked with the districts before presenting this information to the board and to the public. The state should also have anticipated that the graduation rate number would attract enormous amounts of attention and include additional warnings and caveats about the preliminary nature of the number. The primary responsibility was theirs.

The Tennessee Education Association pointed to the state’s responsibility to release accurate data and noted that the initial claims may have helped advance the arguments of school privatization advocates.

From a TEA press release:

Organizations backing privatization schemes like private school vouchers, rapid charter expansion and high-stakes testing, need people to believe that public schools are failing. Undermining confidence in public schools is an important step to build support for radical and dangerous proposals to destroy public education.

“As a state that consistently ranks at the bottom in student investment, we are consistently in the top 10 for graduation rate because of the commitment of Tennessee educators. Our students and teachers already often have the odds stacked against them, they don’t need damaging misinformation piling anything else on,” [TEA Executive Director] Crowder said.

To be sure, both the initial data release and the subsequent reporting created a sense of alarm in the state’s education policy community. Taking just a few extra steps could have prevented what turned into a rather messy scene.

For more on education politics and policy in Tennessee, follow @TNEdReport