A Little Less Creepy

Two years ago I wrote about Tennessee’s Achievement School District (ASD) creeping beyond its original mission. I noted then that the state’s Race to the Top grant which spawned the ASD envisioned a handful of schools receiving highly targeted attention. I argued that rapid growth and a lack of clear communication contributed to a bumpy start for the turnaround effort. I concluded by offering this suggestion:

By creeping beyond its admirable mission, the ASD has become an example of good intentions gone awry. Focusing on the original goal of using highly focused effort to both improve struggling schools AND learn new strategies to help other schools would be a welcome change.

Now, Grace Tatter of Chalkbeat reports the ASD is being scaled back and re-focused. She notes:

In Tennessee’s plan under the Every Student Succeeds Act, the State Department of Education clipped the ASD’s wings with new policies approved this spring by the legislature. They address longstanding concerns, including complaints that the state district had moved beyond its original purpose, lacked a clear exit strategy, and didn’t give local districts enough time to execute their own turnaround plans.

McQueen also announced plans to downsize the ASD’s structure this summer by slashing its team and merging several ASD-related offices in Memphis.

It will be interesting to watch how the “new” ASD evolves. Will it really focus on building partnerships and clear communication? Or, will it revert back to the posturing that caused problems as it grew in Memphis?

For more on education politics and policy in Tennessee, follow @TNEdReport


 

WTF, TNReady?

In what is sure to be surprising news to parents, students, and school district leaders, TNReady quick score results are delayed yet again. Hard to predict this type of disaster since it only happens every year for the past four years now.

This delay is due to a problem the vendor is having with new test scanning equipment.

Here’s a statement from Hamilton County Schools explaining the news they received from the Tennessee Department of Education:

Tennessee Education Commissioner Dr. Candice McQueen informed school directors state-wide today that TNReady assessment vendor Questar has experienced challenges with the scanning software and capacity to scan testing materials; therefore, the company will not be able to meet the raw score deadlines set.

Dr. McQueen has asked Questar for updates on district-specific timing, and says the company currently estimates the scores will be delivered for the state’s upload process no later than the week of June 12.

Questar released a statement, which I have attached to this email.

The message from Dr. McQueen says she plans to meet with several groups this summer, including the TOSS Board, to come up with solutions that may prevent Tennessee testing vendors from missing the deadline set for report cards for a third year in a row. Her office promises to keep districts posted as the state receives more information.

To their credit, Questar is taking responsibility. That’s a change in tone from previous testing vendor Measurement, Inc.

Here’s what Questar had to say:

A Statement from Questar Regarding Raw Score Delivery

Questar committed to spring TNReady raw score delivery on a timeline and terms we set and which were communicated extensively by the Tennessee Department of Education (TDOE), however, our company has experienced scan set-up difficulties that will affect that timeline. Questar recently completed an upgrade to our scanning programs for paper tests, and we underestimated the time needed to fully integrate these upgrades. Therefore, any delays to the timeline are not the fault of the Tennessee Department of Education or the districts across the state. We take complete responsibility for this project, and we are actively working to meet our commitments to the state, districts, and educators. The purpose of this communication is to provide an update on the progress we are making to publish raw scores for grades 3-8 and End-of-Course assessments.

End-of-Course Status Update: Nearly all paper and online EOC raw scores are uploaded to EdTools. We appreciate the communication from the TDOE to reprioritize EOC delivery to ensure districts had these results as soon as possible.

3-8 Status Update: Questar is currently working through delays in scan processing for the 3-8 raw scores, which impact the timeline we previously provided to the TDOE for raw score delivery. We have experienced several software delays with our scan programming that we are actively resolving, and we are working to increase capacity within our scanning operations. We are committed to working with TDOE to mitigate these challenges in order to scan and score TNReady 3-8 assessments as quickly as possible. We will provide additional details and timelines, including district-specific estimations, as we make progress over the week.

The raw score delays will not impact the timeline for delivery of final score reports. Our scoring operations are unaffected by the scanning software issues, and we anticipate providing EOC final score reports this summer and 3-8 score reports this fall after the cut score process is complete, as previously announced.

We understand the importance of having the raw scores to communicate information to educators, students, and families, and we apologize for the inconvenience our delays have caused TDOE and our district partners in getting this information on the timeline we committed to months ago.

Questar sincerely appreciates our partnership with TDOE and the Tennessee educators and students we serve.

Tennessee has unique timing requirements for raw score delivery as compared to other states so we will continue to work with the department and districts to ensure that the future score return process is successful.

-Brad Baumgartner, Chief Partner Officer

At least we’re now paying millions of taxpayer dollars to a vendor who will take some heat, right?
This part of the Hamilton County statement is interesting:
The message from Dr. McQueen says she plans to meet with several groups this summer, including the TOSS Board, to come up with solutions that may prevent Tennessee testing vendors from missing the deadline set for report cards for a third year in a row.
Here’s a solution: Don’t count TNReady scores in student final grades. In fact, since McQueen readily admits that raw score data is of little actual value and since different methods of factoring quick scores lead to vastly different scores assigned to students, why even bother?
It’s clearly been a problem to get reliable quick score data back to districts. The data is of dubious value. Legislation should advance that eliminates the use of TNReady data in student grades.
As I’ve said before, TNReady may well be a solid test properly aligned to standards. Such a test can yield useful information. A more student-centered approach to testing would focus on project-based assessment and use tools like TNReady to provide diagnostic information for students, teachers, and schools.
For more on education politics and policy in Tennessee, follow @TNEdReport


 

 

It Doesn’t Matter Except When It Does

This year’s TNReady quick score setback means some districts will use the results in student report cards and some won’t. Of course, that’s nobody’s fault. 

One interesting note out of all of this came as Commissioner McQueen noted that quick scores aren’t what really matters anyway. Chalkbeat reports:

The commissioner emphasized that the data that matters most is not the preliminary data but the final score reports, which are scheduled for release in July for high schools and the fall for grades 3-8. Those scores are factored into teachers’ evaluations and are also used to measure the effectiveness of schools and districts.

“Not until you get the score report will you have the full context of a student’s performance level and strengths and weaknesses in relation to the standards,” she said.

The early data matters to districts, though, since Tennessee has tied the scores to student grades since 2011.

First, tying the quick scores to student grades is problematic. Assuming TNReady is a good, reliable test, we’d want the best results to be used in any grade calculation. Using pencil and paper this year makes that impossible. Even when we switch to a test fully administered online, it may not be possible to get the full scores back in time to use those in student grades.

Shifting to a model that uses TNReady to inform and diagnose rather than evaluate students and teachers could help address this issue. Shifting further to a project-based assessment model could actually help students while also serving as a more accurate indicator of whether they have met the standards.

Next, the story notes that teachers will be evaluated based on the scores. This will be done via TVAAS — the state’s value-added modeling system. Even as more states move away from value-added models in teacher evaluation, Tennessee continues to insist on using this flawed model.

Again, let’s assume TNReady is an amazing test that truly measures student mastery of standards. It’s still NOT designed for the purpose of evaluating teacher performance. Further, this is the first year the test has been administered. That means it’s simply not possible to generate valid data on teacher performance from this year’s results. You can’t just take this year’s test (TNReady) and compare it to the TCAP from two years ago. They are different tests designed to measure different standards in a different way. You know, the old apples and oranges thing.

One teacher had this to say about the situation:

“There’s so much time and stress on students, and here again it’s not ready,” said Tikeila Rucker, a Memphis teacher who is president of the United Education Association of Shelby County.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

It Wasn’t Me

TNReady results may or may not be included in your student’s report card, though we do know that more than 75% of districts won’t get scores back before the end of May. Don’t worry, though, it’s nobody’s fault.

Certainly, it’s not the responsibility of the Department of Education or Commissioner Candice McQueen to ensure that results are back in a timely fashion.

Today, Commissioner McQueen sent out an update to educators about assessments. There was some interesting information about TNReady going forward and about the timeline for scores for this year’s tests.

Not included? Any sort of apology about the TNReady quick score issue.

Instead, here’s what McQueen had to say:

Finally, I want to share an update on the delivery of raw scores for the 2016-17 assessment. We have received raw score data for nearly all EOC subjects, and grades 3–8 data continues to come in daily. We are in communication with your district leaders regarding the delivery of raw score data. State law and state board rule provides district choice on whether to include TNReady in grades if scores are not received within five days of the end of the school year. If you have questions about your particular district’s timeline or any decisions about including TNReady data in grades, I encourage you to reach out to your local leaders.

Got a problem or question about TNReady data and your student’s scores? Don’t ask Candice McQueen or the Department of Education. Ask your local leaders. Because, after all, we’ve been giving them all the relevant information in the most timely fashion.

I would suggest that leaders at TDOE just apologize and say it won’t happen again. But, as I mentioned, we’ve had testing challenges for four consecutive years now.

Here’s one word of advice to district leaders and teachers: Next year, when the Department of Education says everything is fine, it just might not be. Here’s something you can count on, though: It won’t be the responsibility of anyone at TDOE.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Not Exactly Helpful

In a story yesterday about TNReady scores not being ready in time to be counted in student final grades, I noted a statement published in the Clarksville Leaf-Chronicle and attributed to Tennessee Department of Education spokesperson Sara Gast. Here’s that statement again:

But Sara Gast, spokeswoman for the state Department of Education, said school districts would receive their scores based on how quickly they returned their materials.

This was the first week school districts could receive data back, and districts across the state will get their scores on a rolling basis over the next couple of week through the week of June 5, she said.

She said some districts will not get their scores in time to be counted in final grades “because they did not meet the deadlines.”

Since then, Clarksville-Montgomery County Schools has posted an update on their Facebook page:

The state department of Education has clarified that CMCSS did NOT miss any deadlines. According to Sara Gast from the Tennessee Department of Education, “We provide three different timelines for a reason, and all are equally fine and acceptable for districts to be on. We are neutral on which deadlines districts meet, and it is reasonable that larger districts would need additional time to ship materials back and may use the entire window to do so. We have always fully expected that we will have districts on all three tracks based on their local decisions.” Assistant Education Commissioner Nakia Towns confirmed that with this comment: “We emphasized that there was no “miss” of deadlines. We just provided three timelines.”

What’s not clear from this statement is whether it was anticipated that scores would not be ready by the end of school depending on the track chosen by districts.

It’s also interesting how the DOE’s explanation has shifted from blaming districts for missing deadlines to now saying that having more than 75% of districts not getting scores back before the end of the year was the plan all along.

I offered a solution yesterday. It’s simple, really.

Stop using this single test as the primary indicator of student performance, teacher effectiveness, and school accountability.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

 

The Buck Stops Nowhere

By now it is clear that TNReady simply didn’t go as planned this year. Sure, the tests were administered and students completed them — largely on pencil and paper. But, the raw data from the tests used to generate the “quick scores” for student grades isn’t getting back to districts on time. At least not in time to include it in report cards.

The Department of Education says that’s the fault of districts. Here’s how a TN DOE spokesperson explained it to the Clarksville Leaf-Chronicle:

But Sara Gast, spokeswoman for the state Department of Education, said school districts would receive their scores based on how quickly they returned their materials.

This was the first week school districts could receive data back, and districts across the state will get their scores on a rolling basis over the next couple of week through the week of June 5, she said.

She said some districts will not get their scores in time to be counted in final grades “because they did not meet the deadlines.”

The district says that’s simply not the case and that they met the established timeline:

Shelton said tests from all Clarksville-Montgomery County schools were definitely turned in according to the state’s timeline to have them back before schools let out Wednesday, and the school district was not at fault.

All of this may sound a bit familiar to those who watched the blame game play out last year during the total meltdown of TNReady.

Then, neither testing vendor Measurement, Inc. nor the Department of Education took responsibility for a test that clearly failed.

Here’s what’s interesting this year. The state admits that more than 75% of districts will not get scores back on time. This, they claim, is because of missed deadlines. Districts dispute that claim.

Educators will tell you that if more than 75% of students miss a test question or fail a test, there’s a problem – and the problem is with the question or the test, not the students. More than 75% of districts supposedly missed established deadlines. It seems clear the Department of Education has a problem. It’s also clear that adults are failing kids.

Students are told the tests matter. They are told the tests factor into grades. Schools are held accountable based on results. Despite statistical validity issues, the test is used to evaluate teachers — so, those teachers reinforce the importance of the test to their students.

Then, after the answer sheets have been bubbled in and the test booklets sent to the vendor for grading, students are told the tests won’t count. Or, they may count, but report cards will be late. But it’s okay. The adults got it wrong. Just keep showing up and taking the test seriously. No big deal.

If you want the students to take testing seriously, you should take your job seriously. This is the fourth year in a row that there have been challenges with testing and/or returning results. When district leaders have stood up and spoken out, Commissioner McQueen has threatened to withhold funding.

Here’s a crazy idea: Stop using this single test as the primary indicator of student performance, teacher effectiveness, and school accountability.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

 

Rhetoric vs. Reality: TNReady 2017 Edition

Here’s what Commissioner of Education Candice McQueen had to say about this year’s TNReady tests in an email sent to educators on May 16th:

This year’s administration of TNReady was a success, both online and on paper, in schools across Tennessee. Thank you for you partnership and for preparing students with strong instruction every day. Stay tuned as we continue to share updates and resources.

Since then, it’s become clear that TNReady results won’t actually be ready for most districts in a timely fashion — meaning they’ll either be excluded from student grades or report cards will be held until results are available.

WPLN’s Blake Farmer reports on the scope of the problem:

The state department of education says less than a quarter of districts finished testing in time to get the results by the end of May. For those that did wrap up early, they started getting results back this week.

Yes, that’s right. More than 75% of districts won’t have results back before the end of May.

That’s an odd definition of success.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

McQueen Joins Chiefs for Change

Education Week reports that Tennessee’s Education Commissioner, Candice McQueen, has joined education reform group Chiefs for Change. Her predecessor, Kevin Huffman, was also a member of the group that has advocated for expanding charter schools and using value-added data in teacher evaluations.

Here’s more:

Chiefs for Change, once part of the former Florida Gov. Jeb Bush’s Foundation for Excellence in Education and comprised solely of state education chiefs, has spun off as an independent organization.

And on the addition of McQueen among the group’s four new members:

The lone state chief in the group is Candice McQueen, Tennessee’s commissioner of education.

While McQueen has been Tennessee’s Education Commissioner for three years now, she is new to Chiefs for Change.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Opportunity to Learn

Natalie Coleman, Sumner County Teacher and HSG Tennessee Teacher Fellow

originally posted on TNTeacherTalk

 

Any teacher can tell you that students who miss too much school are at a disadvantage compared to their peers. Regardless of whether absences are a result of illness, personal reasons, or suspensions, missed time in school is detrimental to the individual student’s learning. The Tennessee Department of Education hopes to improve students’ opportunity to learn by reducing absenteeism. In Tennessee’s Every Student Succeeds Act (ESSA) plan, one nonacademic indicator for school and district accountability is the “chronically out of school” metric, which will evaluate progress in reducing the number of students who miss ten percent or more of the school year.

Before finalizing the state’s ESSA plan, the TDOE tasked the Hope Street Group Tennessee Teacher Fellows with collecting feedback from teachers across the state of Tennessee about their experiences with chronic absenteeism and with student discipline. This spring, the Fellows released a report based on the valuable input of over 2,000 teachers who participated in an online survey and nearly 400 who provided their insights in focus groups. The report includes six recommendations that the Fellows presented to Commissioner McQueen and the TDOE and is now available for the benefit of all stakeholders in Tennessee education.

The report details the results of the survey and summarizes the trends of teachers’ comments in focus groups, and a look through the report shows many connections between teachers’ experiences and the recommendations made to the Department of Education.

Recommendation 1 focuses on helping schools and teachers address the problem of students chronically missing school. Based on the survey data, even though 95% of teachers affirmed that chronic absenteeism affects student achievement, many teachers also reported that they have received little or no training in how to reduce student absences. 90% of teachers reported that they had not received training on strategies for reducing chronic absenteeism, and 92% reported that they were unfamiliar or only somewhat familiar with the state’s initiatives in addressing this issue. In response to this feedback, the Fellows recommend, “To ensure that teachers are fully aware of TDOE efforts, CORE offices could build teacher awareness of the draft ESSA plan (2016) through trainings that highlight key plan features that are designed to reduce chronic absenteeism.”   

Recommendation 2 seeks to provide schools and teachers with more resources to address this issue. On the Fall 2016 survey, 69% of teachers reported that they believe problems at home are the most significant barrier to student attendance, but only 30% report that they are aware that Family Resource Centers are available to help families and students who struggle with absenteeism. In fact, teachers who chose to write in their own answers about the family support services offered by their schools overwhelmingly responded with none. As a result, the second recommendation says, “To alleviate teacher concerns about this issue, TDOE could build awareness of an increased TDOE focus in 2017 on reducing chronic absenteeism through Family Resource Centers. Additionally, TDOE could remind teachers of the 103 Family Resource Centers in 78 districts and highlight the various needs-based services and training provided to parents and families through these centers.”

Recommendations 3 and 4 focus on student behavior and discipline. In focus groups, teachers shared various obstacles they encounter in implementing effective discipline policies. The third recommendation connects these teacher concerns to resources the TDOE could provide in conjunction with Response to Instruction and Intervention for Behavior (RTI2-B): “Because TDOE focuses on RTI2-B in the draft ESSA plan (2016), TDOE could expand the RTI2-B framework to reach more districts and schools through CORE offices or Tennessee Behavior Supports Project (TBSP), thereby providing additional targeted support in areas highlighted as obstacles by teachers.” The fourth highlights strategies for improving student behavior that are both research-based and frequently cited by teachers themselves in their focus group responses: “Through CORE offices or Tennessee Behavior Supports Project (TBSP), TDOE could emphasize how the following teacher suggestions for improving student behavior are research-based and addressed in RTI2-B: promoting positive behavior and prevention efforts and encouraging restorative behavior practices; involving parents in student behavior efforts; nurturing positive student-teacher relationships; and providing appropriate consequences in response to student behavior issues.” This recommendation encourages the TDOE to promote these research-based practices which teachers also know to be effective.

Recommendation 5 addresses the all-too-familiar concern of bullying in school. 14% of teachers report they feel unprepared or very unprepared to handle incidents of bullying in their classrooms, and 20% rate the effectiveness of their schools’ response to bullying as ineffective or very ineffective. These numbers show that many schools and teachers need additional support in addressing the issue of bullying and validate the fifth recommendation: “Because 20 percent of teachers shared that their schools’ response to bullying is ineffective, TDOE could provide resources to CORE offices for dissemination to districts and schools.”

Recommendation 6 highlights previous Hope Street Group findings about RTI2 and urges using prior teacher feedback to inform implementation of RTI2-B, Response to Instruction and Intervention for Behavior, which features in the state’s draft ESSA plan. This recommendation reads: “TDOE could revisit the recommendations provided in the Spring 2016 Hope Street Group Report on RTI2, including those related to scheduling and structuring RTI2; promoting whole school support and reducing negative perceptions of RTI2 effectiveness; and providing funding for additional RTI2 resources (e.g., professional development) and staffing.” This previous report, detailing teacher feedback regarding RTI2, is also available on the Hope Street Group website.

To learn more, visit the Hope Street Group Tennessee Teacher Fellows website and download the full 2016-17 report. You can also stay connected by liking and following the Tennessee Teacher Fellows’ Facebook page, Tennessee Teacher Voice.

 

For more on education politics and policy in Tennessee, follow @TNEdReport


 

It May Be Ready, But is it Valid?

In today’s edition of Commissioner Candice McQueen’s Educator Update, she talks about pending legislation addressing teacher evaluation and TNReady.

Here’s what McQueen has to say about the issue:

As we continue to support students and educators in the transition to TNReady, the department has proposed legislation (HB 309) that lessens the impact of state test results on students’ grades and teachers’ evaluations this year.

In 2015, the Tennessee Teaching Evaluation Enhancement Act created a phase-in of TNReady in evaluation to acknowledge the state’s move to a new assessment that is fully aligned to Tennessee state standards with new types of test questions. Under the current law, TNReady data would be weighted at 20 percent for the 2016-17 year.

However, in the spirit of the original bill, the department’s new legislation resets the phase-in of growth scores from TNReady assessments as was originally proposed in the Tennessee Teaching Evaluation Enhancement Act. Additionally, moving forward, the most recent year’s growth score will be used for a teacher’s entire growth component if such use results in a higher evaluation score for the teacher.

We will update you as this bill moves through the legislative process, and if signed into law, we will share detailed guidance that includes the specific options available for educators this year. As we announced last year, if a teacher’s 2015-16 individual growth data ever negatively impacts his or her overall evaluation, it will be excluded. Additionally, as noted above, teachers will be able to use 2016-17 growth data as 35 percent of their evaluation if it results in a higher overall level of effectiveness.

And here’s a handy graphic that describes the change:

TNReady Graphic

 

 

Of course, there’s a problem with all of this: There’s not going to be valid data to use for TVAAS. Not this year. It’s bad enough that the state is transitioning from one type of test to another. That alone would call into question the validity of any comparison used to generate a value-added score. Now, there’s a gap in the data. As you might recall, there wasn’t a complete TNReady test last year. So, to generate a TVAAS score, the state will have to compare 2014-15 data from the old TCAP tests to 2016-17 data from what we hope is a sound administration of TNReady.

We really need at least three years of data from the new test to make anything approaching a valid comparison. Or, we should start over building a data-set with this year as the baseline. Better yet, we could go the way of Hawaii and Oklahoma and just scrap the use of value-added scores altogether.

Even in the best of scenarios — a smooth transition from TCAP to TNReady — data validity was going to be challenge.

As I noted when the issue of testing transition first came up:

Here’s what Lockwood and McCaffrey (2007) had to say in the Journal of Educational Measurement:

We find that the variation in estimated effects resulting from the different mathematics achievement measures is large relative to variation resulting from choices about model specification, and that the variation within teachers across achievement measures is larger than the variation across teachers. These results suggest that conclusions about individual teachers’ performance based on value-added models can be sensitive to the ways in which student achievement is measured.
These findings align with similar findings by Martineau (2006) and Schmidt et al (2005)
You get different results depending on the type of question you’re measuring.

The researchers tested various VAM models (including the type used in TVAAS) and found that teacher effect estimates changed significantly based on both what was being measured AND how it was measured.

And they concluded:

Our results provide a clear example that caution is needed when interpreting estimated teacher effects because there is the potential for teacher performance to depend on the skills that are measured by the achievement tests.

If you measure different skills, you get different results. That decreases (or eliminates) the reliability of those results. TNReady is measuring different skills in a different format than TCAP. It’s BOTH a different type of test AND a test on different standards. Any value-added comparison between the two tests is statistically suspect, at best. In the first year, such a comparison is invalid and unreliable.

So, we’re transitioning from TCAP to TNReady AND we have a gap in years of data. That’s especially problematic — but, not problematic enough to keep the Department of Education from plowing ahead (and patting themselves on the back) with a scheme that validates a result sure to be invalid.

For more on education politics and policy in Tennessee, follow @TNEdReport