Listen to the Money Talk

Does basing teacher evaluation on student test scores get results that impact student outcomes?

No.

That’s the conclusion from a years-long study funded by the Gates Foundation that included Memphis/Shelby County Schools.

Education Week reports:

The Bill & Melinda Gates Foundation’s multi-million-dollar, multi-year effort aimed at making teachers more effective largely fell short of its goal to increase student achievement—including among low-income and minority students, a new study found.

Under its intensive partnerships for effective teaching program, the Gates Foundation gave grants to three large school districts—Memphis, Tenn. (which merged with Shelby County during the course of the initiative); Pittsburgh; and Hillsborough County, Fla.—and to one charter school consortium in California starting in the 2009-10 school year. The foundation poured $212 million into these partnerships over about six years, and the districts put up matching funds. The total cost of the initiative was $575 million.

The school sites agreed to design new teacher-evaluation systems that incorporated classroom-observation rubrics and a measure of growth in student achievement. They also agreed to offer individualized professional development based on teachers’ evaluation results, and to revamp recruitment, hiring, and placement. Schools also implemented new career pathways for effective teachers and awarded teachers with bonuses for good performance.

During the course of this failed experiment, Tennessee as a state also implemented the TEAM evaluation system and encouraged districts to offer merit pay schemes to teachers. Additionally, the state used a turnaround strategy for “low-performing” schools known as the Achievement School District. Data released after five years of that project indicates it has made essentially no impact on student outcomes.

Also, for the past four years, Tennessee has been attempting to administer TNReady — to no avail.

Tennessee policymakers are spending millions on education experiments that have yielded no results.

Here’s one thing that hasn’t changed: In 2010, Tennessee was ranked 45th in investment in education per student. In 2017, we’d improved — all the way up to 43rd.

Instead of directing funds to experiments that end up not doing much of anything, perhaps we should be investing our dollars in our schools and teachers. Then, we should also try the one thing we haven’t: Dramatically increasing our per pupil investment in schools.

Tennessee should be funding excellent teacher pay instead of trying to get and keep teachers at discount rates.

Tennessee should be investing in school buildings, to ensure all students have a safe, excellent environment in which to learn.

If Tennessee really wants to turn the tide, we ought to invest like it — ask teachers what they need to be successful and put our money there. For too long, education reform has been something “done to” teachers instead of done with them.

Here’s what we don’t need: Another round of expensive experiments that will leave our students and schools right where we started – behind.

We can do better — we know the answer. Does Tennessee have the political will to make lasting change for our schools through sustained investment in the people that make them work?

For more on education politics and policy in Tennessee, follow @TNEdReport

Keep the education news coming!


 

Beyond TNReady

At least one school system in Tennessee is taking steps to move beyond TNReady. According to a story in the Wilson Post, Wilson County Schools is seeking legislative action that would allow them to choose and administer their own annual tests in place of the state-mandated TNReady.

Here’s more:

Wilson County Director of Schools Dr. Donna Wright told county commissioners Monday the local school system is pursuing a private act from the state Legislature that would allow it to use an assessment other than the one currently mandated.

In her monthly report to the commission, Wright expressed her dissatisfaction with the TNReady test, saying that, “We are four years in without any or little actionable data that teachers can use.”

Wright added that while district leaders support accountability, the lack of timely, reliable data from the state tests is problematic:

“We are absolutely advocates of accountability because that’s how we know what to improve and where to improve,” Wright said adamantly. “But the fallacy in all this is that we haven’t had an effective system in four years, but we still keep using information that is not only in error, but late in coming.”

The action in Wilson County follows a resolution passed in Johnson City calling for a significant reduction in state-mandated testing.

The movement to reduce or replace TNReady follows yet another year of testing problems and a litany of excuses offered by the Department of Education and the state’s testing vendor.

Wright is correct that mishaps in testing and the late return of results call the usefulness of the data into question. However, even in the best of circumstances, it would be difficult to arrive at valid, actionable data based on the early years of a new test.

It will be interesting to see if other school systems follow the lead of Johnson City and Wilson County. Perhaps we’re finally seeing district leaders stand up and say “enough!”

For more on education politics and policy in Tennessee, follow @TNEdReport

Your support keeps the education news coming!


 

It’s All Been a Pack of Lies

By now, it should come as no surprise that our Commissioner of Education and the department she leads has a troubled relationship with the truth. That said, today’s revelation at a legislative hearing that an alleged hack of the state’s TNReady test didn’t actually happen again raises the question: Why does Candice McQueen still have a job?

Back on April 17th, the day after TNReady failed to work on day one of this year’s testing, the Tennessee Department of Education noted that the Day 2 failures were related to someone hacking the vendor:

At a legislative hearing today, the Tennessee Bureau of Investigation (TBI) indicated there was no evidence of a hack.

Additionally, the Department of Education issued this statement, which notes:

  • It appears, thankfully, that there was not an outside actor who attacked Questar’s data system. No student data was breached.
  • It is now clear that the event that Questar initially thought presented like a denial of service attack on Tuesday, April 17 was not created by an external actor with malicious intent, but, rather, can be traced in large part to the caching issues connected to how text-to-speech was configured by Questar.
  • Questar implemented a significant and unauthorized change to text-to-speech, which had previously operated successfully during the state’s fall administration. We now know this decision led to the severity of other issues we experienced during online testing.
  • Questar continues their internal investigation and is cooperating with additional external audits to make sure we have all of the facts.

Questar’s Chief Operating Officer Brad Baumgartner has provided this statement: “Questar’s internal and external investigations indicate that the source of the anomalous data pattern is believed to be the result of a configuration with the cache server. We have applied a configuration change and believe to have resolved the issue. We will continue to work with our internal technology team and external partners to validate this.”

The text-to-speech feature was also blamed for students receiving the wrong tests.

While at the time, the hacking excuse sounded pretty far-fetched, today’s hearing confirms that the Department advanced a lie offered by the state’s testing vendor. Of course, later on in the testing cycle, a dump truck was blamed for disrupting testing. That excuse was also later proven untrue.

All of this may explain why at least one school district is calling for a significant reduction in TNReady testing next year.

If this year had been the first time our state had faced testing challenges, one might understand (and forgive) the excuse-making. However, this is now the fifth consecutive year of some sort of problem and the fourth year testing administration has been, to say the least, a challenge.

One may recall the saga of Measurement, Inc. The company that hired test graders from Craigslist and was ultimately fired in 2016 after that year’s TNReady test failed.

The bottom line: If TNEdu tells you something about testing, you should question it. The track record shows that to our state’s Department of Education, truth is a relative concept.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Your support keeps the education news coming!

If this is what success looks like…

In a story about the Tennessee Department of Education scaling back the requirements for online testing next year in light of this year’s testing challenges, this caught my attention:

Even with the problems this year, it was one of the most successful online administrations for the state to date. More than 2.5 million TNReady tests were administered this spring, with about 300,000 students taking the test online. Only high school students were required to take the online version this year.

What does the word “success” mean? Because my recollection of this year’s TNReady administration is that it was a debacle.

I’m not the only one. As I noted last week:

While lots of states are moving to online testing, one expert says Tennessee is unique:

“I’m not aware of a state that has had a more troubled transition” to online testing, said Douglas A. Levin of the consulting group EdTech Strategies.

And there’s this helpful explainer:

Why is Tennessee in the unique position of having the worst online testing transition in the country?

The reality is that Tennessee’s online-testing mess has left everyone in a difficult position, said Chad Aldeman, a principal at Bellwether Education Partners, a consulting organization.

“The state has not [made] stability a key priority in their testing vendors,” Aldeman said.

Nevertheless, Commissioner of Education Candice McQueen says:

The state will put out a request for contract proposals in the fall, with a new vendor to be identified in the spring. Questar Assessment could again win the contract, but McQueen said who wins the proposal will have to show the ability and history of seamlessly administering an online test.

“We look for a company with a track record of success in administering online testing and who can manage our test well.”

Haven’t we heard that before?

For more on education politics and policy, follow @TNEdReport

Keep the education news coming!


 

Definitely Something Wrong

The Texas Tribune reports:

A couple of weeks after Texas penalized its main testing vendor over glitches with thousands of standardized tests, another potential testing mishap is under investigation after more than 100 students in a high-performing Houston-area high school received zeros on their English essays.

Valerie Vogt, chief academic officer at Lamar Consolidated Independent School District, said she was confused this spring when about 157 students at George Ranch High School, which generally performs higher than state average on standardized tests, received zeros on their English 1 and English 2 essays. In the other four high schools in the district, just 10 or fewer students received zeros on the essays.

“There’s definitely something wrong,” she said.

The testing vendor responsible is Educational Testing Services (ETS), which owns Tennessee’s testing vendor, Questar. This is the latest in a series of problems with ETS in Texas:

Last month, the TEA levied a $100,000 penalty against ETS after tens of thousands of Texas students were kicked out of the testing software or encountered connection problems when taking computerized tests in April and May. The agency also announced it would throw out the scores of students who experienced those glitches and reduce their effect on state accountability ratings for schools and districts.

Tennessee’s Department of Education announced recently ETS would be taking over more responsibility for TNReady after Questar’s administration of the testing this year was plagued by hackers and dump trucks.

Of course, ETS is not without a history of test administration problems. Edsurge.com notes:

The changes highlight a possible strategic shift for ETS whose reputation came under fire last year when the nonprofit had to pay $20.7 million dollars in damages and upgrades after multiple testing problems in Texas.

A recent analysis of the transition to online testing in the states indicates it is going well in most places, with Tennessee being the one glaring exception.

So, of course Tennessee hires the parent company of Questar — a company that has experienced consecutive years of testing problems in Texas — to come in and … make things right?

Yep, there’s definitely something wrong.

For more on education politics and policy in Tennessee, follow @TNEdReport

Your support keeps the education news coming!


 

Shrinking TNReady

That’s the hope behind a resolution passed by the Johnson City Board of Education this week.

The Johnson City Press reports:

Some changes included shrinking testing timing back from three weeks to one week for grades 3 through 8, pushing the writing assessment back to February to give the state more time to get grades in by the end of the school year, and drawing back on pre-K and kindergarten ELA assessments to be less time-consuming for teachers.

The proposal comes after another year of testing trouble in Tennessee. In fact, a recent report noted that while most states transitioning to online testing are doing so smooth, Tennessee is the one glaring exception.

Broad Support?

Now that the Johnson City School Board has given unanimous approval to this proposal, the Director of Schools hopes to spread the message to other districts and build support for changing TNReady:

What I’d like to do if the board approves this resolution is reach out to all the other school superintendents and talk to them about the resolution and get feedback from them,” Barnett said at the meeting. “I think we’d have some support.”

It’s possible this is the beginning of a move that will see district leaders stand up to the state and say “Enough!”

The Board also referenced the problematic implementation of portfolios to evaluate teachers in Pre-K/Kindergarten:

Anderson said that the state estimated those assessments would take about 15 to 17 hours, but some teachers reported spending as many as 44 hours on the project, most of that time being spent in the English Language Arts component of the assessment.

She added that portfolio assessment is considered an appropriate avenue to track student learning in those early grades, and the portfolios can be completed with video or audio taping or with written assessment.

“I don’t think anybody has anything against the concept of portfolios for pre-K (and kindergarten),” she said. “Though the piloting process went fairly well, it ended up morphing into a process this past year that I think was just very complicated and very unwieldy.

It will be interesting to see how the state moves forward in revising those portfolios and if there is any move toward making significant change in the TNReady tests.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

One Glaring Exception

That’s how this article in Education Week defines the TNReady testing experience.

It starts like this: Just about everything that could go wrong did go wrong with online state testing this year in Tennessee.

Yep.

The piece walks through the saga that has been TNReady. Here are some highlights:

Then, thanks to human error at some schools, about 1,400 students ended up taking the wrong version of the TNReady exam

Except it wasn’t human error at the schools. As I reported on April 26th, the Department of Education said about the issue:

 

“There was a poorly designed feature of the online testing system that contributed to some users accidentally administering a test to students that was below their grade level, including those at Norris Middle School. We’ve provided guidance to the district staff and the building testing coordinator to invalidate these tests. Students are not required to re-test, and their tests will not be scored.

Then, again with the dump truck:

And a rogue dump truck severed one of the state’s main fiber-optic cables, causing temporary connectivity problems during the testing period.

Except not really:

“There is no evidence this was anything other than a side effect of the issue with the fiber cut, but we continue to look into it,” Sara Gast, a spokeswoman for the state education department, said last week.

But internet provider Education Networks of America disputes that, saying that the West Tennessee issues were not related to the cable cut.

What happened in those cases remains a mystery, for now.

Unanticipated?

The article says:

On the second day of testing, Questar was flooded with unanticipated traffic that overwhelmed the company’s servers and prevented some students from connecting to the TNReady testing platform.

How was the testing traffic unanticipated? Was Questar counting on a bunch of students missing school on the second day of testing? Did they not know how many students would be logging on ant the relative times that would happen? They were paid $30 million to figure that out… and didn’t.

While lots of states are moving to online testing, one expert says Tennessee is unique:

“I’m not aware of a state that has had a more troubled transition” to online testing, said Douglas A. Levin of the consulting group EdTech Strategies.

 

Why is Tennessee in the unique position of having the worst online testing transition in the country?

The reality is that Tennessee’s online-testing mess has left everyone in a difficult position, said Chad Aldeman, a principal at Bellwether Education Partners, a consulting organization.

“The state has not [made] stability a key priority in their testing vendors,” Aldeman said.

 

Ultimately, responsibility should rest at the feet of Education Commissioner Candice McQueen, who so far has avoided any accountability for the ongoing testing mishaps in the state.

For more on education politics and policy in Tennessee, follow @TNEdReport

Help keep the education news coming!

Give Me My Money Back

If you thought Eric’s Story was the last word in the saga that is this year’s Kindergarten portfolio evaluation, you’d be wrong.

True to form, the Tennessee Department of Education has created another mess in the ongoing quest to catch and eliminate all those “bad teachers.”

In our previous visit to Kindergarten, here’s a bit of what we saw:

Moving on to the scoring process, each teacher self-scores the submitted portfolio. Then, another teacher evaluates. If the scores are more than two levels apart, an “expert” receives the portfolio to make a determination.

What do those experts have that the initial teacher evaluating did not? First, a willingness to assess even more portfolios. Second, “guidance” from the Tennessee Department of Education.

The initial portfolios were to be evaluated by May 15th. Then, the portfolios with score disputes go on to the “experts.”

Here’s the latest challenge: Portfolio evaluators were to evaluate 40 collections (10 portfolios) in order to complete their paid assignment. These teachers were paid $500 from the state for what was estimated to be 15 hours of work. Kindergarten teachers who are evaluators indicate the process takes more like 40-45 hours of work. At the end of May, as school years ended around the state, many portfolio evaluators had also completed their assessment of a minimum of 40 collections. Or so they thought.

Here’s an email a number of evaluators recently received (I have a copy of one sent June 1st):

Dear Peer Reviewer,

Based on the most current Educopia reports, you have not made progress in completing your commitment of 40 collections (=10 portfolios). Not completing this commitment may affect your stipend and/or future leadership opportunities in the portfolio work.

If you have questions and/or concerns, please let us know how we can assist you. If you are no longer able to review, you must notify your district point of contact.

Here’s an earlier email noting the expectations (sent April 25th):

Dear Peer Reviewer,

 

Thank you so much for your contribution to the peer review process! As some of you have already started live scoring, you may have noticed that you are able to keep track of the number of your scored submissions (collections). As stated in the Peer Review General Training Overview, the workload expectation is that reviewers score 40 submissions (collections). In the event that collections continue to be sent to your rater que after you have met this expectation, please know that these will be recirculated to other peer reviewers for scoring.

Here’s the problem:

Evaluators receiving the first email (indicating they had not completed the required minimum number of evaluations) were provided information via Educopia (the online platform for submitting/evaluating portfolios) saying they had completed evaluation of at least 40 collections.

Why are these same evaluators being told they have more work to do and also told the state is coming for their money?

Further investigation indicates that Educopia counted “practice” evaluations toward the initial total of 40. So, teachers conducting evaluations believed they had completed the process when some had as many as six actual portfolios remaining.

Now, the Department of Education is left with a number of unrated portfolios while evaluating teachers believe they are finished (and most likely have been paid).

Imagine that… the Tennessee Department of Education utilizing an online assessment platform that fails to deliver expected results!

Now, the question is: Will they come after teachers to claw back the meager stipends? How will the remaining portfolios be evaluated, or will they be evaluated at all?

Stay tuned…it’s still early this summer and the deadline for completion of this portfolio assessment has been extended until June 30th.

For more on education politics and policy in Tennessee, follow @TNEdReport

Keep the education news coming!


 

Honest Feedback

WREG out of Memphis has a story about the amount of time Tennessee students spend taking TNReady tests. It’s a topic I’ve written about before and one that continues to be relevant in light of ongoing challenges with the administration of the test.

Here’s more about the time students spend taking tests:

By the time a high schooler gets finished, he or she would have tested over three weeks for 590 minutes, that’s almost 10 hours.

That’s longer than it takes for tests for graduate school, law school even med school.

“To put it into perspective, if you are going to law school, the admissions test to become a lawyer is just 210 minutes,” said Cranford.

Middle schoolers aren’t far behind with a total of more than nine hours of testing.

“And the thing that really made me reach out to you was looking back at the third grade.”

The English Language Arts portion alone runs three hours and 36 minutes for third graders.

Cranford said, “That`s ridiculous.”

The total testing time for third graders is more than seven hours.

“If I was a parent of a third grader I would, and I saw these times, I think I would be gathering up a group of parents and contacting Nashville.”

While the total time spent testing is of concern, what compounds that frustration is that in the last three years, TNReady has experienced huge problems in two of those years. Last year, there were also issues with returning scores and with factoring the scores into teacher evaluations.

In light of these problems, the Department of Education’s response to the WREG story is particularly interesting. Here’s what they had to say about the value and importance of TNReady:

In large part because of TNReady, we are providing more honest feedback to families about their child’s performance, and our students are learning and growing to meet these high expectations. (See more here.) TNReady is a test that looks for students’ critical thinking and problem solving skills and is fully aligned to what our teachers are teaching. 

Let’s examine that claim more closely. My daughter was in fourth grade during the first disastrous administration of TNReady online. Because the State of Tennessee and then-vendor Measurement, Inc. could not effectively administer that test, there was NO feedback.  Assuming the test was an accurate reflection of what was to have been taught that year (a big assumption), there’s no way to know how my daughter or other students met those standards — the results didn’t come back. The state failed.

Of course, after that first year, Tennessee fired Measurement, Inc. That matter is now in court.

The next year, the test returned to pencil and paper and seemed to go mostly fine, except when it came to getting results back in a timely manner. Oh, and then there were problems with factoring the results in to teacher evaluation.

Then, this year, our test was hit by hackers and dump trucks and a bunch of students were given the wrong test. Now, there’s legislation that holds students harmless and also prevents any “adverse action” based on the test.

No serious person believes the results from this year’s test hold any real meaning. Of course, that means Candice McQueen puts a lot of faith in those results.

To be clear: In two of the last three years, there is no feedback at all — not honest, not dishonest, just nothing. Parents: When you get TNReady scores back this year, they will tell you nothing. Except that your child completed the test (maybe) and was (finally) able to submit an answer.

Oh, and there’s still no testing transparency. We can’t see the questions and answers, so we can’t be sure the tests are  “fully aligned to what our teachers are teaching.” 

Here’s some honest feedback: TNReady hasn’t worked. It didn’t work in year one. There were real problems in year two. This year’s administration was a debacle. In fact, going back to even the year before we started TNReady, there was a fiasco with quick scores.

More honesty: Over the past four years, with two different types of tests and multiple testing vendors, testing simply hasn’t worked in Tennessee. The one constant has been Candice McQueen. As a parent, I’ve had enough.

As if all of this weren’t enough, our state’s Education Commissioner and Governor appear to believe Tennesseans are too stupid to notice their bait and switch tactic regarding testing vendors. Alternatively, they may just believe no one cares.

The state’s Assessment Task Force keeps meeting. The Department of Education puts out more pie charts. The testing continues.

The TN DOE spokesperson closed the story by saying, “Now we need to focus on ensuring that administration of the test is seamless.”

Honestly?

For more on education politics and policy in Tennessee, follow @TNEdReport

Like stories like this? Show some love!


 

The State Continues to Fail

Here’s another take on “Eric’s Story” about the Kindergarten portfolio evaluation process. The bottom line: Teachers are being disrespected and students are losing valuable learning time. All in the name of assigning a number to teachers in an evaluation process that leaves much to be desired.

Here’s what this teacher had to say:

I’m a teacher that has experienced this process from the view of teacher, portfolio district lead, and portfolio reviewer. Also, being chosen for the second round of scoring. I received both the emails you discussed as well as a third stating I’d been chosen for more scoring with the “guidance document” attached.

So I begin my second round of scoring tomorrow. A process none of us knew would exist. We thought our deadline was May 15 on scoring and we would be done.

I spent two full 8 hour days trying to score submissions (pulled away from my kindergarten screening duties) only for them not to be available to me so I did not complete the task and score the number they wanted me to score. Was this my fault? No! I tried but the state wouldn’t push them out to us. So that’s why I was chosen for round two.

Now summer is beginning. Teachers need summer to recuperate mentally and prepare for our next class which we happily look forward to receiving. We don’t need to spend it stressing over continued work load.

MORE on K portfolios>

If you have a story to tell about the portfolio process or another aspect of the intersection between policy and practice, send it to: andy@tnedreport.com

For more on education politics and policy in Tennessee, follow @TNEdReport

Keep the stories alive!