Category Archives: Candice McQueen

Posted on by

User Error

Dear Teachers,

It’s your fault.

It always is.

That’s essentially the sentiment expressed by the Tennessee Department of Education led by Candice McQueen after the latest round of problems, this time with portfolio evaluation of Pre-K/Kindergarten teachers.

The Tennessean has more:

Tennessee’s teacher union is blaming a vendor glitch for issues with some teachers’ low kindergarten and prekindergarten portfolio scores. But the state says the problems are due to user error.

“There was no error by our vendor. The vendor has double-checked all of the peer review scores and everything has been correctly and accurately reported,” according to a statement from Sara Gast, Tennessee Department of Education spokeswoman.

But Gast said Monday that portfolios are reviewed and scored by peers through a rubric. In some cases, Gast said, an educator mismatched students or standards, which made it impossible to score. In that case, she said, educators were given a score of 1.

The Department of Education, an entity with a serious allergy to the truth, is blaming teachers instead of accepting responsibility.

This is the same DOE that seemed surprised when May 15th arrived this year and portfolio reviewers hadn’t been provided guidance:

The initial portfolios were to be evaluated by May 15th. Then, the portfolios with score disputes go on to the “experts.”

Here’s the text of an email about that sent on May 15th:

Dear Educator,

Thank you for all your hard work! The portfolio scoring in the general pool concludes at 11:59pm tonight. The consensus review scoring begins tomorrow, Wednesday, May 16, 2018.

In the event that you were unable to meet your 10 portfolio review requirement (the same as 40 collections) AND you have demonstrated competence during the certification process and/or general pool scoring, you may receive additional portfolios to score. Reviewers who will receive additional portfolio submissions in this next phase and Expert Reviewers will be provided additional guidance to support the scoring process.

Thanks for all that you do! Please look for our next communication in 24 hours.

Here’s a follow-up email sent on May 16th:

Thanks again for your patience and support. We are still developing the guidance documents for the next phase of peer review. Our goal is to make sure you have the most comprehensive and best information to be successful. We appreciate your understanding and will communicate in the next 24-48 hours with updates.

This is also the same DOE that gave teachers one rubric for preparing their portfolios while providing reviewers with a rubric with significantly more difficult standards by which to assess those same portfolios.

Teachers received:

Reviewers were given:

This is the same DOE that set a June 15th deadline for returning scores, then moved it to June 30th, then released the scores last week — in late July.

By all means, let’s give Commissioner McQueen and her department the benefit of the doubt despite all the mishaps during her tenure at the helm.

In Candice McQueen’s world, it’s blame everyone all the time and it’s NEVER her fault or her responsibility.

Never fear, though, the state is now switching to a new platform for portfolio submissions. This means rolling out new training for teachers well after the academic year has started. For teachers in the few districts using Fine Arts Portfolios, this will be the third platform for submission in the last three years. Yes, each year is spent preparing for the portfolio collection and submission AND learning a new platform well into the school year.

If one wonders what Governor Haslam thinks of Tennessee’s teachers, let’s be clear: He’s been standing steadfastly behind Commissioner McQueen. In short, he doesn’t respect our teachers or the work they do.

Frankly, any lawmaker not demanding McQueen be held to account is complicit in this mistreatment of our teachers. The message is and has been clear: Everyone is accountable and responsible EXCEPT the Commissioner of Education. Teachers will continue to pay the price and must go along because no one with authority will stand up and make this stop.

I’d caution those sitting silently to note the teacher uprisings in places like West Virginia, Kentucky, Oklahoma, and Arizona. We may be inviting just this sort of direct action here.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Posted on by

Eric Goes Back to School

As the 2017-18 school year came to a close, we reviewed the Kindergarten experience of a student named Eric and his teacher. Eric’s teacher was participating in an exciting new mandate from the Tennessee Department of Education known as a portfolio. The idea was to assess that teacher’s ability to impart knowledge by looking at her ability to move Eric and other students from Point A to Point B according to state standards.

Here’s how Eric’s year ended:

Eric has now just about completed Kindergarten. He knows no other “normal” environment for school. Complete the task, be recorded, do it again at the end of the year.

The story doesn’t note the hours his teacher spent tagging evidence and uploading it instead of (or in addition to) preparing for learning activities for her class. The story also isn’t over. The results of the first year of statewide, mandatory Kindergarten portfolios have not been recorded.

Now that the 2018-19 school year is about to start, we return to the questions posed as teachers waited on portfolio review and scoring:

How many parents are aware that their child is spending time in Kindergarten working as evidence collection specimens for a system used to assign a 1-5 number for their child’s teacher? How many know just how much instructional time is lost to this process?

Results?

Teachers were initially told by the Department of Education that results would be back by June 15th. Then, they were told it would be June 30th. Since the TN DOE has never met a deadline it could find ways to miss, June 30th came and went with no scores available to teachers. Also missing: Any clear explanation or accountability from the state.

Finally, on July 26th, the Department of Education sent a memo that included these lines teachers across the state had been waiting for:

The department is excited to announce that early grades portfolio scores (pre-K, kindergarten, and first grade) are now available. TNCompass displays the Teacher Effectiveness Indicator, which will be used as the 35 percent student growth measure in calculating the level of overall effectiveness (LOE);

Finally, after a frustrating year of jumping through hoops and losing instructional time, teachers could see the results. According to the state’s propaganda, these results would help teachers improve future instruction, so all that lost time helping kids would be worth it in the long haul. Plus, all that frustration was just because the system was new, it would get better with time.

But wait, this is the Tennessee Department of Education we’re talking about. It’s not like they have a track record of treating teachers fairly or getting evaluation and assessment right.

So, you’re saying there were problems?

Yes!

Initially, teachers noted relatively low scores. That might be expected in the first year of a new evaluation. However, closer examination revealed the low scores were the result of a grading error. For many teachers, entire sections were not reviewed. When a section was not reviewed, that section’s assessment score defaulted to a 1 – the lowest possible score. Because of this default to a low score, a teacher’s entire portfolio score was impacted.

A number of teachers shared score reports that had one or more sections that had not been reviewed.

What’s going on?

Well, it’s time for another memo from the Department of Education. Here’s what they say happened:

Portfolios are designed to demonstrate student growth across time and for standards which teachers select and are assessed through student work. In cases where evidence of student growth is significantly below expectations, these collections would receive a score of 1.

Collections could also receive a score of 1 because of a mismatch either between samples of students or between the standards that were chosen at Point A and the standards that were chosen at Point B. When the standards are mismatched in either of these ways, those portfolio collections/submissions were both flagged by a peer reviewer and autoscored as a 1. • It’s important to note that, for teachers who used all standards consistently from Point A to Point B for all three samples of students, no autoscore was generated, and the collection was scored by a peer reviewer and an expert reviewer if needed. If the difference between these two scores is more than one performance level, an expert reviewer scores the collection/submission, providing an additional level of review. If an expert reviewer scores a collection/submission, those scores are considered final.

So, a teacher who has the highest possible score on three sections and a 1 on a single section either demonstrated below expectations growth OR mismatched the collection and the standard — in other words, it was the teacher’s fault.

An analysis of department guidance by staffers at the Tennessee Education Association noted:

Based on the TDOE’s initial guidance, the cause of the debacle is a glitch in the Educopia platform that allowed a “mismatch” to occur between the two data collection points.  According to TDOE, this mismatch could result in one of two ways, either from a lack of continuity in the student sample or in the standard selected.  Apparently, the Educopia platform allowed erroneous mismatched data to be submitted without alerting the submitter to the apparent mismatch.  When that happened, the platform proceeded to “autoscore” the mismatched collections assigning a default score of “1” rather than flagging such collections for further human review and possible correction. This procedure was not in accordance with the TDOE’s published guidance.

 

The Hotline reviewed all of the TDOE guidance prior to the release of the portfolio scores and it appears there was nothing to advise teachers that errant mismatched data submissions were possible, or that such mismatches could result in an autoscore of “1” for the collection.  To the contrary, TDOE’s guidance repeatedly suggests all collections would be subjected to a progressive human review process in response to any discrepancies greater than one performance level.  So not only does it appear there was no way teachers reasonably could have known these mismatch errors could occur, there was nothing to alert teachers to the possibility such mismatched collections could receive an autoscore of 1 without later being reviewed.

Look at that again: The Tennessee Department of Education — the same department that blamed this year’s TNReady problems on non-existent hackers and incredibly powerful dump trucks — is now blaming the Pre-K/K portfolio problems on a computer glitch. Boy, we sure do have bad luck with testing and evaluation vendors. I’m sure glad our DOE is not to blame, that’d be terrible if they were held accountable for all these crazy coincidences.

A New Approach

Thankfully, Eric has been happily enjoying his summer, blissfully ignorant of the challenges his teacher is having with all that portfolio madness. Now, though, he’s ready to go back to school and the teachers are feeling defeated as their Commissioner of Education and the department she leads continue to disrespect them.

Good news! Now, the DOE is going to provide a new portfolio vendor — one without all those pesky glitches! Here’s more on that:

One way that we plan to reduce the amount of time for portfolio development and provide enhanced support is through a partnership with Portfolium, Inc. to give educators a new e-portfolio platform: TNPortfolio. TNPortfolio will house TEAM portfolios in 2018-19, and in the future, it will support educator micro-credentialing and Work-Based Learning Student ePortfolio systems. In supporting the TEAM student growth portfolios, the platform provides functionality that will automate and improve several processes associated with portfolio development. The improved platform will allow schools to track student work samples longitudinally across multiple years. The TEAM Portfolio element of the platform will be available for teacher use on Aug. 24. On that date, you will receive an email from portfolium.com to access the system for the first time. All portfolio scoring guides and other resources will be available on that date.

YAY! A new, better vendor. Where have we heard this before? This is coming from the same Department of Education that fired Measurement, Inc. from TNReady and hired Questar. Both of them flopped. Then, in an attempt at trickery, the DOE announced Questar was being replaced by the company that owns Questar?!?

Still, there’s hope, right? I mean, this new vendor, Portfolium, is a tried and true teacher evaluation provider ready for the task, right? Well, maybe…

Here’s a bit more about the new company taking over the portfolio process — not just for Pre-K and Kindergarten teachers, but for all teachers submitting portfolios:

SAN DIEGO, July 26, 2018 /PRNewswire-PRWeb/ — The Tennessee Department of Education has awarded the contract for its TEAM student growth portfolio, student Work-Based Learning, and educator micro-credential ePortfolio systems to Portfolium, Inc., the only Student Success Network.

Portfolium’s TNPortfolio platform, which can accommodate up to 500,000 users, will allow Tennessee teachers to demonstrate the impact of classroom instruction on student growth through the TEAM student growth portfolio, allow students to demonstrate the value of the education they’re receiving through work-based learning, and allow educators to acquire and maintain micro-credentials earned through personalized professional learning. The TEAM Portfolio element of the platform will be available for teacher use starting August 24; other elements of the platform will be available for use during the 2019-20 school year.

Note that the Portfolium press release announcing the awarding of the State of Tennessee contract came on July 26th, the same day as the state’s memo to teachers regarding this year’s portfolio scores.

Who is Portfolium?

Portfolium is a startup company designed to provide college students with a way to highlight accomplishments and work samples for future employers. Yes, you read that right: The new evaluation platform is a startup company that was founded in 2013 and just three years ago, began raising a small amount of capital to launch:

Portfolium, a Web-based social network for students preparing to start their careers, said it has closed on $1.2 million in new venture funding, bringing its total funding to $2.1 million since 2013, when the San Diego-based startup was founded.

I mean, sure, this could work out great. But let’s be cautious — this is a partnership between a Department of Education with a record of failed testing ventures and a startup company seeking to expand market share. What could go wrong?

Oh, and all that time and energy teachers spent learning how to use Educopia? Gone out the window. Time to learn a new platform and see if it works.

In fact, the DOE plans to roll out training for the new platform via webinar on August 23rd. Teachers will have login credentials for Portfolium on August 24th.

Parents, you should be aware that your child’s Pre-K/K teachers may be just a tad frustrated as the year starts — it’s probably not your kid (I mean, it might be). It’s most likely the poor treatment they are receiving at the hands of the Tennessee Department of Education.

How much longer must we tell Eric’s story before policymakers in Nashville make changes that support rather than deflate our teachers?

Have a portfolio experience you’d like to share? Email me at andy@tnedreport.com

For more on education politics and policy in Tennessee, follow @TNEdReport


 

 

 

Posted on by

Haslam’s TNReady Pit

Chalkbeat has a story demonstrating just how out of touch outgoing Governor Bill Haslam is. The story details Haslam’s belief that tying teacher evaluation to TNReady results is a key element in Tennessee’s recent education success.

Here’s some truth: Over the past few years, Tennessee has seen high school graduation rates and average ACT scores climb while also seeing the number of students requiring remediation at state schools decline. All of that is encouraging. All of it happened in a climate where the TNReady test was unreliable and poorly administered. In other words, Tennessee’s testing system had nothing to do with student performance. All other indicators point to teachers getting the job done and students hitting ever higher marks.

Here’s more truth:

Does basing teacher evaluation on student test scores get results that impact student outcomes?

No.

That’s the conclusion from a years-long study funded by the Gates Foundation that included Memphis/Shelby County Schools.

It’s also worth noting that while Haslam touts the “fastest-improving” NAEP results from back in 2013, further evidence suggests the results then were likely an outlier.

Here’s more from Chalkbeat:

Gov. Bill Haslam says he had a “pit” in his stomach every day of Tennessee’s testing season this spring when a parade of technical problems vexed students and teachers in the bumpy transition to computerized exams.

He also worries that three straight years of frustrations with the state’s 3-year-old standardized assessment, TNReady, could unravel policies that he believes led to students’ gains on national tests.

“Do we really want to go back? Do we really want to go back to when Tennessee was in the 40s out of the states ranked 1 to 50?”  the outgoing Republican governor asked recently in an exclusive interview with Chalkbeat.

First, no serious policymaker is suggesting Tennessee adopt weaker or lower standards for students.

Second, as noted above, other significant indicators demonstrate Tennessee students are improving — even without a reliable annual test.

Third, Haslam’s “beliefs” about policies have not been tested on a statewide level – in part due to the failure of his own Administration to execute the tests. Haslam has allowed Commissioner of Education Candice McQueen to keep her job despite multiple testing failures with different vendors. In fact, Haslam joined McQueen in touting a “new” testing vendor that turned out to actually be the parent company of the current vendor.

More from Chalkbeat:

“Hopefully Tennessee and the new administration won’t have the same struggles we’ve had this year with testing. But there will be some struggles; there just are by the very nature of it,” he said. “I worry that the struggles will cause us to say, ‘OK, we give. We’re no longer going to have an evaluation that’s tied to an assessment.’

To this, I’d note that experts suggest no state has had a more tumultuous transition to online testing than Tennessee:

“I’m not aware of a state that has had a more troubled transition” to online testing, said Douglas A. Levin of the consulting group EdTech Strategies.

In terms of an evaluation tied to an assessment, even if TNReady had gone well, the results in the initial years would not be in any way valid for use in teacher evaluation. That’s because the nature of value-added assessment requires multiple years of similar testing in order to produce results that are even vaguely reliable predictors of teacher performance. Here’s a bit more on that:

Here’s what Lockwood and McCaffrey (2007) had to say in the Journal of Educational Measurement:

We find that the variation in estimated effects resulting from the different mathematics achievement measures is large relative to variation resulting from choices about model specification, and that the variation within teachers across achievement measures is larger than the variation across teachers. These results suggest that conclusions about individual teachers’ performance based on value-added models can be sensitive to the ways in which student achievement is measured.
These findings align with similar findings by Martineau (2006) and Schmidt et al (2005)
You get different results depending on the type of question you’re measuring.

The researchers tested various VAM models (including the type used in TVAAS) and found that teacher effect estimates changed significantly based on both what was being measured AND how it was measured.

And they concluded:

Our results provide a clear example that caution is needed when interpreting estimated teacher effects because there is the potential for teacher performance to depend on the skills that are measured by the achievement tests.

If you measure different skills, you get different results. That decreases (or eliminates) the reliability of those results. TNReady is measuring different skills in a different format than TCAP. It’s BOTH a different type of test AND a test on different standards. Any value-added comparison between the two tests is statistically suspect, at best. In the first year, such a comparison is invalid and unreliable. As more years of data become available, it may be possible to make some correlation between past TCAP results and TNReady scores.

But, TNReady hasn’t gone well. At all. It’s been so bad, the Department of Education has been unveiling a bunch of pie charts to demonstrate how they are attempting to correlate test scores and teacher evaluation. First, it went like this:

Second, this chart is crazy. A teacher’s growth score is factored on tests from three different years and three types of tests.

15% of the growth score comes from the old TCAP (the test given in 2014-15, b/c the 2015-16 test had some problems). Then, 10% comes from last year’s TNReady, which was given on paper and pencil. Last year was the first year of a full administration of TNReady, and there were a few problems with the data calculation. A final 10% comes from this year’s TNReady, given online.

So, you have data from the old test, a skipped year, data from last year’s test (the first time TNReady had truly been administered), and data from this year’s messed up test.

There is no way this creates any kind of valid score related to teacher performance. At all.

After teachers expressed outrage that the DOE was going to count this year’s scores in their evaluations, the legislature finally took action and passed legislation that said teachers could face “no adverse action” based on this year’s test results.

So, now the Department of Education has more pie charts and a lot of explanations:

What is included in teacher evaluation generally?

There are many factors that go into a teacher’s overall evaluation. One of those, the individual growth component (in gray in the charts in this document), is typically based on a three-year TVAAS measure if data is available. However, for the phase-in period there are two key items to note for the growth component:

• If the current single-year year growth score – in this case, 2017-18 data – provides the educator with a higher overall composite, it will be used as the full growth score.

• Additionally, if a teacher has 2017-18 TNReady data included in any part of their evaluation, they will be able to nullify their entire LOE this year.

What is included in teacher evaluation in 2017-18 for a teacher with 3 years of TVAAS data?

There are three composite options for this teacher:

• Option 1: TVAAS data from 2017-18 will be factored in at 10%, TVAAS data from 2016-17 will be factored in at 10% and TVAAS data from 2015-16 will be factored in at 15% if it benefits the teacher.

• Option 2: TVAAS data from 2017-18 and 2016-17 will be factored in at 35%.

• Option 3: TVAAS data from 2017-18 will be factored in at 35%. The option that results in the highest LOE for the teacher will be automatically applied. Since 2017-18 TNReady data is included in this calculation, this teacher may nullify his or her entire LOE this year.

And if you only have one or two years of TVAAS data or if you teach in a non-tested subject? Well, the key line continues to apply: Since 2017-18 TNReady data is included in this calculation, this teacher may nullify his or her entire LOE this year.

What does this mean? Well, it means you’d have a year with no evaluation score. Sounds fine, right? No. It’s not fine. In order to achieve tenure, a teacher must have consecutive years of evaluation scores at Level 4 or 5. But a year with no score at all means that teacher would then need to have TWO MORE YEARS of high scores in order to be tenure eligible. While it seems unlikely a teacher would choose to nullify their entire score if they achieved a high rank, it also seems only fair to allow that teacher to simply exclude the TNReady data and receive their LOE rating based on all the other factors that go into a TEAM rating.

But wait, excluding 2017-18 TNReady data is NOT an option provided. It’s either count it as 10%, count it as 35%, or nullify your entire LOE score. Doing so could certainly have an adverse impact on a teacher.

In short, the TNReady mess has made teacher evaluation a mess. Still, a host of indicators suggest Tennessee’s teachers are hitting the mark. One might conclude that tying a suspect teacher evaluation model to an unreliable test is, in fact, not the key to educational progress in our state. Unfortunately, Governor Bill Haslam has concluded the opposite.

For more on education politics and policy in Tennessee, follow @TNEdReport

Help keep the education news coming!


 

Posted on by

Meaningless Scores Make Big Splash

The Tennessee Department of Education today released results from this year’s troubled administration of TNReady testing.

This Tennessean story indicates Commissioner McQueen is taking the results seriously in spite of what has become an annual inability to get the job done right:

“We see reason to be encouraged, but we also have a lot of work to do to meet our higher expectations for all students,” Education Commissioner Candice McQueen said in a statement. “While we’ve focused extensively on early grades reading and are starting to see a shift in the right direction, we know middle school remains a statewide challenge across the board. TNReady serves as a vital feedback loop for teachers, parents, and administrators to tell us where we are, and the results inform what steps we need to take to help all students and schools succeed.”

Umm…what?? Are you even serious, Candice? THE TESTS DIDN’T WORK! Also, you were caught lying time and again about WHY they didn’t work.

The legislature passed legislation at the end of session, during testing to account for the failure of TNReady.

This isn’t even the first year the tests didn’t work. It happens EVERY year.

Here’s the deal: These scores can’t credibly be used to tell us ANYTHING. Students in schools all across the state faced disruptions caused by dump trucks and hacking (also known as the ineptitude of DOE leadership). Some kids got the wrong test and the extent of that problem is not totally known.

Note to district leaders: If you use these results to say your district or a given school are “doing well,” you have ZERO credibility. Treating these results as anything other than the complete trash that they are is unacceptable.

Some districts have already begun pushing for change in either how TNReady happens next year or even a totally different type of test.

Oh, and note that the DOE makes it sound like they are making improvements:

The Tennessee Department of Education also announced changes after this year’s problems, including searching for a new vendor. It also adjusted how fast it will phase-in online testing.

That sounds great, but the truth is, the DOE and Governor Haslam are treating Tennesseans like we’re stupid:

Let’s get this straight: Governor Haslam and Commissioner McQueen think no one in Tennessee understands Google? They are “firing” the company that messed up this year’s testing and hiring a new company that owns the old one and that also has a reputation for messing up statewide testing.

So, we had a testing season full of lies, deception, disruption, and mixed-up tests but we’re supposed to look at the “results” of those tests and take them seriously? No thanks.

 

For more on education politics and policy in Tennessee, follow @TNEdReport

Help keep the education news coming!


 

Posted on by

It’s All Been a Pack of Lies

By now, it should come as no surprise that our Commissioner of Education and the department she leads has a troubled relationship with the truth. That said, today’s revelation at a legislative hearing that an alleged hack of the state’s TNReady test didn’t actually happen again raises the question: Why does Candice McQueen still have a job?

Back on April 17th, the day after TNReady failed to work on day one of this year’s testing, the Tennessee Department of Education noted that the Day 2 failures were related to someone hacking the vendor:

At a legislative hearing today, the Tennessee Bureau of Investigation (TBI) indicated there was no evidence of a hack.

Additionally, the Department of Education issued this statement, which notes:

  • It appears, thankfully, that there was not an outside actor who attacked Questar’s data system. No student data was breached.
  • It is now clear that the event that Questar initially thought presented like a denial of service attack on Tuesday, April 17 was not created by an external actor with malicious intent, but, rather, can be traced in large part to the caching issues connected to how text-to-speech was configured by Questar.
  • Questar implemented a significant and unauthorized change to text-to-speech, which had previously operated successfully during the state’s fall administration. We now know this decision led to the severity of other issues we experienced during online testing.
  • Questar continues their internal investigation and is cooperating with additional external audits to make sure we have all of the facts.

Questar’s Chief Operating Officer Brad Baumgartner has provided this statement: “Questar’s internal and external investigations indicate that the source of the anomalous data pattern is believed to be the result of a configuration with the cache server. We have applied a configuration change and believe to have resolved the issue. We will continue to work with our internal technology team and external partners to validate this.”

The text-to-speech feature was also blamed for students receiving the wrong tests.

While at the time, the hacking excuse sounded pretty far-fetched, today’s hearing confirms that the Department advanced a lie offered by the state’s testing vendor. Of course, later on in the testing cycle, a dump truck was blamed for disrupting testing. That excuse was also later proven untrue.

All of this may explain why at least one school district is calling for a significant reduction in TNReady testing next year.

If this year had been the first time our state had faced testing challenges, one might understand (and forgive) the excuse-making. However, this is now the fifth consecutive year of some sort of problem and the fourth year testing administration has been, to say the least, a challenge.

One may recall the saga of Measurement, Inc. The company that hired test graders from Craigslist and was ultimately fired in 2016 after that year’s TNReady test failed.

The bottom line: If TNEdu tells you something about testing, you should question it. The track record shows that to our state’s Department of Education, truth is a relative concept.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Your support keeps the education news coming!

Posted on by

One Glaring Exception

That’s how this article in Education Week defines the TNReady testing experience.

It starts like this: Just about everything that could go wrong did go wrong with online state testing this year in Tennessee.

Yep.

The piece walks through the saga that has been TNReady. Here are some highlights:

Then, thanks to human error at some schools, about 1,400 students ended up taking the wrong version of the TNReady exam

Except it wasn’t human error at the schools. As I reported on April 26th, the Department of Education said about the issue:

 

“There was a poorly designed feature of the online testing system that contributed to some users accidentally administering a test to students that was below their grade level, including those at Norris Middle School. We’ve provided guidance to the district staff and the building testing coordinator to invalidate these tests. Students are not required to re-test, and their tests will not be scored.

Then, again with the dump truck:

And a rogue dump truck severed one of the state’s main fiber-optic cables, causing temporary connectivity problems during the testing period.

Except not really:

“There is no evidence this was anything other than a side effect of the issue with the fiber cut, but we continue to look into it,” Sara Gast, a spokeswoman for the state education department, said last week.

But internet provider Education Networks of America disputes that, saying that the West Tennessee issues were not related to the cable cut.

What happened in those cases remains a mystery, for now.

Unanticipated?

The article says:

On the second day of testing, Questar was flooded with unanticipated traffic that overwhelmed the company’s servers and prevented some students from connecting to the TNReady testing platform.

How was the testing traffic unanticipated? Was Questar counting on a bunch of students missing school on the second day of testing? Did they not know how many students would be logging on ant the relative times that would happen? They were paid $30 million to figure that out… and didn’t.

While lots of states are moving to online testing, one expert says Tennessee is unique:

“I’m not aware of a state that has had a more troubled transition” to online testing, said Douglas A. Levin of the consulting group EdTech Strategies.

 

Why is Tennessee in the unique position of having the worst online testing transition in the country?

The reality is that Tennessee’s online-testing mess has left everyone in a difficult position, said Chad Aldeman, a principal at Bellwether Education Partners, a consulting organization.

“The state has not [made] stability a key priority in their testing vendors,” Aldeman said.

 

Ultimately, responsibility should rest at the feet of Education Commissioner Candice McQueen, who so far has avoided any accountability for the ongoing testing mishaps in the state.

For more on education politics and policy in Tennessee, follow @TNEdReport

Help keep the education news coming!

Posted on by

Story Time

Our story begins in the early weeks of Kindergarten with a student we’ll call Eric.

Eric is excited about starting school. He loves the new friends he’s making and he really likes his teacher.

Very early in the academic year, all the students are handed a small packet of worksheets as the teacher and a teaching assistant set up an iPad at a table in the room. The students are told to work quietly and that each of them will be called to the teacher to answer a few questions.

Eric’s name is called. His teacher explains what’s happening, that he’ll be asked a few questions and he’ll be recorded by the assistant. After the teacher establishes that Eric is comfortable with what’s about to happen, she poses a question. Eric’s mind searches, and he offers an answer. Now, he has to demonstrate his understanding. The assistant is aiming the iPad at him while attempting to watch the other 19 students in the classroom.

The interaction takes a little more than 5 minutes. The teacher and assistant make sure the event is recorded and labeled and set up for the next student. Eric returns to his desk and begins the worksheets.

Eric and his classmates (and all Kindergarten students in Tennessee) are participating in an evidence collection that is required as part of the Tennessee Educator Acceleration Model (TEAM) Pre-K/Kindergarten portfolio. This portfolio of student work will be used to determine the effectiveness of Eric’s teacher.

At the end of the school year, another Kindergarten teacher who has been trained in the evaluation process will review the portfolio submitted by Eric’s teacher and assign a score. This score will be combined with classroom observations and an achievement measure to determine whether or not Eric’s teacher was “effective.”

The portfolio was required for all Kindergarten teachers for the first time this school year. The idea is that since Kindergarten teachers have students who don’t take TNReady, there has to be some way to evaluate their effectiveness besides classroom observation. Previously, these teachers received a “growth score” based on the school’s overall growth as determined by testing results.

Teachers submit evidence of students performing at high, middle, and low levels on standards at the beginning of the year and then at the end of the year. This requires evidence collection at various points, the most cumbersome being at the beginning of the year, when the students are largely unknown to the teachers.

Kindergarten teachers I talked to estimate the evidence collection process takes up a minimum of five instructional days. This means students aren’t actively engaged in the learning process during the evidence collection days. As in the scenario with Eric, it requires the full attention of the teacher (and if possible, an assistant) in order to collect the evidence. This doesn’t include the tagging of evidence or the uploading to an often unreliable online platform known as Educopia. Some districts report hiring subs on evidence collection days so teachers can document the evidence from their students.

One might suspect the same Department of Education that can’t coordinate a statewide test administration would also have difficulty coordinating the evaluation of Kindergarten teachers via an online portfolio system. Such a suspicion is proving to be correct as we come to the end of the first year of this mandated system.

Here’s one example. At the beginning of this school year, teachers were provided with a rubric to indicate the demonstrated skills for various performance levels. Here’s what that rubric indicated was a level 4:

Now, the submitted evidence is graded by a Kindergarten teacher who has been “trained” and who receives a very small stipend to complete the evaluation. Here’s what the evaluation rubric indicates is a Level 4:

So, is it WITH or WITHOUT prompts? Maybe we should ask U2’s Bono?

Imagine teaching all year and basing your evidence collection and tagging on one rubric only to discover that you are being evaluated on a different, more difficult standard? Oh, and this is only one of the many standards included in the portfolio evaluation.

Moving on to the scoring process, each teacher self-scores the submitted portfolio. Then, another teacher evaluates. If the scores are more than two levels apart, an “expert” receives the portfolio to make a determination.

What do those experts have that the initial teacher evaluating did not? First, a willingness to assess even more portfolios. Second, “guidance” from the Tennessee Department of Education.

The initial portfolios were to be evaluated by May 15th. Then, the portfolios with score disputes go on to the “experts.”

Here’s the text of an email about that sent on May 15th:

Dear Educator,

Thank you for all your hard work! The portfolio scoring in the general pool concludes at 11:59pm tonight. The consensus review scoring begins tomorrow, Wednesday, May 16, 2018.

In the event that you were unable to meet your 10 portfolio review requirement (the same as 40 collections) AND you have demonstrated competence during the certification process and/or general pool scoring, you may receive additional portfolios to score. Reviewers who will receive additional portfolio submissions in this next phase and Expert Reviewers will be provided additional guidance to support the scoring process.

Thanks for all that you do! Please look for our next communication in 24 hours.

Here’s a follow-up email sent on May 16th:

Thanks again for your patience and support. We are still developing the guidance documents for the next phase of peer review. Our goal is to make sure you have the most comprehensive and best information to be successful. We appreciate your understanding and will communicate in the next 24-48 hours with updates.

So, no guidance YET for those scoring the second round. This despite the fact the portfolios were required for all teachers THIS year after being piloted by a few districts last year.

The Department of Education has had two full years to develop guidance for “consensus review scoring” and it is still not available. In fact, according to these two emails, the guidance is being developed right now. Was the Department of Education surprised that May 15th actually arrived this year?

How many parents are aware that their child is spending time in Kindergarten working as evidence collection specimens for a system used to assign a 1-5 number for their child’s teacher? How many know just how much instructional time is lost to this process?

Eric has now just about completed Kindergarten. He knows no other “normal” environment for school. Complete the task, be recorded, do it again at the end of the year.

The story doesn’t note the hours his teacher spent tagging evidence and uploading it instead of (or in addition to) preparing for learning activities for her class. The story also isn’t over. The results of the first year of statewide, mandatory Kindergarten portfolios have not been recorded.

Due to complaints at the start of this school year, legislators passed “hold harmless” legislation that will not allow this year’s portfolio results to negatively impact a teacher’s overall evaluation score. This may sound familiar, as “no adverse action” legislation was passed for those teachers impacted by TNReady scores.

Eric’s story is just one more example of a Department of Education that claims victory when the evidence suggests much improvement is needed. It’s a Department hellbent on pursuing supposedly lofty goals no matter the consequences to students or their teachers.

Lost instructional time due to portfolio evidence collection? No problem!

Days of stress and chaos because TNReady doesn’t work? Outstanding!

Teachers faced with confusing, invalid evaluations? Excellent!

Eric and his teachers and Tennessee’s schools and communities deserve better.

Do you have a portfolio story to share? Email me at andy@tnedreport.com

For more on education politics and policy in Tennessee, follow @TNEdReport

Keep the education news coming!


 

Posted on by

You’re Fired….uh, Hired!

The Chattanooga Times Free Press notes that Governor Bill Haslam and Education Commissioner Candice McQueen are considering ending the state’s relationship with Questar:

Gov. Bill Haslam said the state is conducting an independent review of its current contractor running the problem-plagued TNReady student testing system and, depending on its findings, the company could be out of the picture once its current contract ends in November.

The likely replacement for Questar is Education Testing Service (ETS):

McQueen said that in addition to the state’s third-party review of Questar’s operations, the state is already going to move “all of our test development and design” to Educational Testing Services, which she said has a “reputation for very high quality work.”

Sounds great, right? Firing the vendor that was baffled by hackers and dump trucks and replacing them with a group with a solid reputation.

Except for just one thing:

Education Testing Services, the global billion dollar nonprofit that administers more than 50 million tests (including the GRE and TOEFL) across the world, recently sealed an agreement to acquire Questar, a Minnesota-based for-profit testing service, for $127.5 million. According to the press release, Questar will become a separate for-profit subsidiary of ETS.

Questar offers what they describe as a “fresh and innovative” method of testing for grades 3-8—providing states with summative assessments, design support, scalable technological innovation, administrative help, scoring and reporting services.

Ok, so maybe ETS will step in and give its baby brother Questar some guidance going forward? Well:

The changes highlight a possible strategic shift for ETS whose reputation came under fire last year when the nonprofit had to pay $20.7 million dollars in damages and upgrades after multiple testing problems in Texas.

Let’s get this straight: Governor Haslam and Commissioner McQueen think no one in Tennessee understands Google? They are “firing” the company that messed up this year’s testing and hiring a new company that owns the old one and that also has a reputation for messing up statewide testing.

Solid move.

For more on education politics and policy in Tennessee, follow @TNEdReport

Keep the education news coming!


 

YOU’RE FIRED Rubber Stamp over a white background.

Posted on by

An “F” for TNReady

The Johnson City Press offers a grade to the state for TNReady testing this year and it’s not a good one.

Here’s some of what they had to say:

Tennessee deserves a resounding “F” for TNReady. Schools should be able to test their pupils without hampering instruction and limiting the scope of education, and both parents and teachers should be able to have confidence in the scores. Surely, other states have a model Tennessee can apply.

The writers note that this is not the first year of testing trouble:

For more than a quarter century, Tennesseans have watched the state’s Department of Education fumble around with standardized testing and school accountability measurements. The last four years have been especially comical, leaving teachers and parents without a consistent understanding of achievement while squandering valuable learning time for students.

TNReady Irony?

So the state leapt into the ironically named TNReady, a new set of tests replacing the TCAP, in 2015. TNReady has been a disaster from the word go. The first year, the state canceled the online tests altogether for grades 3-8 and fired the original vendor, which failed to integrate the test online.

Last fall, the problems mounted as the Department of Education announced a new vendor had incorrectly scored about 9,400 TNReady assessments, affecting 70 schools in 33 districts. This year, that same vendor was the victim of what state officials described as a deliberate cyberattack, and connectivity issues slowed the whole system, thrusting everything into question yet again.

While the legislature took some action this year to address the immediate crisis, the state’s next Governor and the 2019 General Assembly should carefully examine our state’s testing culture. In the meantime, local school boards should be more aggressive in pushing back against a Commissioner of Education who has exhibited indifference to the chaos caused by years of bad testing management.

 

For more on education politics and policy in Tennessee, follow @TNEdReport

Keep the education news coming!


 

Posted on by

Not So Harmless

After a fourth day of TNReady trouble, the Tennessee General Assembly took action today to make changes to how the test impacts schools, students, and teachers.

While some are billing the report of a joint committee of the House and Senate as a “hold harmless” for schools, students, and teachers, that’s not entirely accurate.

Also, the legislature stopped short of putting a stop to TNReady entirely, claiming federal law “requires” them to test students.

Here’s the deal: Federal law does say that districts should administer tests to at least 95% of students and that states should test all students in reading and math from grades 3-8 and at least once in high school, with a suggestion for additional high school testing as appropriate.

BUT: Is there really a penalty for districts (or states) where the testing threshold falls below 95%?

As I reported in 2016, the last time we had a major failure of online testing in Tennessee:

There’s just one problem: The federal government has not (yet) penalized a single district for failing to hit the 95% benchmark. In fact, in the face of significant opt-outs in New York last year (including one district where 89% of students opted-out), the U.S. Department of Education communicated a clear message to New York state education leaders:  Districts and states will not suffer a loss of federal dollars due to high test refusal rates. The USDOE left it up to New York to decide whether or not to penalize districts financially.

In other words, the likelihood of a single Tennessee district losing funds due to stopping a test that isn’t working is very close to zero. Tennessee is not having problems due to opt-outs or a low number of students being tested. Kids in districts across the state are showing up for a test that is not happening. Districts are doing everything right and a vendor and the Tennessee Department of Education are failing to serve students. Unless TNDOE is going to fine districts, there is truly no risk of funds being lost.

Now, about the “hold harmless” law (pictured below):

  1. The law does say that districts and schools will not receive an “A-F” score based on the results of this year’s test. It also says schools can’t be placed on the state’s priority list based on the scores. That’s good news.
  2. The law gives districts the option of not counting this year’s scores in student grades. Some districts had already said they wouldn’t count the test due to the likelihood the scores would arrive late. Now, all districts can take this action if they choose.
  3. The law says any score generated for teachers based on this year’s test cannot be used in employment/compensation decisions.

Here’s what the law didn’t say: There will be NO TVAAS scores for teachers this year based on this data.

Commissioner McQueen said yesterday that the data from these tests will be used to generate a TVAAS score and it will count for 20% of a teacher’s evaluation. This law does NOT change that. It just says if you get a low score based on this number, you can’t be fired or denied compensation.

Below is an excerpt from current law (taken from TCA 49-1-302, the section governing teacher evaluation):

(E)  For teachers with access to individual data representative of student growth as specified in subdivision (d)(2)(B)(ii), the following provisions shall apply:

  • (i)  In the 2016-2017 school year, the evaluation criteria identified in subdivision (d)(2)(B)(ii) shall be adjusted so that student growth data generated by assessments administered in the 2016-2017 school year shall account for ten percent (10%) of the overall evaluation criteria identified in subdivision (d)(2)(B);
  • (ii)  In the 2017-2018 school year, the evaluation criteria identified in subdivision (d)(2)(B)(ii) shall be adjusted so that student growth data generated by assessments administered in the 2016-2017 and 2017-2018 school years shall account for twenty percent (20%) of the overall evaluation criteria identified in subdivision (d)(2)(B);
  • (iii)  In the 2018-2019 school year and thereafter, the student growth component of the evaluation criteria shall be determined under subdivision (d)(2)(B)(ii);
  • (iv)  The most recent year’s student growth evaluation composite shall account for the full thirty-five percent (35%) of growth data required in a teacher’s evaluation if such use results in a higher evaluation score;

  • (v)  For the 2015-2016 through 2017-2018 school years, student growth evaluation composites generated by assessments administered in the 2015-2016 school year shall be excluded from the student growth measure as specified in subdivision (d)(2)(B)(ii) if such exclusion results in a higher evaluation score for the teacher or principal. The qualitative portion of the evaluation shall be increased to account for any necessary reduction to the student growth measure.

Here’s what this means: If the current tests give you a “good” evaluation score, it will count for 35% of your total evaluation. If the score is not “good,” it only counts for 20% this year. The legislation adopted today by way of the Conference Committee does NOT change that.

In other words, the test data from the 2017-18 administration of TNReady WILL count in a teacher’s evaluation.

Here’s why that matters: An educator’s evaluation score factors into the number of observations they have each year as well as Professional Development Points (PDPs). PDPs are needed for license advancement or renewal.

The Department of Education addresses PDPs and notes:

Overall level of effectiveness rating (approved TN model) Overall Score of 5 = 20 PDPs
Overall Score of 4 = 15 PDPs

Overall Score of 3 = 10 PDPs

 Information is maintained by the department. No additional documentation is required; points may be accrued annually.

Even if this year’s scores only end up counting 20%, that’s enough to change a teacher’s overall TEAM rating by a level. A TEAM score below a three means no PDPs, for example. The overall TEAM score also impacts the number of observations a teacher has in a year — which also places an additional burden on administrators.

Also, districts now have to meet to decide how to handle the tests and student grades. For some, that decision has already been made. For others, this will require a meeting in pretty short order to let students, parents, and teachers know what’s happening.

Here’s the language of the conference committee report:

 

For more on education politics and policy in Tennessee, follow @TNEdReport

Your support is appreciated and helps keep content like this coming!