That’s Not That Much, Really

So, statewide TCAP results are out and as soon as they were released, the Achievement School District (ASD) touted its gains.

Embedded image permalink

But, what does all that mean? How are these schools doing relative to the goal of taking them from the bottom 5% of schools to the top 25% within 5 years, as founder Chris Barbic boasted before his recent revelation that educating poor kids can be difficult.

Fortunately, Gary Rubinstien has done some analysis. Here’s what he found:

By this metric the top performing ASD school from the first cohort was Corning with a score of 48.6 followed by Brick Church (47.9), Frayser (45.2), Westside (42.1), Cornerstone (37.6), and Hume (33.1).  To check where these scores ranked compared to all the Tennessee schools, I calculated this metric for all 1358 schools that had 3-8 math and reading and sorted them from high to low.

The values below represent the school’s overall score and their percentile relative to the rest of the state, in that order.

Hume 33.1 1.5%
Cornerstone 37.6 2.6%
Westside 42.1 3.2%
Frayser 45.2 4.1%
Brick Church 47.9 5.2%
Corning 48.6 5.5%

As you can see, four of the original six schools are still in the bottom 5% while the other two have now ‘catapulted’ to the bottom 6%.  Perhaps this is one reason that Chris Barbic recently announced he is resigning at the end of the year.

So, the schools that have been in the ASD the longest, making the greatest gains, are at best in the bottom 6% of all schools in the state. That’s a long, long way from the top 25.

But here’s something else. Back in December, the ASD decided to take over Neely’s Bend Middle School in Nashville. The school had been on the priority list, after all, and it was declared the victor in a school vs. school battle against Madison Middle.

I reported earlier in the week about the impressive gains at Neely’s Bend. In fact, the state’s TVAAS website shows Neely’s Bend receiving a 5 overall in its growth score — the state’s highest number.

I wondered where Neely’s Bend might fall in comparison to Rubinstein’s analysis of the ASD schools that had been under management for the past three years. Turns out, Neely’s Bend’s proficient/advanced composite for reading and is 54.4.

Yes, you read that right. Neely’s Bend’s score is 5.8 points higher than the best performing school that’s been under ASD control the longest.

Neely’s Bend is being taken over and converted to a charter school and yet the school posted significant gains (above district average), has a TVAAS overall score of 5, and has a higher percentage of students at the proficient/advanced level than the BEST schools under ASD management.

For more on education politics and policy in Tennessee, follow @TNEdReport



The Worst Teachers?

“There is a decently large percentage of teachers who are saying that they feel evaluation isn’t fair,” he (state data guru Nate Schwartz) said. “That’s something we need to think about in the process we use to evaluate teachers … and what we can do to make clear to teachers how this process works so they feel more secure about it.”

This from a story about the recently released 2015 Educator Survey regarding teacher attitudes in Tennessee.

One reason teachers might feel the evaluation is unfair is the continued push to align observation scores with TVAAS (Tennessee Value-Added Assessment System) data – data that purportedly captures student growth and thereby represents an indicator of teacher performance.

From WPLN:

Classroom observation scores calculated by principals should roughly line up with how a teacher’s students do on standardized tests. That’s what state education officials believe. But the numbers on the state’s five point scale don’t match up well.

“The gap between observation and individual growth largely exists because we see so few evaluators giving 1s or 2s on observation,” the report states.

“The goal is not perfect alignment,” Department of Education assistant commissioner Paul Fleming says, acknowledging that a teacher could be doing many of the right things at the front of the class and still not get the test results to show for it. But the two figures should be close.

In order to be better at aligning observation scores with TVAAS scores, principals could start by assigning lower scores to sixth and seventh grade teachers. At least, that’s what the findings of a study by Jessica Holloway-Libell published in June in the Teachers College Record suggest.

Holloway-Libell studied value-added scores assigned to individual schools in 10 Tennessee districts — Urban and suburban — and found:

In ELA in 2013, schools were, across the board, much more likely to receive positive value-added scores for ELA in fourth and eighth grades than in other grades (see Table 1). Simultaneously, districts struggled to yield positive value-added scores for their sixth and seventh grades in the same subject-areas. Fifth grade scores fell consistently in the middle range, while the third-grade scores varied across districts

Table 1. Percent of Schools that had Positive Value-Added Scores in English/language arts by Grade and District (2013) (Districts which had less than 25% of schools indicate positive growth are in bold)
District      Third      Fourth    Fifth     Sixth     Seventh      Eighth
Memphis      41%       43%        45%      19%        14%           76%
Nashville      NA        43%        28%      16%        15%           74%
Knox             72%       79%        47%      14%         7%            73%
Hamilton     38%      64%        48%      33%      29%            81%
Shelby           97%     76%         61%       6%        50%            69%
Sumner         77%     85%         42%       17%      33%            83%
Montgomery NA      71%         62%       0%        0%              71%
Rutherford     83%   92%         63%      15%     23%             85%
Williamson    NA      88%        58%      11%      33%           100%
Murfreesboro NA     90%        50%     30%     NA              NA

SOURCE: Teachers College Record, Date Published: June 08, 2015 ID Number: 17987, Date Accessed: 7/27/2015

In examining three-year averages, Holloway-Libell found:

The three-year composite scores were similar except even more schools received positive value-added scores for the fifth and eighth grades. In fact, in each of the nine districts that had a composite score for eighth grade, at least 86% of their schools received positive value-added scores at the eighth-grade level.

By contrast, results in math were consistently positive across grade level and district type:

In particular, the fourth and seventh grade-level scores were consistently higher than those of the third, fifth, sixth, and eighth grades, which illustrated much greater variation across districts. The three-year composite scores were similar. In fact, a majority of schools across the state received positive value-added scores in mathematics across all grade levels.

So, what does this mean?

Well, it could mean that Tennessee’s 6th and 7th grade ELA teachers are the worst in the state. Or, it could mean that math teachers in Tennessee are better teachers than ELA teachers. Or, it could mean that 8th grade ELA teachers are rock stars.

Alternatively, one might suspect that the results of Holloway-Libell’s analysis suggest both grade level and subject matter bias in TVAAS.

In short, TVAAS is an unreliable predictor of teacher performance. Or, teaching 6th and 7th grade students reading is really hard.

Holloway-Libell’s findings are consistent with those of Lockwood and McCaffrey (2007) published in the Journal of Educational Measurement:

The researchers tested various VAM models and found that teacher effect estimates changed significantly based on both what was being measured AND how it was measured.

That is, it’s totally consistent with VAM to have different estimates for math and ELA teachers, for example. Math questions are often asked in a different manner than ELA questions and the assessment is covering different subject matter.

So, TVAAS is like other VAM models in this respect. Which means, as Lockwood and McCaffrey suggest, “caution is needed when interpreting estimated teacher effects” when using VAM models (like TVAAS).

In other words: TVAAS is not a reliable predictor of teacher performance.

Which begs the question: Why is the Tennessee Department of Education attempting to force correlation between observed teacher behavior and a flawed, unreliable measure of teacher performance? More importantly, why is such an unreliable measure being used to evaluate (and in some districts, reward with salary increases) teachers?

Don’t Tennessee’s students and parents deserve a teacher evaluation system that actually reveals strong teaching and provides support for teachers who need improvement?

Aren’t Tennessee’s teachers deserving of meaningful evaluation based on sound evidence instead of a system that is consistent only in its unreliability?

The American Statistical Association has said value-added models generally are unreliable as predictors of teacher performance. Now, there’s Tennessee-specific evidence that suggests strongly that TVAAS is biased, unreliable, and not effective as a predictor of teacher performance.

Unless, that is, you believe that 6th and 7th grade ELA teachers are our state’s worst.

For more on education politics and policy in Tennessee, follow @TNEdReport