Ezra Howard offers some insight into the ASD matching process in Memphis

Personal Experience

When the Achievement School District (ASD) announced that it would restructure its Achievement Advisory Council (AAC) and rebrand it as Neighborhood Advisory Council (NAC), I was cautiously optimistic. Last fall, I sat on the Northeast Region of the Achievement Advisory Council (NE AAC) and was thoroughly disappointed with my experience. Due to an automatic match between Green Dot and Raleigh-Egypt High School, the NE AAC dealt solely with engaging the community. This ordeal was well covered by the media, specifically the decision for an automatic match, the heated community meeting, and Green Dot’s decision to pull out of the matching process. My frustration didn’t stem from fellow AAC members; they were wonderful and I have nothing but respect for them. It also didn’t come from speaking with community members, teachers, and families at schools considered for takeover; they were frustrated but needed to be heard. Rather, my frustration stemmed from receiving limited support when working with the Achievement School District and the matching charter management organizations (CMOs) as well as the lack of transparency throughout the entire matching process.

My biggest fear for the AAC process was that, as applied to REHS, it was all lip service for the veneer of community engagement. So when I heard that the AAC would make some changes and become the NAC, I was cautiously optimistic. I recognized there were new strengths in the process, like the use of scored rubrics and ensuring parents were on the council. However, there were weaknesses, like limiting the number of members in each NAC despite the size of the school or the number of grades taught. Furthermore, the ASD pledged to honor the NAC’s recommendations. Malika Anderson, the incoming Superintendent of the ASD, echoed these sentiments with a promise to emphasize community engagement. I thought this year would be different.

Continued Concern

I didn’t sign up for the NAC, however, I moved to Lisbon, Portugal with my wife in order for us to both pursue PhDs. Despite the distance, I closely followed the events from afar. I was immediately concerned when I read about the comments made by Latoya Robinson, an NAC member, in the Commercial Appeal.

Latoya Robinson said she served on one of the neighborhood advisory councils that the ASD employed to rate charter operators who applied to take over the struggling schools. Robinson said the way their input was calculated allowed for Kirby Middle to be taken over against the council’s wishes.

“We did not put down information saying that we wanted Kirby to be taken over by Green Dot,” she said.

So I filled out an Open Records Request with the ASD for the individual rubrics and the scoring sheets used by the ASD. The results were not particularly encouraging.

The Matching Process

A little background on this year’s matching process: Green Dot was matched with Hillcrest High School and Kirby Middle school. Green Dot has been active in ASD schools for less than two years. They operated Fairley High School last year (Composite TVAAS of 2) and Wooddale Middle School this year (no TVAAS). Scholar Academies began with Florida-Kansas Elementary this year (no TVAAS). All of the operators considered this year were CMOs. Achievement Schools, the ASD’s direct-run schools, were not considered for matching.

[Note: I was paying attention specifically to the quantitative data. However, I suggest everyone read the individual rubrics and pay close attention to the notes. There are some very interesting facts. Names of the NAC members were redacted, and probably for good reason.]

A Problematic Process: A Low Bar, Grade Inflation, and Erasures

When reviewing the NAC rubrics and scoring sheets, I was immediately surprised to find that the cut score on the rubric was 50%. A passing mark of 50% seems extraordinarily low and, honestly, reminds me of the long abandoned grade scale used by the ASD where 46 was considered passing. Second, I noticed that a couple CMOs were barely passing with scores in the low to mid 50s. Third, I noticed quite a few scores were erased from consideration due to “insufficient evidence bordering on opinion.”

Let’s take each point in turn. It seems very plain that the bar set for CMOs is really low in the matching process. To make matters worse, passing the 50% mark is almost guaranteed using the scoring rubric set by the ASD. The ASD used a four point Likert scale to measure if the CMO was meeting expectations. Each level of measurement was given a score from 1 to 4 (except Demonstration of Community Outreach, which receives a score of 3, 6, 9, 12); the scored are then added up and divided by the total to provide an average percentage. First and foremost, the 1 to 4 scale is problematic because even if a CMO fails to meet the standard of any measure by a NAC member, they still score 25%. These free points set the floor high enough that it tells a CMO that they only have to strive for 25 percentage points in order to pass. In theory, if every NAC member thought the CMO “partially met the standard” then the CMO would be matched with a school. With most scores hovering in the 50s, that’s exactly what happened.

The floor is so high and the ceiling is so low, it makes it very difficult for a CMO to not match with a school. The scoring mechanism for the NAC rubrics is blatantly biased in favor of matches and takeovers. Furthermore, the glaringly low expectations for CMOs in this process stands completely against the rhetoric of high expectations espoused by the ASD, most recently by Chris Barbic himself. It’s hypocritical to the standards held to the schools slated for takeover, which requires a TVAAS 4 or 5 from schools on the priority list. Take Raleigh-Egypt Middle School, which scored a TVAAS of 3 this year with a TVAAS of 5 last year, which is now in the hands of Scholar Academies, which partially met expectations with a score of 54.17%. That’s absurd, but it doesn’t end there.

The sheer amount of redactions is shocking. The most appalling is Green Dot’s rubric for Kirby Middle, where several individual scores were erased and at least two whole rubrics were taken out of consideration. However, if you look at the rubrics yourself, you see that this NAC was very thorough, their reasoning is sound, and they substantiated their arguments with evidence. Take for example this apparently redacted assessment for Green Dot partially meeting the standard for community engagement:

The application demonstrated plans for parent engagement but the operator presentation of how parents are involved in the school was not convincing. When asked about engaging parents at the level required to made (sic) significant cultural changes, the operator was not able to give a strong plan of action.

The NAC member then provides a long list of concerns about community engagement with references to documentation and examples of personal experience. And would it be a surprise that the redacted scores appear to be 1s and 2s rather than 3s and 4s? Consider this assessment of “meeting, with reservations” with a rationale of “Operator understands necessary interventions for the student population. Green Dot has shown academic success at Fairley” with the strengths being “As stated they are the leaders in the school turnaround. They have demonstrated success rates in schools not only in Memphis but other cities.” With Fairley’s composite TVAAS of 2 last year, I disagree.

A Second Look

So there are two major issues with the NAC rubric scores. First, there is the low bar set for CMOs. Second, the redactions on the NAC scores appear inconsistent and biased toward high scores, ensuring that a CMO matches. So I decided to recalculate the rubric scores. I first make “not meeting expectations quantifiably unacceptable by recalculating the scores without the free points while keeping the passing score at the ASD’s 50% mark. Second, I analyzed the final NAC averages using the redacted scores. Finally, combine the former to methods to see where the NAC scores stand. All data came directly from the individual rubrics and scoring sheets provided by the ASD.

In order to the account for the roughly free 25 percentage embedded in the measurement tool of the NAC rubrics, I simply changed the scale from 1 to 4 to 0 to 3. In essence, CMOs are awarded no points for “not meeting expectations.” If CMOs surpass the 50% mark, simply put, this would put the aggregate score must be somewhere between “meeting expectations with reservations” and “partially meets the standard.” Even with the ASD’s redacted score, the results are not promising. Caldwell-Gutherie is the only school matched with 80.79 percent. Kirby has 48.98%, Hillcrest has 34.52%, Raleigh-Egypt Middle has 38.33 percent, 29.39%. It’s apparent that a buffer of about 25 percentage point drastically changes the outcome.

When the redacted scores are accounted for, there is a dramatic change in the average rubric score. Only Caldwell-Guthrie with 83.90% and Raleigh-Egypt Middle School with 53.75% would be considered for the matching process. Kirby with 47.22%, Hillcrest with 49.17%, and Sheffield with 45.25% would not be considered matches. Without a doubt, the redactions changed the results.

When you take away the 25 percentage point buffer and use all data available, there are some very interesting findings. First and foremost, only one school passes the low 50% mark: Caldwell-Guthrie at 78.53%. Every other school is below 50%. Kirby is 29.63%, Hillcrest is 32.22%, Raleigh-Egypt is 38.33%, and Sheffield is 27.00%.

It seems apparent that the assessment tool for the NAC rubrics are not only biased, but highly massaged. I believe these three additional forms of analysis illustrates how the NAC truly perceived the CMO and potential matches, and that view is quantifiably poor. It also points to the importance and impact of how a test is scored.

There is another interesting observation that I want to put on the table. In the case of each of the five schools scoring below 50% in my assessment, there are two individuals in each group that score the CMO significantly higher than their peers. Sometimes the scores are two or three times higher than the others and there are two very impressive perfect scores for Caldwell-Guthrie. I may be wrong, but if I were a betting man, I would bet the farm that the two highest scores in each NAC are the two “community members.” The difference these community members make ranges anywhere from 7 to 23 percentage points. If I am correct, then the community members’ assessment of the CMOs may not be representative of the parent members on the NAC.

I went through some of the names of the NAC members and some of the community members also sat on the previous AAC (some of whom I worked with and are those for whom I still have great respect). Several individuals on the NAC appear to be members of the contentious Memphis Lift. Memphis Lift, which is funded by Democrats for Education Reform, headed by Chris Barbic’s wife. Apparently, as one NAC member noted, Lift assisted some CMOs in community organizing activities as well. Despite the community member’s background, the data suggests these outliers, which appear unrepresentative of majority of the NAC members, were highly influential in the outcome of the NAC rubrics.

Conclusion

Whichever way you view it, the NAC results are very problematic. The entire process seems to favor CMO matching from the outset. With low cut scores, a buffer of 25 percentage points, numerous redactions, and influential members that may not represent the consensus, CMO matching seems to be a foregone conclusion from the beginning. Simply looking at the data, the NAC and the matching process calls into question authentic and meaningful community engagement. The rubric data undermines everything the NAC proclaimed to be and refutes the constant message of high standards, accountability, data-centric approaches, student-centered mindsets, transparency, and community engagement espoused both by the current superintendent, Chris Barbic, and the incoming leader, Malika Anderson. At best, it’s disingenuous; at worse, its strategic community disenfranchisement.

Quite frankly, the NAC appears to be the same thing as the AAC, lip service. Now we have the data to prove it.

The Need for Action, The Need for Change

When I sat on the AAC, Margo Roen, then Deputy Chief Portfolio Officer, said she was surprised that I didn’t write a blog post about my AAC experience. I said, half-jokingly, that I didn’t want to be mean, but now I’m angry. Honestly, anyone looking at this data and reading these rubrics should be angry too. The ASD seems to be speaking out of both sides of its mouth. It sets the bar pretty high for priority schools trying to avoid a takeover, but sets the bar laughably low for matching operators. It claims to care about community engagement, but then puts extreme limits on the community voice and agency. It claims urgency for action, but asks for patience when it comes to turnaround.

I wanted to believe in the education reform espoused by the Achievement School District. I want kids to shoot from the bottom 5% to the top 25%. I do not take joy in the fact that academic institutions view the ASD’s effect as “statistically insignificant.” I am not happy that NAC rubrics illustrate low opinions of charters and a scoring mechanism that undermine the matching process. But something quickly needs to change, because Tennessee families continue to be sold a bill of goods by the ASD and out-of-town charters.

I have a simple suggestion for our legislature, because it seems very apparent that it will take a bill to entice honest operation by the ASD. Raumesh Akbari, as most educators know, passed a widely praised bill that took schools with a TVAAS composite of a 4 or 5 off the table for turnaround. There should be a bill that reflects the same for the expansion of the ASD. For any operator wishing to expand in the ASD, an aggregate TVAAS 4 or 5 must be necessary for all schools. Allowances could be made for new operators. However, for those already operating in the ASD, growth must be required. Second, an NAC-type structure is necessary, but must be heavily restructured and regulated outside of the ASD. There’s nothing wrong with the shape of the NAC, but the measurement tool makes very little sense and shows clear bias.

The worst part about it is that the ASD could have taken this exact approach to takeovers this year. A great example is the direct-run Achievement Schools. After some inconsistent results and a few rocky starts, all five of the Achievement Schools scored a composite 4 or a 5 last year. However, no schools were considered for the Achievement Schools in the matching process. Barbic is right to celebrate that fact in his opinion piece on Saturday; but his use of Whitney to prop up the cadre of charters considered for this year’s matching process is inaccurate and misleading. Green Dot’s Fairley High School has a TVAAS of 2 and Scholar Academies has no real track record in Tennessee. By all accounts, that should be unacceptable. Ultimately, it makes me think the expansion of the ASD is less about children and learning and more about adults and business plans.

NAC Rubric – no redacts, unweighted

For more on education politics and policy in Tennessee, follow @TNEdReport