Apples and Oranges

Here’s what Director of Schools Dorsey Hopson had to say amid reports that schools in his Shelby County district showed low growth according to recently released state test data:

Hopson acknowledged concerns over how the state compares results from “two very different tests which clearly are apples and oranges,” but he added that the district won’t use that as an excuse.

“Notwithstanding those questions, it’s the system upon which we’re evaluated on and judged,” he said.

State officials stand by TVAAS. They say drops in proficiency rates resulting from a harder test have no impact on the ability of teachers, schools and districts to earn strong TVAAS scores, since all students are experiencing the same change.

That’s all well and good, except when the system upon which you are evaluated is seriously flawed, it seems there’s an obligation to speak out and fight back.

Two years ago, ahead of what should have been the first year of TNReady, I wrote about the challenges of creating valid TVAAS scores while transitioning to a new test. TNReady was not just a different test, it was (is) a different type of test than the previous TCAP test. For example, it included constructed response questions instead of simply multiple choice bubble-in questions.

Here’s what I wrote:

Here’s the problem: There is no statistically valid way to predict expected growth on a new test based on the historic results of TCAP. First, the new test has (supposedly) not been fully designed. Second, the test is in a different format. It’s both computer-based and it contains constructed-response questions. That is, students must write-out answers and/or demonstrate their work.

Since Tennessee has never had a test like this, it’s impossible to predict growth at all. Not even with 10% confidence. Not with any confidence. It is the textbook definition of comparing apples to oranges.

Here’s a statement from the academic article I cited to support this claim:

Here’s what Lockwood and McCaffrey (2007) had to say in the Journal of Educational Measurement:

We find that the variation in estimated effects resulting from the different mathematics achievement measures is large relative to variation resulting from choices about model specification, and that the variation within teachers across achievement measures is larger than the variation across teachers.
You get different value-added results depending on the type of test you use. That is, you can’t just say this is a new test but we’ll compare peer groups from the old test and see what happens. Plus, TNReady presents the added challenge of not having been fully administered last year, so you’re now looking at data from two years ago and extrapolating to this year’s results.
Of course, the company paid millions to crunch the TVAAS numbers says that this transition presents no problem at all. Here’s what their technical document has to say about the matter:
In 2015-16, Tennessee implemented new End-of-Course (EOC) assessments in math and English/language arts. Redesigned assessments in Math and English/language arts were also implemented in grades 3-8 during the 2016-17 school year. Changes in testing regimes occur at regular intervals within any state, and these changes need not disrupt the continuity and use of value-added reporting by educators and policymakers. Based on twenty years of experience with providing valueadded and growth reporting to Tennessee educators, EVAAS has developed several ways to accommodate changes in testing regimes.
Prior to any value-added analyses with new tests, EVAAS verifies that the test’s scaling properties are suitable for such reporting. In addition to the criteria listed above, EVAAS verifies that the new test is related to the old test to ensure that the comparison from one year to the next is statistically reliable. Perfect correlation is not required, but there should be a strong relationship between the new test and old test. For example, a new Algebra I exam should be correlated to previous math scores in grades seven and eight and to a lesser extent other grades and subjects such as English/language arts and science. Once suitability of any new assessment has been confirmed, it is possible to use both the historical testing data and the new testing data to avoid any breaks or delays in value-added reporting.
A couple of problems with this. First, there was NO complete administration of a new testing regime in 2015-16. It didn’t happen.
Second, EVAAS doesn’t get paid if there’s not a way to generate these “growth scores” so it is in their interest to find some justification for comparing the two very different tests.
Third, researchers who study value-added modeling are highly skeptical of the reliability of comparisons between different types of tests when it comes to generating value-added scores. I noted Lockwood and McCaffrey (2007) above. Here are some more:
John Papay (2011) did a similar study using three different reading tests, with similar results. He stated his conclusion as follows: [T]he correlations between teacher value-added estimates derived from three separate reading tests — the state test, SRI [Scholastic Reading Inventory], and SAT [Stanford Achievement Test] — range from 0.15 to 0.58 across a wide range of model specifications. Although these correlations are moderately high, these assessments produce substantially different answers about individual teacher performance and do not rank individual teachers consistently. Even using the same test but varying the timing of the baseline and outcome measure introduces a great deal of instability to teacher rankings.
Two points worth noting here: First, different tests yield different value-added scores. Second, even using the same test but varying the timing can create instability in growth measures.
Then, there’s data from the Measures of Effective Teaching (MET) Project, which included data from Memphis. In terms of reliability when using value-added among different types of tests, here’s what MET reported:
Once more, the MET study offered corroborating evidence. The correlation between value-added scores based on two different mathematics tests given to the same students the same year was only .38. For 2 different reading tests, the correlation was .22 (the MET Project, 2010, pp. 23, 25).
Despite the claims of EVAAS, the academic research raises significant concerns about extrapolating results from different types of tests. In short, when you move to a different test, you get different value-added results. As I noted in 2015:

If you measure different skills, you get different results. That decreases (or eliminates) the reliability of those results. TNReady is measuring different skills in a different format than TCAP. It’s BOTH a different type of test AND a test on different standards. Any value-added comparison between the two tests is statistically suspect, at best. In the first year, such a comparison is invalid and unreliable. As more years of data become available, it may be possible to make some correlation between past TCAP results and TNReady scores.

Or, if the state is determined to use growth scores (and wants to use them with accuracy), they will wait several years and build completely new growth models based on TNReady alone. At least three years of data would be needed in order to build such a model.

Dorsey Hopson and other Directors of Schools should be pushing back aggressively. Educators should be outraged. After all, this unreliable data will be used as a portion of their teacher evaluations this year. Schools are being rated on a 1-5 scale based on a growth model grounded in suspect methods.

How much is this apple like last year’s orange? How much will this apple ever be like last year’s orange?

If we’re determined to use value-added modeling to measure school-wide growth or district performance, we should at least be determined to do it in a way that ensures valid, reliable results.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Charters on the March?

Charter schools have not gained much ground outside of Memphis and Nashville, but that doesn’t mean potential charter operators and the Tennessee Charter School Center aren’t trying. Just a few years ago, there was quite a fight over a proposed charter school in Cheatham County. That application was ultimately denied.

Yesterday, the Clarksville Rotary Club hosted charter school lobbyist Emily Lilley to talk about charter schools and the process of creating one.

Of course, Clarksville residents might not be too eager to “think outside the box” as their current public schools appear to be performing quite well.

Where else are charter proponents planning to expand?

For more on education politics and policy in Tennessee, follow @TNEdReport


 

The Data Wars: Herb Strikes Back

Yes, the Data Wars continue. Metro Nashville Public Schools (MNPS) gained new hope recently when 33 members of Nashville’s Metro Council penned a letter supporting resistance to the Achievement School District’s request for student data.

Now, Tennessee’s Attorney General has weighed-in and says the alliance of MNPS and Shelby County must comply with the ASD’s request. What happens if they don’t? Nate Rau notes in the Tennessean:

McQueen’s warning leaves open the possibility the state would dock education dollars from Metro and Shelby schools if they continue to deny her request.

It wouldn’t be the first time for Nashville, as the Haslam administration withheld $3.4 million in state funds in 2012 after the school board refused to approve controversial Great Hearts charter school.

Withholding state BEP funds is a favorite “ultimate weapon,” used in the Great Hearts controversy and also threatened during the TNReady debacle in year one of that test that wasn’t.

During the debate that ultimately saw Nashville schools lose funds in a BEP penalty, Commissioner Kevin Huffman and the Department of Education had an ally in then-Nashville Mayor Karl Dean. Joey Garrison reported in the (now defunct) City Paper at the time:

By this point, Huffman had already facilitated a July 26 meeting to discuss Great Hearts’ next move, a gathering that took place just hours before Great Hearts’ revised application would go before the Metro board for second consideration. The meeting site: the office of Mayor Karl Dean, also a Great Hearts backer. In attendance, among others, were Huffman, Dean, Barbic, Deputy Mayor Greg Hinote, Great Hearts officials Dan Scoggin and Peter Bezanson, and Bill DeLoache, a wealthy Nashville investor and one of the state’s leading charter school proponents.

As Rau points out, the current controversy stems from a newly-passed state law giving charter schools the opportunity to request student data from district schools. It seems, however, that there is some dispute over the intent of that law. Rau explains:

Slatery’s opinion also said that the student data may be used for the ASD to promote its schools to prospective students. State Rep. John Forgety, who chairs a House education committee and supported the legislation, told The Tennessean the intent was not to create a law that allowed districts to market to each other’s students.

So it seems the legislature may need to revisit the issue to clear things up.

Also unclear: Where do the current candidates for Governor stand on protecting student data vs. providing marketing information to competing districts and schools?

Stay tuned for more. Will the Shelby-MNPS alliance continue their resistance? Will Commissioner McQueen unleash the power of BEP fund withholding? Will this issue end up in court?

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Goodbye, Grace Tatter

Chalkbeat’s Grace Tatter wrote her last piece for the online education news site this week. She’s moving on to Harvard for graduate school.

I could go back and count the number of blog posts of mine that included the words “Grace Tatter reported…” or some variation of that phrase, but that would take too long.

Tatter did an incredible job providing comprehensive coverage of education policy in Tennessee. She was there and wrote about numerous key events.

Her stories were clear, concise, and accessible.

Often, a paragraph from a Grace Tatter story would inspire me to dig a little deeper, find out a little more, and write a post of my own.

So, I say goodbye to Grace Tatter. Your work will be missed. I know the newer faces at Chalkbeat will continue doing sound work, and I look forward to it.

And, one last time, I’ll cite something Grace said to make a point:

Even when stories don’t seem to be about money, they usually are. How much money is being spent on testing, teacher salaries, school discipline reform? How much should be available for wraparound services? Why do some schools have more money than others? Is there enough to go around? Tennessee leaders have steadily upped public education spending, but the state still invests less than most other states, and the disparities among districts are gaping. That’s why more than a handful of school districts are battling with the state in court. Conversations about money are inextricable from conversations about improving schools

Once again, in typical Tatter fashion, she nails it. We can’t have the conversation about improving our schools without the conversation about investing in our schools. Money matters.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Metro Council Members Back MNPS in Data Wars

I’ve written before about the escalating Data Wars between the state’s Achievement School District (ASD) and the two largest school districts – Shelby County and MNPS.

Now, Nashville’s Metro Council is weighing-in, at least in the form of a letter signed by 33 Council Members to MNPS Board Chair Anna Shepard.

The Tennessean notes:

The 33 Nashville Metro Council members signed a letter, dated Tuesday, that commends the district for “taking steps to protect the personal information of students and families.”

“We understand the state has taken a confrontational position on this issue, seeking to compel Nashville and Memphis schools to continue sharing personal information in opposition to federal and without state statute supporting their position,” the letter reads. “However, as elected representatives of the same constituents whose privacy rights are being violated, we encourage you to continue to advocate for our families by the just and proper means that are available to you.”

As I’ve noted before, Commissioner McQueen has asked for an Attorney General’s opinion on the various interpretations of a new state law that some suggest mandates the data-sharing the ASD seeks.

What happens if MNPS doesn’t share the data? There’s always the possibility the state will punish them by withholding some BEP funds.

That happened back in 2012 over the Great Hearts controversy. Those who follow MNPS closely will recall that then-Mayor Karl Dean was a prime backer of Great Hearts, which put him at odds with the elected School Board at that time.

As Joey Garrison, writing for the City Paper at the time, reported:

Emails show DeLoache, long known as an unofficial education adviser to Dean, served as a resource for Huffman, as well. After the Metro board denied Great Hearts in May, DeLoache told Huffman he hoped its rejection might “provide an opportunity to highlight to the Governor” the need to push for a statewide charter school authorizer during the 2013 legislative session. (A statewide charter authorizer would effectively supersede and therefore negate authority of local charter authorizers such as Metro.)

That’s Bill DeLoache, the wealthy Nashville investor and charter proponent who has spent heavily in the past to help elect pro-charter candidates to the MNPS School Board.

Will MNPS and Shelby County Schools face fines if they continue on their current path of protecting student data from the ASD? Will more Metro leaders stand up and support the School Board?

The Data Wars continue.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Next?

Tennessee’s Achievement School District (ASD) is again looking for a Superintendent as it was announced today that current Superintendent Malika Anderson is on her way out.

Chalkbeat has the story:

Malika Anderson, who has sought to steer Tennessee’s school turnaround district to stability after its contentious early work in Memphis and Nashville, is stepping down as its second superintendent at the end of this month.

Education Commissioner Candice McQueen had this to say about the move:

“This transition in no way disrupts our work,” McQueen said in a press release. “We are taking what we have learned about school improvement over the past five years and using that knowledge to maximize students’ success by putting in place a strong set of evidence-based options that will drive improvements in students performance.”

Anderson is the second Superintendent in the ASD’s short history, replacing Chris Barbic. Barbic noted on his departure:

In his email early Friday, Barbic offered a dim prognosis on that pioneering approach. “As a charter school founder, I did my fair share of chest pounding over great results,” he wrote. “I’ve learned that getting these same results in a zoned neighborhood school environment is much harder.”

The ASD has been plagued with both lackluster results and challenges connecting with the communities it serves during its brief but tumultuous existence.

According to the Department of Education’s release, a search will begin immediately for Anderson’s replacement. In the meantime, Deputy Commissioner of Education Kathleen Airhart will serve as Interim Superintendent. Before coming to the Department of Education, Airhart was the Director of Schools in Putnam County.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

MNPS Statement on Trump’s DACA Action

As U.S. Attorney General Jeff Sessions rolled out the Trump Administration’s plan to rescind DACA (Deferred Action for Childhood Arrivals), MNPS issued a statement calling the President’s decision “unacceptable.”

Here’s the full statement:

Former President Lyndon B. Johnson once shared, “If we succeed, it will not be because of what we have, but it will be because of what we are; not because of what we own, but, rather because of what we believe.”  In light of President Trump’s announced intention to end Deferred Action of Childhood Arrivals (DACA), Metro Nashville Public Schools wants to reassert our belief that all school-aged students should have access to an excellent education, and thus access to enhanced opportunities, without regard to their immigration status or the immigration status of their parents.

Students affected by ending DACA include high school students who are presently participating in the program and younger students (age 10-14) who will be eligible upon turning 15. Moreover, and perhaps more tragic, it exposes parents of United States citizens to deportation even though the parent arrived in this country as a child and the United States may be the only home he/she has known.  In effect, their children are second generation Americans and the living embodiment of the American dream. Nevertheless, the rescission of DACA will either require these young U.S. citizens to leave the country or be separated from their parents despite their parents’ longstanding residency and contribution to our community.

The intended rescission of DACA denies our schools and communities many ambitious, intelligent, and highly-motivated students, parents, teachers and staff and will result in fear and uncertainty for many of the families and students we serve. Plainly stated, the result of the President’s announced ending to DACA is unacceptable. We call on Congress to enact the Dream Act or otherwise codify DACA with legislation immediately.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Herb Backs Down

Earlier this year, I featured an excerpt from a piece written by Mike Stein about Tennessee’s Attorney General, Herb Slatery, and his support for ending the DACA — Deferred Action for Childhood Arrivals – program.

Stein notes that while Slatery joined with Attorneys General in several other states in sending a letter to U.S. Attorney General Jeff Sessions calling for an end to DACA (and threatening a lawsuit), the conservative CATO Institute actually supports maintaining DACA for its economic benefits.

Now that some reports suggest President Trump may be taking action to end DACA, let’s look at who Herb Slatery would have deported.

Chalkbeat had this report of a Nashville student-turned-educator who is also a beneficiary of the DACA program:

Ruiz knows what it’s like to live with uncertainty about his future.

His mother brought him to the United States to give him a better shot at graduating from high school and going to college, which she hadn’t been able to do in Mexico.

He attended public schools in Nashville, where he mastered English by the third grade.

When DACA was announced during his freshman year at Trevecca, Ruiz applied on the very first day. “DACA was an avenue for me to work hard and do what I wanted with that,” he said. “It made me feel in control and empowered.”

After graduating with a degree in history, Ruiz applied to Teach For America and was assigned to an elementary school in Denver. Realizing that his passion is working with high school students, he moved this year to STRIVE Prep Excel, a charter high school where he teaches Spanish.

For background, here’s how Chalkbeat describes DACA:

The policy gives protections, but not citizenship, for two years at a time to undocumented immigrants who came here as children.

Carlos Ruiz was brought to Nashville at age 6. He didn’t ask to come here. He didn’t deliberately evade the nation’s laws. He attended public schools in Nashville. He graduated from a college in Nashville. He decided to become a teacher.

Slatery sent a letter TODAY to Tennessee’s U.S. Senators announcing he’s pulling Tennessee out of litigation over DACA. Specifically, Slatery notes:

There is a human element to this, however, that is not lost on me and should not be ignored. Many of the DACA recipients, some of whose records I reviewed, have outstanding accomplishments and laudable ambitions, which if achieved, will be of great benef,rt and service to our country. They have an appreciation for the opportunities afforded them by our country.

The sad reality is that our Congress hasn’t taken a serious look at immigration reform that would address situations like Ruiz’s. Until they do, DACA provides protection for the children of immigrants. Children like Carlos Ruiz who has decided to take the opportunity he was given and serve others.

Slatery’s letter calls for legislative solution – seemingly in direct opposition to the Trump Administration’s position.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

*An earlier version of this story did not include details of Slatery’s letter released today.

Trade Offer

I reported last week on the Data Wars brewing between the state’s two largest school districts and the Tennessee Department of Education.

Now, as both Nashville and Memphis dig in, MNPS is offering a trade of sorts.

Chalkbeat reports on a letter sent by MNPS Board Chair Anna Shepard to Education Commissioner Candice McQueen.

In her letter, Shepard proposes cooperation between the state’s Achievement School District (ASD) and MNPS based on several conditions.

Specifically:

I would personally be willing to consider a coordinated initiative under which MNPS, using its existing communications infrastructure, would inform families about ASD choice options — if they choose to “opt in” to such communications. I cannot speak for my board colleagues until such time as we have had the opportunity to deliberate on this concept.

Shepard’s conditions:

  1. A moratorium on ASD expansion
  2. State subsidies for schools that lose students to the ASD
  3. The State engage in discussions around a new “fiscal impact” component of the BEP to address the impact charter schools have on local school districts

Regarding that fiscal impact, an audit of MNPS published in 2015 noted this:

“The key question for determining fiscal impacts is whether enrollment reductions allow a district to achieve expenditure reductions commensurate with revenue reductions. Fixed costs are incurred regardless of whether students attend traditional or charter schools. The problem is that some fixed costs, such as building maintenance, computer network infrastructure, and health services do not vary based on enrollment. Therefore, teachers and their salaries are a key cost driver tied to student enrollment … However, it is not always possible to reduce teacher costs proportionate to losses in revenue. For these costs to be reduced significantly, the school would need to close altogether.”

As for the ASD moratorium, it seems that the turnaround district continues to produce underwhelming results. Combine this with a track record of poor communication and you begin to understand why districts aren’t eager for the ASD to open more schools in their backyards.

For her part, Commissioner McQueen is seeking an Attorney General’s opinion on the MNPS and Shelby County interpretation of the data-sharing law passed in the 2017 legislative session.

It seems unlikely that McQueen would agree to the conditions set forth by Shepard. It seems possible both MNPS and Shelby County will face the threat of fines should they continue resisting.

Stay tuned as the Data Wars heat up.

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Holden: Trust Teachers

Former teacher and current education blogger Mary Holden offers her thoughts on how to address the teacher shortage. Of course, even if there wasn’t a teacher shortage, this is the right way to treat teachers. Here’s some of what she has to say:

Trusting teachers to do their job – BECAUSE THEY ARE TRAINED PROFESSIONALS – should be commonplace practice in every school district. But it’s not.

An article from last year in The Atlantic discussed what happened when some Finnish teachers taught here in the U.S. Guess what they noticed? The lack of autonomy. And that’s a very bad thing: “According to a National Center for Education Statistics (NCES) report, teacher autonomy is positively associated with teachers’ job satisfaction and retention. And while most U.S. public-school teachers report a moderate amount of control in the classroom, many say they have little autonomy.”

Let me repeat that in a different way: We have a teacher shortage. Want to retain teachers and attract new ones? Then trust them to do their jobs. Ask them what they need, and then give them the support they need. (Oh, and paying them more would help, too!)

Holden also offers this suggestion:

I wish more districts would recognize what teachers have been saying for years – stop focusing on the data and the test scores and all the punitive measures that have been in place since the dawn of the accountability movement, and instead, focus on what matters: People. Relationships. Community. Developing the joy of learning. And trust our teachers to teach the subjects for which they are trained to teach.

This (and the rest of her article) offer sound advice on how to support and nurture teachers. I hear people say all the time that decisions in education should be made based on what’s good for kids instead of what’s good for adults — as if the two are mutually exclusive. Guess what? Supported teachers who are given autonomy are happy teachers. Which means they are better teachers. Which is GREAT for kids.

For more on education politics and policy in Tennessee, follow @TNEdReport