Outlier

Statisticians define an outlier as an observation point that is distant from other observations in a statistical analysis. Often, this occurs by chance. Additional modeling or deeper analysis (including more data, for example, or a longer range of data) can often correct for this. Outliers that are not the result of measurement error are often excluded from analysis about a data set.

Today, the 2017 results from the National Assessment of Educational Progress (NAEP) were released. This release made me think of a particular outlier.

Back in 2013, Tennessee demonstrated what some heralded as an incredible achievement on the NAEP. In fact, a press release from Governor Haslam at the time noted:

Gov. Bill Haslam today announced that Tennessee had the largest academic growth on the 2013 National Assessment of Educational Progress (NAEP) of any state, making Tennessee the fastest improving state in the nation. (emphasis added)

Those words — “fastest improving state in the nation” — have been uttered by Haslam and many political leaders in our state for years now. Often, this 2013 “success” is used as justification for “keeping our foot on the gas” and continuing an aggressive agenda of test-based accountability and teacher evaluation based on methods lacking validity.

Here’s what I wrote back in 2013 when these results were released:

Yes, Tennessee should celebrate its growth.  But policymakers should use caution when seeing the results from the last 2 years as a validation of any particular policy.  Long-term trends indicate that big gains are usually followed by steady maintenance. And, even with the improvement, Tennessee has a long way to go to be competitive with our peers. Additionally, education leaders should be concerned about the troubling widening of the rich/poor achievement gap  – an outcome at odds with stated policy goals and the fundamental principle of equal opportunity.

Two years later, when the 2015 results were released, I noted:

This year’s scores, in which Tennessee remained steady relative to the 2013 scores suggest, if anything, that the 2013 jump was likely an outlier. Had the 2013 gains been followed by gains in 2015 and again in 2017, more could be suggested. And frankly, it is my hope that we see gains (especially in reading) in 2017. But, it’s problematic to suggest that any specific reform or set of reforms caused the one-time jump we saw in 2013. Saying we are the fastest improving state in the nation over the last 4 years when we only saw a jump in 2013 is like saying we started the first quarter of a football game way behind, scored a bunch in the second quarter, (so we’re not as far behind), and then scored the same number of points in the third quarter. The result is we’re still behind and still have a long way to go.

Fast forward to today. The leveling off I suggested was likely back in 2013 has happened. In fact, take a look at this chart put out by the Tennessee Department of Education:

First, notice that between 2009 and 2011, Tennessee saw drops in 4th and 8th grade reading and 8th grade math. That helps explain the “big gains” seen in 2013. Next, note that in 4th and 8th grade reading and 4th grade math, our 2017 scores are lower than the 2013 scores. There’s that leveling off I suggested was likely. Finally, note that in 4th and 8th grade reading, the 2017 scores are very close to the 2009 scores. So much for “fastest-improving.”

Tennessee is four points below the national average in both 4th and 8th grade math. When it comes to reading, we are 3 points behind the national average in 4th grade and 5 points behind in 8th grade.

All of this to say: You can’t say you’re the fastest-improving state on NAEP based on one testing cycle. You also shouldn’t make long-term policy decisions based on seemingly fabulous results in one testing cycle. Since 2013, Tennessee has doubled down on reforms with what now appears to be little positive result. Instead, as Rep. Jeremy Faison said recently, our policies are “driving teachers crazy.”

Oh, and that new TNReady test has so far not been very ready.

But what about the good policy coming from this? You know, like Governor Haslam’s plan to make Tennessee the “fastest-improving state in teacher pay?”

About that:

Average teacher salaries in the United States improved by about 4% from the Haslam Promise until this year. Average teacher salaries in Tennessee improved by just under 2% over the same time period. So, since Bill Haslam promised teachers we’d be the fastest improving in teacher pay, we’ve actually been improving at a rate that’s half the national average. No, we’re not the slowest improving state in teacher pay, but we’re also not even improving at the average rate.

Surely, though, all this focus on education since the NAEP buzz has meant meaningful investment in schools, right? Well, no:

Tennessee earns a grade of F when it comes to funding effort compared to funding ability. The researchers looked at Gross State Product and Personal Income data in order to determine a state’s funding ability then looked at dollars spent per $1000 (in either GSP or Personal Income) to determine effort. Tennessee spends $29 on schools for every $1000 generated in Gross State Product. When it comes to Personal Income, Tennessee spends just $33 per $1000 of average personal income. That’s a rank of 42 in both.

Then, the report looks at wage competitiveness — how much teachers earn relative to similarly-educated professionals. I’ve written about this before, and Tennessee typically doesn’t do well in this regard.

Maybe we’ve taken a minute to get serious about investing in programs targeting struggling students? Also, no:

One possible solution would be to embed funding for school-level RTI2 specialists in the state’s funding formula for schools, the BEP. In fact, Rep. Joe Pitts offered legislation that would do just that last year. His plan would have added funding for three RTI2 specialists at each school for a total projected cost of $167 million. Commissioner McQueen was quick to shoot that idea down and came back this year with the funding proposal of $13 million, or one specialist per district. That’s only $154 million short of adopting a plan that would actually meet the needs of a program many suggest is an important way to improve educational outcomes for Tennessee students.

Maybe we are closing achievement gaps? Again, no.

Back in 2013, Tennessee students eligible for free/reduced lunch had an average NAEP reading score of 256 and scored 20 points below the non-eligible students. Now, that average score is 252 (four points worse) and 19 points below. For 4th grade, there’s a similar story, with free/reduced lunch eligible students scoring 25 points below their non-eligible peers this year. Four years ago, it was 26 points.

We’re not moving the needle. Our most vulnerable students continue to be left behind. Meanwhile, we hear nice words from top policymakers and see little actual result in terms of tangible improved investment in schools or any meaningful upgrade in teacher pay. Our testing system has yet to be proven.

Maybe now Tennessee policymakers will stop repeating the “fastest-improving” line and start doing the actual work of investing in and supporting our schools.

In any case, the next time you hear someone spout off that tired “fastest-improving” line, just yell back: OUTLIER!

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Bill Dunn Wrong

Yesterday, in his advocacy for HJR 493, legislation that would remove the Tennessee Constitution’s requirement that the General Assembly adequately fund schools, State Representative Bill Dunn suggested that increasing funding for schools across the state actually does not improve student outcomes. He cited the initial BEP investment, started in 1992 and said that from beginning to end, the program actually resulted in lower student achievement numbers.

This would be a great way to prove Dunn’s case that the General Assembly need not provide additional funds to schools in order to provide an adequate education.

It’s also not true.

Dunn cited ACT scores from the start of the BEP until 1998 and suggested they’d gone down slightly. What he failed to mention is that between 1995 and 1998, the number of students taking the ACT increased by 25%. That would seem to indicate that Rep. Kevin Dunlap was correct when he suggested that new BEP funds created new opportunities for students in rural districts. As the State of Tennessee noted in the 1998 State Report Card:

The ACT is one of three tests approved by the State Board of Education to fulfill the requirement in state law that all students take an exit exam to receive a full high school diploma. The total number of Tennessee graduates taking the ACT rose 25% during the first three years of this new requirement: from 32,628 in 1995 to 40,782 in 1998. Included among those tested were 14,284 who had not completed a college preparatory course of study. Even with these dramatic increases in the number and percentage of students tested, Tennessee’s students were able to narrow the gap between the state and national composite scores in 1998.

So, more students than ever were taking the ACT and by 1998, the state was turning around an initial decline in scores. That’s a different story than the one Bill Dunn told.

Another way to look at the data is to see what happened on the National Assessment of Educational Progress (NAEP) during the early BEP years. While reading scores from 1992 are not readily available, math scores are. Here’s the comparison:

4th Grade Math

1992         211

2000        220

8th Grade Math

1992        259

2000       263

These results show statistically significant improvements in math scores over the same time period the General Assembly was significantly improving investment in public schools. That is, what Bill Dunn said yesterday was just plain wrong.

Finally, it’s worth examining the ACT score differences among districts during the early BEP years. An examination of data beginning in 1991 (the year before BEP) and ending in 2001 (so as to provide 10 years of comparable data) indicates that the top scoring districts in the state on the ACT were also among the top spending districts. In fact, over those years, while not technically statistically significant, it can be said with 92% confidence that the difference in ACT scores among the highest- and lowest-performing districts is explained by per pupil expenditures. That is, the higher the spending, the more likely the district is to be among the state’s top performers on the ACT.

Additionally, during this same ten year time period, the gap between the highest and lowest scores among districts is clearly explained by the gap in per pupil expenditures among those districts. You spend more, you get better results. The impetus for all this spending was the new BEP formula that sent more money to all school systems. Those districts already at the top were most able to take advantage and boost ACT scores while those at the bottom saw an increase in the number of students taking the ACT, resulting in the statewide slight ACT decline Dunn references.

Investing in schools matters. Our state’s constitution requires the General Assembly to provide a system of free public schools, including providing adequate funding for those schools. Bill Dunn doesn’t think spending levels matter. The data suggests otherwise.

For more on education politics and policy in Tennessee, follow @TNEdReport

It’s Raining Money, But Not on Schools

Tom Humphrey reports on the most recent budget projections which predict a surplus of between $300-$400 million for the fiscal year ending June 30, 2016.

This money, combined with the $600 million surplus from the fiscal year which ended on June 30, 2015, means the state will have about $1 billion in unanticipated, uncommitted revenue.

On top of that, economists are projecting growth in the $400-$500 million range for the upcoming budget year.

As Humphrey notes, proposals are floating so spend the surplus on road projects or tax cuts or both.

What’s not being mentioned?

Schools.

Despite a pair of lawsuits contending the state’s school funding formula, the BEP, is inadequate, lawmakers and the Governor are not rushing to suggest significant new investments in Tennessee schools.

This in spite of the fact that after a one year bounce on NAEP results, our state is now holding flat. Maybe that’s because Senate Education Committee Chair Dolores Gresham is suggesting our state aspire toward “mediocrity” while holding a forum on disastrous (and expensive) school voucher schemes.

If the state invested half of the available surplus on the BEP formula, that would be a $500 million injection of funds to local schools. That would be a sure way to hold down local property tax increases while also beefing up the resources school systems have to provide an education. The revenue projections for the 2016-17 fiscal year indicate an investment of this magnitude is sustainable. And, by holding a portion of the surplus in reserve, the state could ensure against any unanticipated shortfalls.

All of that would still leave $200-$300 million available to spend on one-time costs like road projects.

Will Tennessee put its foot on the accelerator and invest in schools so our students have the resources they need to compete with the rest of the country? Will Bill Haslam use the surplus and projected new revenue to truly make Tennessee the fastest improving state in the nation in teacher pay?

The General Assembly will have answers to these questions starting in January.

For more on education politics and policy in Tennessee, follow @TNEdReport

Aspiring Toward Mediocrity

That’s what Senate Education Committee Chair Dolores Gresham had to say in terms of Tennessee’s education goals right after praising the state’s recent NAEP results.

Her remarks came at the beginning of a hearing her committee conducted on school choice earlier this week. They can be found just past the two minute mark in this video. 

What’s frustrating is that she then proceeded to spend hours conducting a hearing that was nothing short of a celebration of all the supposed benefits of school voucher schemes.

Here’s a quick summary of some key arguments against vouchers.

The bottom line is they are expensive, can be susceptible to fraud, reduce accountability, and most importantly do not improve student academic outcomes.

While Gresham continues to put forward school vouchers as a solution to at least get Tennessee to mediocrity, the state’s BEP Review Committee is busy telling legislators that all the funding woes of the past have been miraculously cleared up.

After hours of hearings, we are still no closer to a path to that mediocrity to which Gresham hopes our state can achieve. At least we now have a clear understanding of her expectations.

For more on education policy and politics in Tennessee, follow @TNEdReport

 

The NAEP Spin Room

Yesterday, I wrote about the very rosy interpretation of NAEP data being advanced by Tennessee leaders. Governor Haslam said:

“Today, we’re very excited to say that based on 2015 NAEP results, we’re still the fastest improving state in the nation since 2011. What this means is a new set of fourth- and eighth-graders proved that the gains that we made in 2013 were real.”

After analyzing the Tennessee results and putting them in context with national results (both of which essentially remained steady from 2013) , I noted:

It’s also worth noting that states that have adopted aggressive reforms and states that haven’t both remained flat. The general trend was “holding steady,” and it didn’t seem to matter whether your state was using a reform agenda (charters, vouchers, value-added teacher scores in teacher evaluations) or not.

Again, this makes it difficult to suggest that any one or even a package of educational practices drives change.

Then, I read the statement issued by SCORE (Statewide Collaborative on Reforming Education) Executive Director Jamie Woodson. Here’s what she had to say:

Since 2011, Tennessee has made record-setting gains, held them, and progressed in state rankings because of a multi-faceted strategy of high standards, great teaching, accountability, and common-sense adjustments based on the feedback of educators and citizens.

Note that she assigns causality based on these results. I wonder, then, what to make of the states that didn’t adopt the multi-faceted strategy she references? Last year, a number of states showed significant gains on NAEP. Some, like DC and Tennessee were reform-oriented states, others were not.

Additionally, in a post about the NAEP results two years ago, I noted:

Kentucky and Tennessee have posted gains over time on NAEP — in most categories, Kentucky started out tied or very slightly ahead of Tennessee and today, Kentucky remains ahead.  Kentucky posted some pretty big gains in the mid-90s and again from 2003-2009.  Since then, they’ve held fairly steady.  That’s an expected result, by the way — a big gain followed by steady maintenance of the new level.  For Tennessee, that won’t be enough, but celebrating the big gain is certainly warranted.  It’s also important to take care in assigning causality.

Note here that what I suggested then was an expected result (big gain, followed by holding steady) is exactly what happened in Tennessee this year. That’s good news — it means we’re not declining. But it also means we can’t really say that 2013 was something special.  As I noted last year, Kentucky had a series of big gains in the 1990s and then again in the early 2000s. It wasn’t just a big bump one time. So far, Tennessee has had one banner year (2013) and this year, returned to normal performance.

However, the narrative of “fastest-improving” keeps being repeated. In fact, Bethany Bowman of Professional Educators of Tennessee (PET) released a statement that said in part:
Tennessee students are still the fastest improving in the nation since 2011 according to the 2015 National Assessment of Educational Progress (NAEP), commonly known as the Nation’s Report Card. “This year’s results from National Assessment of Educational Progress (NAEP) show that Tennessee has maintained the positive gains that we achieved in 2013.

We had one year in which we made a big splash and then, as I noted in 2013:

As the data shows, Kentucky and Tennessee in many cases posted similar net gains over time, with Kentucky seeing big jumps in the mid-90s and again in the early part of the last decade.

That is to say, over a 20-year period, both states saw similar net gains. This year’s scores, in which Tennessee remained steady relative to the 2013 scores suggest, if anything, that the 2013 jump was likely an outlier. Had the 2013 gains been followed by gains in 2015 and again in 2017, more could be suggested. And frankly, it is my hope that we see gains (especially in reading) in 2017. But, it’s problematic to suggest that any specific reform or set of reforms caused the one-time jump we saw in 2013. Saying we are the fastest improving state in the nation over the last 4 years when we only saw a jump in 2013 is like saying we started the first quarter of a football game way behind, scored a bunch in the second quarter, (so we’re not as far behind), and then scored the same number of points in the third quarter. The result is we’re still behind and still have a long way to go.

So, yes, let’s celebrate that we made a big jump and held it steady. But, let’s also put those results in context and focus on how we can move forward instead of using these results to advance our favorite plays. For example, I’m not a huge fan of vouchers, but NAEP data doesn’t really help me make the case for or against. Likewise, states with and without strong collective bargaining posted gains in 2013 and held steady in 2015 — that is, the presence or absence of bargaining has no impact on NAEP scores.

NAEP can be an important source of information — but, too often, the results are subjected to spin that benefits a political agenda. As that narrative gets reinforced, focus on progress can be lost.

For more on education politics and policy in Tennessee, follow @TNEdReport

 

 

 

NAEP: First Take

The 2015 NAEP results are out today and there is already discussion about what they mean both state-by-state and nationally.

Here’s what Governor Haslam had to say:

“Today, we’re very excited to say that based on 2015 NAEP results, we’re still the fastest improving state in the nation since 2011. What this means is a new set of fourth- and eighth-graders proved that the gains that we made in 2013 were real.”

That’s pretty strong language. Proved. Governor Haslam said this year’s results proved that the gains seen in 2013 were real.

Here’s what we know: Tennessee remained relatively flat – no significant growth, relatively small decline in reading scores. Basically, we are where we were in 2013.

Here’s what else we know: The entire nation remained relatively flat — no significant growth, some decline in math.

So, here’s what that means: In 2013, Tennessee gained faster than the national average. In exactly one testing cycle. In 2015, Tennessee didn’t do worse than the rest of the country. We also didn’t do better. Like the rest of America, we remained steady.

That is, it’s entirely possible the 2013 gains seen in Tennessee were a one-time occurrence. An outlier.

Had Tennessee again made gains that outpaced the nation, one could say the results suggest something special or different is happening in Tennessee that may be causing the gains. It’s important to be cautious until you have several years of data and more thorough analysis.

It’s also worth noting that states that have adopted aggressive reforms and states that haven’t both remained flat. The general trend was “holding steady,” and it didn’t seem to matter whether your state was using a reform agenda (charters, vouchers, value-added teacher scores in teacher evaluations) or not.

Again, this makes it difficult to suggest that any one or even a package of educational practices drives change.

Was Tennessee’s performance on NAEP in 2013 a blip or an indicator of actual progress? The 2015 results don’t provide much insight.

The good news: Tennessee held steady. The related news: So did everyone else.

I’ll be doing some more digging in to the data to examine trends over time and what more can be learned from 2015. Stay tuned…

For more on education politics and policy in Tennessee, follow @TNEdReport

 

 

 

Williamson County Teachers Vote “No Confidence” in Huffman

The Williamson County Education Association is the latest teacher group to express a lack of confidence in Tennessee Education Commissioner Kevin Huffman’s ability to lead the state’s schools.

The announcement of the vote adds to a growing list of teacher groups, superintendents, and school boards expressing frustration at Huffman’s lack of collaboration and top-down leadership style.

Williamson County’s Director of Schools, Mike Looney, signed a letter also signed by directors across the state expressing frustration with Huffman.

While the vote was taken before last week’s release of NAEP results, Williamson County teachers maintain they are not opposed to positive reform, but are frustrated with leadership at the state level they say is not listening to teachers.

For more on Tennessee education politics and policy, follow us @TNEdReport