Outlier

Statisticians define an outlier as an observation point that is distant from other observations in a statistical analysis. Often, this occurs by chance. Additional modeling or deeper analysis (including more data, for example, or a longer range of data) can often correct for this. Outliers that are not the result of measurement error are often excluded from analysis about a data set.

Today, the 2017 results from the National Assessment of Educational Progress (NAEP) were released. This release made me think of a particular outlier.

Back in 2013, Tennessee demonstrated what some heralded as an incredible achievement on the NAEP. In fact, a press release from Governor Haslam at the time noted:

Gov. Bill Haslam today announced that Tennessee had the largest academic growth on the 2013 National Assessment of Educational Progress (NAEP) of any state, making Tennessee the fastest improving state in the nation. (emphasis added)

Those words — “fastest improving state in the nation” — have been uttered by Haslam and many political leaders in our state for years now. Often, this 2013 “success” is used as justification for “keeping our foot on the gas” and continuing an aggressive agenda of test-based accountability and teacher evaluation based on methods lacking validity.

Here’s what I wrote back in 2013 when these results were released:

Yes, Tennessee should celebrate its growth.  But policymakers should use caution when seeing the results from the last 2 years as a validation of any particular policy.  Long-term trends indicate that big gains are usually followed by steady maintenance. And, even with the improvement, Tennessee has a long way to go to be competitive with our peers. Additionally, education leaders should be concerned about the troubling widening of the rich/poor achievement gap  – an outcome at odds with stated policy goals and the fundamental principle of equal opportunity.

Two years later, when the 2015 results were released, I noted:

This year’s scores, in which Tennessee remained steady relative to the 2013 scores suggest, if anything, that the 2013 jump was likely an outlier. Had the 2013 gains been followed by gains in 2015 and again in 2017, more could be suggested. And frankly, it is my hope that we see gains (especially in reading) in 2017. But, it’s problematic to suggest that any specific reform or set of reforms caused the one-time jump we saw in 2013. Saying we are the fastest improving state in the nation over the last 4 years when we only saw a jump in 2013 is like saying we started the first quarter of a football game way behind, scored a bunch in the second quarter, (so we’re not as far behind), and then scored the same number of points in the third quarter. The result is we’re still behind and still have a long way to go.

Fast forward to today. The leveling off I suggested was likely back in 2013 has happened. In fact, take a look at this chart put out by the Tennessee Department of Education:

First, notice that between 2009 and 2011, Tennessee saw drops in 4th and 8th grade reading and 8th grade math. That helps explain the “big gains” seen in 2013. Next, note that in 4th and 8th grade reading and 4th grade math, our 2017 scores are lower than the 2013 scores. There’s that leveling off I suggested was likely. Finally, note that in 4th and 8th grade reading, the 2017 scores are very close to the 2009 scores. So much for “fastest-improving.”

Tennessee is four points below the national average in both 4th and 8th grade math. When it comes to reading, we are 3 points behind the national average in 4th grade and 5 points behind in 8th grade.

All of this to say: You can’t say you’re the fastest-improving state on NAEP based on one testing cycle. You also shouldn’t make long-term policy decisions based on seemingly fabulous results in one testing cycle. Since 2013, Tennessee has doubled down on reforms with what now appears to be little positive result. Instead, as Rep. Jeremy Faison said recently, our policies are “driving teachers crazy.”

Oh, and that new TNReady test has so far not been very ready.

But what about the good policy coming from this? You know, like Governor Haslam’s plan to make Tennessee the “fastest-improving state in teacher pay?”

About that:

Average teacher salaries in the United States improved by about 4% from the Haslam Promise until this year. Average teacher salaries in Tennessee improved by just under 2% over the same time period. So, since Bill Haslam promised teachers we’d be the fastest improving in teacher pay, we’ve actually been improving at a rate that’s half the national average. No, we’re not the slowest improving state in teacher pay, but we’re also not even improving at the average rate.

Surely, though, all this focus on education since the NAEP buzz has meant meaningful investment in schools, right? Well, no:

Tennessee earns a grade of F when it comes to funding effort compared to funding ability. The researchers looked at Gross State Product and Personal Income data in order to determine a state’s funding ability then looked at dollars spent per $1000 (in either GSP or Personal Income) to determine effort. Tennessee spends $29 on schools for every $1000 generated in Gross State Product. When it comes to Personal Income, Tennessee spends just $33 per $1000 of average personal income. That’s a rank of 42 in both.

Then, the report looks at wage competitiveness — how much teachers earn relative to similarly-educated professionals. I’ve written about this before, and Tennessee typically doesn’t do well in this regard.

Maybe we’ve taken a minute to get serious about investing in programs targeting struggling students? Also, no:

One possible solution would be to embed funding for school-level RTI2 specialists in the state’s funding formula for schools, the BEP. In fact, Rep. Joe Pitts offered legislation that would do just that last year. His plan would have added funding for three RTI2 specialists at each school for a total projected cost of $167 million. Commissioner McQueen was quick to shoot that idea down and came back this year with the funding proposal of $13 million, or one specialist per district. That’s only $154 million short of adopting a plan that would actually meet the needs of a program many suggest is an important way to improve educational outcomes for Tennessee students.

Maybe we are closing achievement gaps? Again, no.

Back in 2013, Tennessee students eligible for free/reduced lunch had an average NAEP reading score of 256 and scored 20 points below the non-eligible students. Now, that average score is 252 (four points worse) and 19 points below. For 4th grade, there’s a similar story, with free/reduced lunch eligible students scoring 25 points below their non-eligible peers this year. Four years ago, it was 26 points.

We’re not moving the needle. Our most vulnerable students continue to be left behind. Meanwhile, we hear nice words from top policymakers and see little actual result in terms of tangible improved investment in schools or any meaningful upgrade in teacher pay. Our testing system has yet to be proven.

Maybe now Tennessee policymakers will stop repeating the “fastest-improving” line and start doing the actual work of investing in and supporting our schools.

In any case, the next time you hear someone spout off that tired “fastest-improving” line, just yell back: OUTLIER!

For more on education politics and policy in Tennessee, follow @TNEdReport


 

Memphis to Join NAEP TUDA

Shelby County Schools is among six districts joining the “Nation’s Report Card” via the Trial Urban District Assessment (TUDA) program.

Here’s the press release:

The National Assessment of Educational Progress (NAEP) will include six more urban school districts from around the country after a unanimous vote Saturday by the National Assessment Governing Board to expand the Trial Urban District Assessment (TUDA) program.

The six districts — Clark County School District (including Las Vegas); Denver Public Schools; Fort Worth Independent School District (Texas); Guilford County Schools (including Greensboro, North Carolina); Milwaukee Public Schools; and Shelby County Schools (including Memphis, Tennessee) — volunteered to be part of NAEP administration starting in 2017. TUDA is a special part of the NAEP program that provides results of how fourth- and eighth-graders perform in reading and mathematics in some of the nation’s largest urban school districts. The vote of the Governing Board, which sets policy for NAEP, brings the total number of TUDA districts to 27.

 

The idea for a big-city version of NAEP, also known as The Nation’s Report Card, originated in 2000, when the Council of the Great City Schools — a coalition of the nation’s large urban public school districts led by Executive Director Michael Casserly — requested that the Governing Board conduct a trial NAEP assessment for large urban school districts that volunteered to participate. Congress first authorized funding for TUDA in 2002, and increases in funding over time have enabled the Governing Board to expand the program.

 

“The Governing Board values Mr. Casserly’s foresight and leadership and the bipartisan support from Congress, the president and the Department of Education to support the expansion of this program,” said Governing Board Chair Terry Mazany. “TUDA provides school district leaders, parents and civic leaders with objective and comparable data to measure the progress of student achievement over time in many of the country’s largest school districts.”

 

“The addition of these six new cities to the Trial Urban District Assessment of NAEP is a major step forward for the program and will help sustain efforts to improve the nation’s large-city public schools well into the future,” Casserly said. “We are thrilled that 27 cities will be participating in 2017.”

 

TUDA tests representative samples of students and it reports district-level student achievement results, including trends over time. To be eligible for TUDA, a district must be in a city with a population of 250,000 or more, and at least half of its student population must include minority racial or ethnic groups or must be eligible for free and reduced-price lunch. New TUDA districts must be large enough to support testing three NAEP subjects per year in grades four and eight. The six districts join these other school systems:
  • Albuquerque Public Schools
  • Atlanta Public Schools
  • Austin Independent School District
  • Baltimore City Public Schools
  • Boston Public Schools
  • Charlotte-Mecklenburg Schools
  • Chicago Public Schools
  • Cleveland Metropolitan School District
  • Dallas Independent School District
  • Detroit Public Schools
  • District of Columbia Public Schools
  • Duval County Public Schools (Jacksonville, Florida)
  • Fresno Unified School District (California)
  • Hillsborough County Public Schools (Florida)
  • Houston Independent School District
  • Jefferson County Public Schools (Kentucky)
  • Los Angeles Unified School District
  • Miami-Dade County Public Schools
  • New York City Public Schools
  • School District of Philadelphia
  • San Diego Unified School District
“We now have an ever-greater geographic representation in TUDA, with four more states included. This will provide the nation with an objective picture of the achievement spanning the diversity of our nation’s students, recognizing that the majority of students in our nation’s schools is now composed of minority populations,” Mazany said.

 

View a list of current and eligible TUDA districts at www.nagb.org/policies/list-tuda-districts.html.

 

###

 

The National Assessment of Educational Progress is the only nationally representative, continuing evaluation of the condition of education in the United States. It has served as a national yardstick of student achievement since 1969. Through The Nation’s Report Card, NAEP informs the public about what American students know and can do in various subject areas and compares achievement among states, large urban districts, and various student demographic groups.

 

The National Assessment Governing Board is an independent, nonpartisan board whose members include governors, state legislators, local and state school officials, educators, business representatives, and members of the general public. Congress created the 26-member Governing Board in 1988 to oversee and set policy for NAEP.
For more on education politics and policy in Tennessee, follow @TNEdReport

It’s Raining Money, But Not on Schools

Tom Humphrey reports on the most recent budget projections which predict a surplus of between $300-$400 million for the fiscal year ending June 30, 2016.

This money, combined with the $600 million surplus from the fiscal year which ended on June 30, 2015, means the state will have about $1 billion in unanticipated, uncommitted revenue.

On top of that, economists are projecting growth in the $400-$500 million range for the upcoming budget year.

As Humphrey notes, proposals are floating so spend the surplus on road projects or tax cuts or both.

What’s not being mentioned?

Schools.

Despite a pair of lawsuits contending the state’s school funding formula, the BEP, is inadequate, lawmakers and the Governor are not rushing to suggest significant new investments in Tennessee schools.

This in spite of the fact that after a one year bounce on NAEP results, our state is now holding flat. Maybe that’s because Senate Education Committee Chair Dolores Gresham is suggesting our state aspire toward “mediocrity” while holding a forum on disastrous (and expensive) school voucher schemes.

If the state invested half of the available surplus on the BEP formula, that would be a $500 million injection of funds to local schools. That would be a sure way to hold down local property tax increases while also beefing up the resources school systems have to provide an education. The revenue projections for the 2016-17 fiscal year indicate an investment of this magnitude is sustainable. And, by holding a portion of the surplus in reserve, the state could ensure against any unanticipated shortfalls.

All of that would still leave $200-$300 million available to spend on one-time costs like road projects.

Will Tennessee put its foot on the accelerator and invest in schools so our students have the resources they need to compete with the rest of the country? Will Bill Haslam use the surplus and projected new revenue to truly make Tennessee the fastest improving state in the nation in teacher pay?

The General Assembly will have answers to these questions starting in January.

For more on education politics and policy in Tennessee, follow @TNEdReport

The NAEP Spin Room

Yesterday, I wrote about the very rosy interpretation of NAEP data being advanced by Tennessee leaders. Governor Haslam said:

“Today, we’re very excited to say that based on 2015 NAEP results, we’re still the fastest improving state in the nation since 2011. What this means is a new set of fourth- and eighth-graders proved that the gains that we made in 2013 were real.”

After analyzing the Tennessee results and putting them in context with national results (both of which essentially remained steady from 2013) , I noted:

It’s also worth noting that states that have adopted aggressive reforms and states that haven’t both remained flat. The general trend was “holding steady,” and it didn’t seem to matter whether your state was using a reform agenda (charters, vouchers, value-added teacher scores in teacher evaluations) or not.

Again, this makes it difficult to suggest that any one or even a package of educational practices drives change.

Then, I read the statement issued by SCORE (Statewide Collaborative on Reforming Education) Executive Director Jamie Woodson. Here’s what she had to say:

Since 2011, Tennessee has made record-setting gains, held them, and progressed in state rankings because of a multi-faceted strategy of high standards, great teaching, accountability, and common-sense adjustments based on the feedback of educators and citizens.

Note that she assigns causality based on these results. I wonder, then, what to make of the states that didn’t adopt the multi-faceted strategy she references? Last year, a number of states showed significant gains on NAEP. Some, like DC and Tennessee were reform-oriented states, others were not.

Additionally, in a post about the NAEP results two years ago, I noted:

Kentucky and Tennessee have posted gains over time on NAEP — in most categories, Kentucky started out tied or very slightly ahead of Tennessee and today, Kentucky remains ahead.  Kentucky posted some pretty big gains in the mid-90s and again from 2003-2009.  Since then, they’ve held fairly steady.  That’s an expected result, by the way — a big gain followed by steady maintenance of the new level.  For Tennessee, that won’t be enough, but celebrating the big gain is certainly warranted.  It’s also important to take care in assigning causality.

Note here that what I suggested then was an expected result (big gain, followed by holding steady) is exactly what happened in Tennessee this year. That’s good news — it means we’re not declining. But it also means we can’t really say that 2013 was something special.  As I noted last year, Kentucky had a series of big gains in the 1990s and then again in the early 2000s. It wasn’t just a big bump one time. So far, Tennessee has had one banner year (2013) and this year, returned to normal performance.

However, the narrative of “fastest-improving” keeps being repeated. In fact, Bethany Bowman of Professional Educators of Tennessee (PET) released a statement that said in part:
Tennessee students are still the fastest improving in the nation since 2011 according to the 2015 National Assessment of Educational Progress (NAEP), commonly known as the Nation’s Report Card. “This year’s results from National Assessment of Educational Progress (NAEP) show that Tennessee has maintained the positive gains that we achieved in 2013.

We had one year in which we made a big splash and then, as I noted in 2013:

As the data shows, Kentucky and Tennessee in many cases posted similar net gains over time, with Kentucky seeing big jumps in the mid-90s and again in the early part of the last decade.

That is to say, over a 20-year period, both states saw similar net gains. This year’s scores, in which Tennessee remained steady relative to the 2013 scores suggest, if anything, that the 2013 jump was likely an outlier. Had the 2013 gains been followed by gains in 2015 and again in 2017, more could be suggested. And frankly, it is my hope that we see gains (especially in reading) in 2017. But, it’s problematic to suggest that any specific reform or set of reforms caused the one-time jump we saw in 2013. Saying we are the fastest improving state in the nation over the last 4 years when we only saw a jump in 2013 is like saying we started the first quarter of a football game way behind, scored a bunch in the second quarter, (so we’re not as far behind), and then scored the same number of points in the third quarter. The result is we’re still behind and still have a long way to go.

So, yes, let’s celebrate that we made a big jump and held it steady. But, let’s also put those results in context and focus on how we can move forward instead of using these results to advance our favorite plays. For example, I’m not a huge fan of vouchers, but NAEP data doesn’t really help me make the case for or against. Likewise, states with and without strong collective bargaining posted gains in 2013 and held steady in 2015 — that is, the presence or absence of bargaining has no impact on NAEP scores.

NAEP can be an important source of information — but, too often, the results are subjected to spin that benefits a political agenda. As that narrative gets reinforced, focus on progress can be lost.

For more on education politics and policy in Tennessee, follow @TNEdReport

 

 

 

NAEP: First Take

The 2015 NAEP results are out today and there is already discussion about what they mean both state-by-state and nationally.

Here’s what Governor Haslam had to say:

“Today, we’re very excited to say that based on 2015 NAEP results, we’re still the fastest improving state in the nation since 2011. What this means is a new set of fourth- and eighth-graders proved that the gains that we made in 2013 were real.”

That’s pretty strong language. Proved. Governor Haslam said this year’s results proved that the gains seen in 2013 were real.

Here’s what we know: Tennessee remained relatively flat – no significant growth, relatively small decline in reading scores. Basically, we are where we were in 2013.

Here’s what else we know: The entire nation remained relatively flat — no significant growth, some decline in math.

So, here’s what that means: In 2013, Tennessee gained faster than the national average. In exactly one testing cycle. In 2015, Tennessee didn’t do worse than the rest of the country. We also didn’t do better. Like the rest of America, we remained steady.

That is, it’s entirely possible the 2013 gains seen in Tennessee were a one-time occurrence. An outlier.

Had Tennessee again made gains that outpaced the nation, one could say the results suggest something special or different is happening in Tennessee that may be causing the gains. It’s important to be cautious until you have several years of data and more thorough analysis.

It’s also worth noting that states that have adopted aggressive reforms and states that haven’t both remained flat. The general trend was “holding steady,” and it didn’t seem to matter whether your state was using a reform agenda (charters, vouchers, value-added teacher scores in teacher evaluations) or not.

Again, this makes it difficult to suggest that any one or even a package of educational practices drives change.

Was Tennessee’s performance on NAEP in 2013 a blip or an indicator of actual progress? The 2015 results don’t provide much insight.

The good news: Tennessee held steady. The related news: So did everyone else.

I’ll be doing some more digging in to the data to examine trends over time and what more can be learned from 2015. Stay tuned…

For more on education politics and policy in Tennessee, follow @TNEdReport