NAEP in TN: The Rest of the Story

Today, leaders across Tennessee lauded the NAEP (National Assessment of Educational Progress) results.  And they should. Tennessee is first in an education statistic and it’s a good one.  The fastest growing state in the last 2 years in terms of growth in math and reading scores as measured by NAEP.

That’s good news. It’s very good news.  Despite some claims, though, it’s very difficult to say results on the 2013 NAEP are a direct result of reforms that took place in 2011 and 2012. States with more rigid teacher tenure and with collective bargaining for teachers scored higher overall than Tennessee (nevermind Ron Ramsey’s rant against both — they just don’t test out as significant indicators of student achievement in either a positive or negative way). And of course, it’s easier to grow when you have a long way to go — Tennessee has historically been among the lowest performing states on the NAEP.

Let’s take a look at the data on a deeper level, though, and see what’s been happening. For the sake of this comparison, I’m going to look at Tennessee and Kentucky — a similarly situated Southeastern state with a nearly identical level of students in poverty and/or on free/reduced lunch.  I’m going to look at 20 year trends to see what we can learn from the overall education work in both states. So, here goes:

4th Grade Math

1992      KY   215                        TN  211

2013     KY  241                          TN 240

Over the 20 year period, Kentucky increased by 16 points, Tennessee by 19 — and Kentucky still leads by 1 point.  To Tennessee’s credit, the gap in scores was narrowed by 3 points.

Let’s look at the percentage of students in each state who test at or above basic — this is short of the mastery demonstrated by the score of proficient, but still indicates a basic understanding of the concept — below basic is the lowest score and is frankly, unacceptable.

In 1992, 51% of Kentucky kids tested at or above basic and in Tennessee, it was 47%.  Now, 84% of Kentucky kids are at or above basic in 4th grade math while only 80% can say the same in Tennessee.  Both states posted 33 point gains in this important number over the last 20 years and Kentucky remains 4 points ahead of Tennessee.

Now, let’s look at the two states and how they are doing with their poorest students, those on free and reduced lunch.  One of the key goals of many involved in education is closing achievement gaps and moving the lowest performing kids forward quickly.

FREE/REDUCED LUNCH 4th GRADE MATH SCORES

KY  232  TN 228

Achievement gap

KY 19 points  TN 26 points

Not only do Tennessee’s students on free and reduced lunch score lower than Kentucky’s, Tennessee’s gap is wider — by a 7-point margin in the case of 4th grade math.  This begins a troubling pattern.

Before I go further with this analysis, I want to point out that Kentucky doesn’t use value-added data for teacher evaluations, has no charter schools, its teachers are awarded tenure after 4 years, and it hasn’t adopted any of the reforms Tennessee’s current leaders tell us are essential to improving scores.  In fact, their Commissioner has openly expressed skepticism of any evaluation system that bases any part of a teacher’s score on value-added data.  As the rest of the data will demonstrate, both Kentucky and Tennessee have posted gains over time on NAEP — in most categories, Kentucky started out tied or very slightly ahead of Tennessee and today, Kentucky remains ahead.  Kentucky posted some pretty big gains in the mid-90s and again from 2003-2009.  Since then, they’ve held fairly steady.  That’s an expected result, by the way — a big gain followed by steady maintenance of the new level.  For Tennessee, that won’t be enough, but celebrating the big gain is certainly warranted.  It’s also important to take care in assigning causality.

Ok, back to the data.

8th Grade Math

1992  KY 262   TN 259

2013  KY 281  TN 278

Over 20 years, both states made a 19-point gain in 8th grade math and Kentucky maintains a 3-point lead.  Looking at students at or above “basic,” Kentucky was at 51% in 1992 and is at 71% today while Tennessee was at 47% in 1992 and at 69% today.  Kentucky gained 20 points and Tennessee, 22.

FREE/REDUCED LUNCH 8th GRADE MATH SCORES

KY 268  TN 265

Achievement Gap

KY 25 points TN 27 points

 

4th Grade Reading

1992 KY 213   TN 212

2013 KY 224   TN 220

Here, Kentucky makes an 11-point gain and Tennessee makes an 8-point gain over the same time period.  Now, Kentucky has a solid 4-point lead in reading — while in 1992, it was just 1-point.  In terms of students at or above “basic,” Kentucky was at 58% in 1992 and stands at 71% today while Tennessee was at 57% in 1992 and is now at 67% — Kentucky gained 13 points over this time, while Tennessee gained 10.

FREE/REDUCED LUNCH 4th GRADE READING SCORES

KY 213  TN 205

Achievement Gap

KY 24 points TN 32 points

 

8th Grade Reading

1998  KY 262   TN 259

2013 KY 270  TN 265

Here, Kentucky gained 8 points and Tennessee only 6 — giving Kentucky students a 5-point edge over Tennessee’s in 8th grade reading. Both states posted 6-point gains in percentage of students at or above “basic,” with Kentucky maintaining a 3-point edge in that category.

FREE/REDUCED LUNCH 8th GRADE READING SCORES

KY 258  TN 256

Achievement Gap

KY 23  TN 20

As the data shows, Kentucky and Tennessee in many cases posted similar net gains over time, with Kentucky seeing big jumps in the mid-90s and again in the early part of the last decade.  In all categories, Kentucky’s students still outperform Tennessee, though in some cases that gap is narrowing.  Also, in all subjects, Kentucky’s students on free/reduced lunch outperform Tennessee’s students on free/reduced lunch.

ABOUT THAT FREE LUNCH

Possibly the most interesting (and troubling) finding in this data is the widening of the gap between free/reduced lunch students and those not eligible.  Tennessee has a significant population of students who qualify (as does Kentucky) and one of the key aims of reform is to ensure that gaps are closed and that those with the most challenges get more opportunity.  Here’s some data demonstrating that Tennessee’s achievement gap is widening when it comes to its poorest students.

FREE/REDUCED LUNCH ACHIEVEMENT GAPS

4th Grade Math 2011 — 20 points  2013 – 26 points

8th Grade Math 2011 — 25 points  2013 – 27 points

4th Grade Reading 2011 — 26 points  2013 – 32 points

8th Grade Reading 2011 — 20 points  2013 – 20 points

The 4th grade scores in particular present rapidly widening gaps.  That’s absolutely the wrong direction.  Moreover, students on free/reduced lunch saw their scores improve less than those not on free/reduced lunch 3 points vs. 9 points in 4th grade math, 3 points vs. 5 points in 8th grade math, 1 point vs. 7 points in 4th grade reading, and both groups saw a +6 in 8th grade reading.

While we’re told that “poverty is not an excuse” it certainly appears to be a factor (and one growing in importance) in terms of student achievement growth in Tennessee. While we have had significant reforms in some of our poorest urban communities (and even have an Achievement School District to address the most challenged schools), the gap between poor and better off students widened in the last two years.

Yes, Tennessee should celebrate its growth.  But policymakers should use caution when seeing the results from the last 2 years as a validation of any particular policy.  Long-term trends indicate that big gains are usually followed by steady maintenance. And, even with the improvement, Tennessee has a long way to go to be competitive with our peers. Additionally, education leaders should be concerned about the troubling widening of the rich/poor achievement gap  – an outcome at odds with stated policy goals and the fundamental principle of equal opportunity.

For more on education politics and policy in Tennessee, follow us @TNEdReport

 

 

 

19 thoughts on “NAEP in TN: The Rest of the Story

  1. Pingback: NAEP IN TN: THE REST OF THE STORY | ICL Blog

  2. I continue to think “TN Ed Report” should be better titled “TN Ed Selective Bias Narrative”…but oh well, it’s a free country.

    I also don’t know any other current research that looks at 20 year trends to determine next year’s policy goals and legislation. 20 year trends is stuff that goes in history books.

    You can’t criticize the last 2 years of administration reforms (when it was really 4-5 years of work), and then analyze 20 years of NAEP scores.

    The time frame needs to be kept at least close to the same.

    Which, if we do that…

    KY actually had a slight regression from 2011 to 2013 test cycle, which is likely why you had to move it to 20 years.
    http://jaypgreene.files.wordpress.com/2013/11/2013-naep-gains.jpg

    While you do recognize “It’s also important to take care in assigning causality.”…a good deal of this post spends time implying causality.

    “Despite some claims, though, it’s very difficult to say results on the 2013 NAEP are a direct result of reforms that took place in 2011 and 2012.”

    If you actually listened to Gov Haslam, Comm Huffman, and Gov Bredesen, they didn’t make this claim. They attributed the growth to a combination of things, that included higher standards, and yes, professional teacher evaluations. They were intentional about recognizing that it is a mult-administration effort.

    Higher standards were mainly Bredesen admin, and yes, we can’t forget actual history that Bredesen helped put teacher eval framework into law during special RTT legislative session.

    Yes, Haslam & Huffman have worked to implement it and improve it, but teacher eval system has spanned 2 administrations.

    Your post’s claim about reforms “just” being 2011 and 2012 are simply inaccurate.

    I thought it was quite dignified and classy that Gov Haslam recognized the Bredesen administration and their work on ed policy. Given toxic environment of political parties in America today, this is almost unheard of. Your post failed to mention any of that or recognize that, but again, came across as a criticism of the current administration.

    TN Ed Report: Haslam, Huffman, bad! all the time!

    “Before I go further with this analysis, I want to point out that Kentucky doesn’t use value-added data for teacher evaluations, has no charter schools, its teachers are awarded tenure after 4 years, and it hasn’t adopted any of the reforms Tennessee’s current leaders tell us are essential to improving scores.”

    Why you mention charter schools is again another point of selective bias.

    Charters in TN are so small relative to the overall student population, that mathematically, TN would have had the same gains had there been no charters in TN.

    But you had to mention it.

    People read that and associate causality to it. Because it’s part of the narrative you like to continue to promote.

    You spend a good bit of your post predicting that these gains will level out. “That’s an expected result, by the way — a big gain followed by steady maintenance of the new level. ”

    Actual data indicates otherwise:
    https://dl.dropboxusercontent.com/u/2850574/STCSC%20Scann13110810190.pdf

    Massachusetts and Indiana have both had multiple test cycles of big gains.

    While you make some good points in your piece, the main impetus of the narrative was to again criticize the current reforms of the sitting administration.

    It just doesn’t add up and it skews us farther away from the middle, where we should be having the conversation, centered on student outcomes and doing what’s best for them.

    • Hunter:

      I always appreciate your feedback. Looking at 20 year trends is instructive, b/c education is not a one or five year endeavor, it takes 13 years on the K-12 side. Looking at historical policy impacts makes sense when deciding the path forward. Yes, Kentucky’s scores (save in reading) where down slightly in the past two years. I don’t hear anyone saying their move to Common Core is the cause for this very slight regression. And I certainly note that TN should be very happy with the reported results. I applaud the focus that’s been on education in Tennessee over the last 5+ years — from Bredesen’s work on Pre-K and higher standards to this Administration’s work on Praxis standards and the carrying out of the evaluation system. My argument remains that the current Administration is doing a poor job of implementing reform because of a lack of collaboration. Some reform (improving the observation portion of the evaluation piece) is certainly welcome and was certainly needed. I have never been a fan of tying value-added data to teacher evaluations or hiring and firing decisions. Yes, Bredesen started it. And yes, Haslam is now continuing that policy and possibly tying these scores to licensure and pay. Rather than connect with classroom educators and school leaders, they proceed with a policy many educators think is unwise. There’s still no robust support system for early career teachers.

      I note the 20 year trends also because to me, it points out that it is difficult to assign causation to any one policy, let alone a whole series of reforms. Did higher standards play a role? That seems likely. Will improved evaluation prove out to have a long-term benefit? That also seems likely. But it’s difficult to tell at this point.

      Kentucky had some nice gains in the 90s, as I point out, and then again early in the 2000s. Tennessee has also had previous periods of significant gain, though nothing quite like this. That’s impressive.

      I also think it noteworthy, as I mention, that the achievement gap seems to be widening. That should be cause for concern.

      You’ll note I begin and end by suggesting that Tennessee should be encouraged by these results, but cautious about how they use the information going forward.

      Advocates of various reform strategies should also take care in using the results as an argument for or against policies they like/dislike.

      I mention charters b/c so often we hear from advocates and policymakers that if we only had more charters, thinks would improve. Or, that if states like KY adopted charters, they’d improve faster. The truth is, neither state is doing particularly well overall. Kentucky posted big gains in the past after reform of their funding system. Since then, they’ve continued to make steady gains.

      Tennessee has made gains in the past, though the current 2-year cycle is certainly impressive. Saying we should continue or advance a favored policy b/c this data “prove” it is working is just not credible. But I have a feeling we’ll hear something like, “we’ve got to keep our foot on the accelerator and adopt even more strategies like merit pay, etc — see, the NAEP scores show us its working.” That doesn’t hold up to close scrutiny.

    • Hunter,
      ALL quality researchers evaluate trends in data. That’s why researchers in medicine, biology, education, economics, psychology, etc. do literature reviews- to identify factors that have the most probable influence over changes over time. All data, particularly social science data have ‘noise’, also known as error.The problem is that tests scores have massive amounts of error. NAEP data, by design, cannot be interpreted to identify what specific factors effected score changes. Hence, touting score changes as meaning Huffman’s policies caused a few points to change is specious and misleading. Finding the causal change agents in test data are like looking for a needle in a haystack. Hence, a 1 or 2 point change has little meaning unless it is compared to past trends and to other data sources with similar population demographics.

      This is the reason we should not believe the hype over test scores being drummed up by politician’s PR machines. The tests are being misused and misinterpreted by individuals who are ignorant of testing psychometrics and whose vested interest is in a test and punish model.

    • Mass. is instituting the teacher evaluation system the same way Tennessee did- against the desire or need of the teachers! They do not need it! The have had the highest achievement for years.

  3. JCGrim-
    Yes, researchers in other fields do literature reviews, but not often 20 years back for a current policy comparison.

    I’m well aware of the noise implications – that’s why it’s not quite right to say TN was “42nd” or 46th or so on…the standard error could mean they were actually 40th or 50th or what not.

    But you can say TN went from the 40s to the 30s because the scale score plus any standard error is accounted for.

    I’m with you on the causality part – anyone can claim their brand of reform was responsible for the NAEP growth.

    At the end of the day, I think it is a combination of things that led to the NAEP gain, and trying to focus on what’s best for students.

    On the same token – the “haters” that choose certain policies – e.g. professional teacher evaluation, charter schools, etc…cannot make the claim that those initiatives detract from the gains.

    Let’s just keep focusing on what’s best for students. There will be disagreement about what that means, but let’s have a conversation in the middle on that to figure out a way forward (and not let our focus shift to what’s best for the adults).

  4. Pingback: Tale of the NAEP | Spears Strategy

  5. Pingback: Horn & Wilburn on TN NAEP | Spears Strategy

  6. Pingback: Are Techer Evaluations Education ‘Reform’s’ Biggest Bust?

  7. Pingback: Students or Profits | Spears Strategy

  8. Pingback: Tennessee Education Report | Huffman Resigns as TN Ed Commissioner

  9. Pingback: Tennessee Education Report | The TN Poverty Test

  10. Pingback: Toward Grade-Span Testing | Spears Strategy

  11. Pingback: Tennessee Education Report | NAEP: First Take

Leave a Reply

Your email address will not be published. Required fields are marked *