That was Quick

The Tennessee Department of Education is out with an apology for miscommunication that caused confusion regarding this year’s standardized testing “quick scores.”

Grace Tatter over at Chalkbeat has the story, and this quote from a letter sent to Directors of schools from Assistant Commissioner Nakia Towns:

“Our goal is to communicate early and often regarding the calculation and release of student assessment data. Unfortunately, it appears the office of assessment logistics did not communicate decisions made in fall 2014 regarding the release and format of quick scores for the 2014-15 school year in a timely manner. . . . We regret this oversight, and we will continue to improve our processes such that we uphold our commitment to transparency, accuracy, and timeliness with regard to data returns, even as we experience changes in personnel.”

As Tatter notes, this is the second year in a row that release of quick scores has been a problem for the Department of Education.

Read her full story and see the complete text of the letter sent to Directors.

It remains to be seen whether the “commitment to transparency” referenced in the letter from Towns will mean that parents and teachers can see the test questions and answers after next year’s TNReady test is administered.

For more on education politics and policy in Tennessee, follow @TNEdReport

Quick and Confusing

Over at Bluff City Ed, Jon Alfuth digs into the questions surrounding this year’s release of TCAP quick scores and their correlation to student performance on the TCAP.

This year, the way quick scores were calculated in relation to raw scores was shifted so that grades 3-8 (TCAP) scores matched the EOC scores students see in high school.

One key question is why make this change in the last year of TCAP? Next year, Tennessee students will see TNReady — so, making the calculation change now doesn’t seem to serve much purpose.

Alfuth does a nice job of explaining what’s going on and why it matters. Here are some key highlights:

Lack of Communication

They (TN DOE) didn’t make it clear to teachers, parents or students that they were changing the policy, resulting in a lot of confusion and frustration over the past few days as everyone grapples with these new quick scores.

An Explanation?

From the second memo, they note that they changed to raw scores because of concerns about getting final quick scores out on time during the transition to a new test, stating that if they did it based on proficiency, it would take until the middle of the summer to make them happen.

I’d buy that…except that the Department of Education has always been able to get the quick scores out on time before. And last I checked, we weren’t transition to TNReady this year – the transition occurs next year. So why mess with the cut scores this year? Is this just a trial run, an experiment? It feels like we’re either not getting the whole story, or that if we are there is some serious faulty logic behind this decision that someone is just trying to explain away.

It’s worth noting that last year, the quick scores weren’t available on time and most districts received a waiver from including TCAP scores in student grades. I note this to say that concern about getting quick scores out on time has some merit given recent history.

To me, though, this raises the question: Why are TCAP scores factored into a student’s grades? Ostensibly, this is so 1) students take the tests seriously and 2) how a teacher assesses a student matches up with the desired proficiency levels on the appropriate standards.

Of course, quick scores are only available for tested subjects, leaving one to wonder if other subjects are less important or valuable to a student’s overall academic well-being. Or, if there’s another way to assess student learning beyond a bubble-in test or even a test with some constructed response, such as TNReady.

I’d suggest a project-based learning approach as a means of assessing what student’s have actually learned across disciplines. Shifting to project-based learning with some grade-span testing would allow for the accountability necessary to ensure children are meeting state standards while also giving students (and their teachers) a real opportunity to demonstrate the learning that has occurred over an academic year.

Trust

The Department has also opened itself to some additional criticism that it is “massaging” the scores – that is, trying to make parents happy by bringing grades up in the last year under the old testing regime. We can’t say for certain that this is the motivating factor behind this step, but in taking this step without more transparency the Department of Education has opened itself up to this charge. And there will definitely be some people who accuse the state of doing this very thing, especially given the reasons that they cited in their memo. I personally don’t ascribe any sinister motives to the state, but you have to admit that it looks a little fishy.

In fact, TC Weber is raising some important questions about the process. He notes:

If people don’t believe in the fidelity of the system, it becomes too easy to attribute outside factors to the results. In other words, they start to feel that data is being manipulated to augment an agenda that they are not privy to and not included in. I’m not saying results are being manipulated or not being manipulated when it comes to our student evaluation system, but I am saying that there seems be a growing belief that they are, and without some kind of change, that perception will only grow. I’ve always maintained that perception is nine-tenths of reality.

As both Alfuth and Weber note, the central problem is lack of communication and transparency. As we shift to a new testing regime with uncertain results, establishing confidence in the system and those administering it is critical. After last year’s late score debacle and this year’s quick score confusion, establishing that trust will be difficult. Open communication and a transparent process can go a long way to improving perception and building support.

For more on education politics and policy in Tennessee, follow @TNEdReport

PET Releases Testing Survey

Professional Educators of Tennessee (PET) released a survey this week on teacher attitudes toward standardized testing.

Here’s the release and a link to a detailed report:

In April of 2015, Professional Educators of Tennessee surveyed Tennessee educators regarding their opinion of standardized testing in the state of Tennessee. The survey was distributed via email to all members and on social media, as well as being made available to all educators on the Professional Educators of Tennessee website.

208 educators completed the survey, with 134 being classroom teachers. Eighty-five percent of educators stated that standardized testing takes up “too much” of classroom instructional time. And, as the state moves to online testing, there appear to be numerous glitches in the testing procedure.

Based on the survey results, PET recommends:

Based on these survey results, standardized testing in Tennessee proves to be a major driving force in classroom instruction. This survey indicates that virtually every school has broadband internet, yet 89% indicated there were issues with the online testing provided.   These issues can and will negatively impact tests results. Professional Educators of Tennessee proposes that all testing continue to be done on paper/pencil OR that testing sessions interrupted by technical difficulties be coded in a special way and either discarded or given again, with different test items, OR that schools endure the tests with possible difficulties with technology and be held harmless until the percentage of tests taken without technical interference or interruption reaches a threshold of 95% or higher.  Also, before a teacher’s TVAAS scores are linked to students’ testing performance, these online testing malfunctions (computers/websites freezing, connectivity issues) must be addressed.

For more on education politics and policy in Tennessee, follow @TNEdReport

Is John Oliver Reading TN Ed Report?

John Oliver recently took on the issue of standardized testing and it sounds like he’s been reading Tennessee Education Report. In 18 brilliant minutes, he hits on a number of topics covered here time and again.

Oliver discussed teacher merit pay, the recruiting tactics of testing companies, value-added assessment, and testing transparency.

Back in 2013, Tennessee’s State Board of Education moved toward merit pay based on value-added data.

This year, while adding nearly $100 million to the pot for teacher compensation, Governor Haslam continued a push for merit pay.

While Oliver noted that Pearson recruits test scorers on Craigslist, Tennessee’s new testing vendor, Measurement, Inc. uses the same practice.

And of course, there’s the issue of value-added assessment — in Tennessee, called TVAAS. While it yields some interesting information, it’s not a reliable predictor of teacher performance and it’s going to be even more unreliable going forward, due to the shift from TCAP to TNReady. Here’s what we’ve learned from TVAAS in Tennessee:

In fact, this analysis demonstrates that the difference between a value-added identified “great” teacher and a value-added identified “average” teacher is about $300 in earnings per year per student.  So, not that much at all.  Statistically speaking, we’d call that insignificant.  That’s not to say that teachers don’t impact students.  It IS to say that TVAAS data tells us very little about HOW teachers impact students.

Surprisingly, Tennessee has spent roughly $326 million on TVAAS and attendant assessment over the past 20 years. That’s $16 million a year on a system that is not yielding much useful information.

And then there’s testing transparency. Oliver points out that it’s difficult if not impossible to get access to the actual test questions. In fact, Tennessee’s testing vendor, Measurement, Inc., has a contract with Utah’s testing vendor that involves a fine if test questions are revealed — $5000 per question:

The contract further notes that any release of the questions either by accident or as required by law, will result in a fee of $5000 per test item released. That means if Tennessee wants to release a bank of questions generated from the Utah test and used for Tennessee’s assessment, the state would pay $5000 per question.

Here’s the clip from John Oliver:

 

For more on education politics and policy in Tennessee, follow @TNEdReport

 

The End of an Era

Over at Bluff City Ed, Jon Alfuth celebrates the end of the EOC testing era. Those tests will be replaced with TNReady next year.

Alfuth notes that there are many challenges with the current testing regime, including gaming the system and misalignment with current standards.

Here’s what he says he hopes the new tests provide:

First, I’d personally like to see aligned pre- and formative assessments to allow teachers to track tests throughout the year. These could be given to the districts and used to develop a benchmark for where students are starting and track their progress throughout the year. These should be designed by Measurement Inc. to ensure close alignment to the actual test.

Second, we need to see shorter tests. Asking students to sit for between 2 to 4 three hour assessments in a four day period is a lot, and it does stress kids out. I’d like to see the number of questions reduced on the new TNReady assessments to reflect this reality.

Third, we need better special education and special needs accommodations. I’m not a special education teacher myself, but from talking to some of my colleagues my understanding is that the accommodations for the EOC regime aren’t the greatest. Hopefully a technologically advanced test like TNReady (it can be given on paper or on a computer) could include better accommodations for kids with special needs. I also hope it makes automatic adjustments for students who, say, speak English as a second language.

Fourth, we need to see a substantial increase of resources aligned to the new assessments and SOON. Teachers need time to internalize the format at the types of questions that students will be asked to complete on the new assessments. That was one of the failings of PARCC and one reason I believe we no longer have it in Tennessee – teachers didn’t have enough supporting resources and backed off support for the assessment. Lets hope that TNReady doesn’t make the same mistake.

More on TNReady:

TNReady to Borrow Questions from Utah

Transition to TNReady Creates TVAAS Problems

For more on education politics and policy, follow @TNEdReport

TNReady … Already?

Back in November, the State of Tennessee awarded a contract to Measurement Inc. to develop the new assessment that would replace TCAP.

This assessment is to be aligned to state standards (largely based on Common Core State Standards) and should take into account feedback from Tennesseans.

Measurement Inc. will be paid $108 million for the contract.

Chalkbeat noted at the time the contract was awarded:

Measurement Inc. is subcontracting to AIR, a much larger player in the country’s testing market. AIR already has contracts with Utah and Florida, so Tennessee educators will be able to compare scores of Tennessee students with students from those states “with certainty and immediately.” AIR is also working with Smarter Balanced, one of two federally funded consortia charged with developing Common Core-aligned exams. That means that educators in Tennessee will also likely be able to measure their students’ progress with students in the 16 states in the Smarter Balanced Consortium.

The Department of Education notes on its website:

Comparability: While the assessments will be unique to Tennessee, TNReady will allow Tennesseans to compare our student progress to that of other states. Through a partnership between Measurement Inc. and American Institutes for Research, TNReady will offer Tennessee a comparison of student performance with other states, likely to include Florida and Utah.

While Measurement Inc. has an interesting approach to recruiting test graders, another item about the contract is also noteworthy.

The Department and Chalkbeat both noted the ability to compare Tennessee test scores with other states, including Utah and Florida.

Here’s why that’s possible. On December 5th, the Utah Board of Education approved the use of revenue from test licensing agreements with Florida, Arizona, and Tennessee based on contracts with AIR, the organization with which Measurement Inc. has a contract, as noted by Chalkbeat.

The contract notes that Utah’s expected arrangement in Tennessee is worth $2.3 million per year (running from 2015-2017) and that Tennessee will use questions licensed for the Utah assessment in Math and ELA in its 2015-16 assessment.

So, Tennessee’s new test will use questions developed for Utah’s assessment and also licensed to Florida and Arizona.

The contract further notes that any release of the questions either by accident or as required by law, will result in a fee of $5000 per test item released. That means if Tennessee wants to release a bank of questions generated from the Utah test and used for Tennessee’s assessment, the state would pay $5000 per question.

While Tennessee has said it may change or adapt the test going forward, it seems that the 2016 edition of the test may be well underway in terms of its development.

For more on education politics and policy in Tennessee, follow @TNEdReport

 

Ready to Grade?

Measurement, Inc. has been hired by the State of Tennessee to design new standardized tests to replace TCAP. The new test is to be aligned to Tennessee’s new standards and will include constructed-response questions in addition to multiple choice. This means students will write answers or demonstrate work as part of the test. The idea is to demonstrate understanding of a subject, rather than simply guessing on a multiple choice test. Typically, grading a constructed response test is costly, because evaluators have to read and consider the answers and then rate them based on a rubric. Fortunately for Tennessee taxpayers, Measurement, Inc. has found a way to keep these costs low.

Here’s an ad from Measurement seeking Evaluators/Readers for tests:

Thank you for your interest in employment with Measurement Incorporated. We are a diverse company engaged in educational research, test development, and the scoring of tests administered throughout the world. Our company has grown to be the largest of its kind by providing consistent and reliable results to our clients. We are able to do so through the efforts of a professional and flexible staff, and we welcome your interest in becoming a member. Measurement Incorporated Reader/Evaluator Position Recruiting for projects starting in March of 2015 for both day and evening shift at the Ypsilanti Scoring Center. If you qualify as a reader/evaluator, you will be eligible to work on a number of our projects. Many projects require readers to score essays for content, organization, grammatical convention, and/or the student’s ability to communicate and to respond to a specific directive. Other projects involve scoring test items in reading, math, science, social studies, or other subject areas. The tests you will score come from many different states and from students at all grade levels, elementary through college, depending on the project.

LOCATION Measurement Incorporated Ypsilanti Scoring Center 1057 Emerick Ypsilanti, MI 48198 (734) 544-7686

REQUIREMENTS Bachelor’s degree in any field Ability to perform adequately on a placement assessment Completion of a successful interview Access to a home computer with high speed internet in a secure work area for telecommuters

HOURS Readers are hired on a temporary basis by project but are expected to work five days per week, Monday through Friday. Hours vary by shift. Attendance during training (usually the first few days of a project) is mandatory. PAY The starting pay is $10.70 per hour. After successful completion of three major scoring projects (or a minimum of 450 hours), readers who meet the minimum standards of production, accuracy and attendance will receive an increase to $11.45 per hour.

APPLICATION PROCEDURE To apply, please go to http://www.measurementinc.com/Employment/ and select the Reader/Evaluator position. Select Ypsilanti as your location and click on the “Apply Online” tab. Qualified applicants will be contacted to complete an online placement assessment, schedule an interview, and provide proof of degree. If invited to work on a scoring project, proof of employment eligibility in order to complete a federal I-9 from will be required within three days of employment.

Apparently, scorers at the Nashville scoring center can earn starting pay of $11.20 an hour.

 

Certainly, quality scorers for TNReady can be found for $10.70-$11.20 an hour via ads posted on Craigslist. I’m sure parents in the state are happy to know this may be the pool of scorers determining their child’s test score. And teachers, whose evaluations are based on growth estimates from these tests, are also sure to be encouraged by the validity of results obtained in this fashion. So, if you have a Bachelor’s degree and want to make around $11 an hour on a temporary, contract basis by all means, get in touch with the developers of Tennessee’s new standardized tests. For more on education politics and policy in Tennessee, follow @TNEdReport

TCAP, Poverty, and Investment in Schools

Recently, I wrote about the Tennessee Comprehensive Assessment of Poverty, or TCAP. In that piece, I talked about how concentrated poverty combined with low investment in schools led to poor educational outcomes. I also mentioned how the broken BEP impacts districts because it is inadequate to meet the needs of Tennessee’s schools.

Now, I want to share the data I used to make those claims.

This data will show % of investment above BEP requirements, 3 year average ACT score (where applicable) and average TCAP scores.

The Top 10

District                % above BEP           3 yr ACT avg.             TCAP avg.

FSSD                   44.94%                     n/a                                63

Rogersville         19.83%                     n/a                                60

Newport             14.51%                      n/a                                62

Maryville            33.8%                      23.8                               65

Oak Ridge          37.23%                    23.1                               58

Williamson       20.5%                       22.9                             67

Greeneville      27.47%                      22.1                             58

Johnson City  26.77%                       22.1                             61

Kingsport        31.85%                       22                               59

Shelby              17.32%                       20.8                           58

AVERAGE    27.42%                    22.4                         61.1

The Top 10 districts in terms of student achievement invested nearly 28% above the BEP requirements and had an ACT average well above the state average.

The Bottom Ten

District          % above BEP          3 yr. ACT avg.              TCAP avg.

Lake                5.07%                       18.1                                 41

Union             4.91%                       17.9                                 45

Madison         14.22%                    17.9                                 46

Campbell       3.4%                        17.7                                  44

Haywood       6.48%                     17.5                                  41

Hardeman    11.58%                    17                                      46

Hancock       4.49%                     16.6                                   44

Memphis      19.15%                   16.4                                    38

Fayette         9.83%                    16.3                                     42

Humboldt   13.5%                     16.2                                    43

AVERAGE 9.26%                 17.16                                43

The bottom ten districts in terms of student performance invest less than 10% above the BEP formula and have an ACT average well below the state average.

The top 10 districts spend an average of 3 times more than the bottom 10 in terms of investment over the BEP formula. They also have an ACT average that is 5 points higher and a TCAP average that is nearly 20 points higher than the bottom ten.

Interestingly, even the bottom 10 districts spend just over 9% more than the BEP formula on average. That’s a sure sign that districts can’t run on the funds and funding levels established by the current BEP. The BEP is simply inadequate to meet Tennessee’s educational needs.

For more on education politics and policy in Tennessee, follow @TNEdReport

Mandated to Death

That’s how Sewanee Elementary School teacher Rachel Reavis says she feels when it comes to the amount of testing going on in her school these days.

Reavis made the remarks at a forum with a local school board member hosted by the school’s PTO.

The Sewanee Mountain Messenger has the full story of the meeting, where parents and teachers alike expressed frustration at the amount of testing being done, even at the pre-k level.

Nine Tests in Pre-K?

Parent Janna McClain, a former academic interventionist in Murfreesboro, said her son will take nine mandated tests this year in pre-K. “Who thought that was a good idea? As a parent it would be helpful to know what it is our teachers are being forced to do,” McClain said. “I think the rationale is to prepare for these tests that are connected to dollars, so we have to do more and more tests,” she added. “I understand mandated testing, but I don’t want my child tested nonstop.”

Principal Agrees: Testing is Excessive

“The pendulum has swung to excessive testing,” said SES principal Mike Maxon. “There needs to be a balance.” Certain programs that involve mandated testing also require additional interventions in specific areas, which can be detrimental because it draws students away from other core subjects and creative learning.

Maxon went on to note that the interventions, a part of the Response to Intervention program (RTI2), are being conducted by related arts teachers and guidance counselors because financial support is not provided by the state to pay for the required interventions.

For his part, school board member Adam Tucker said he understands parent and teacher concerns about excessive testing and wants to explore options to reduce testing so students can focus on related arts and other areas that enrich the educational experience.

More on Testing

A Tennessee Teacher Challenges Arne Duncan

Toward a New Model of Testing in Tennessee?

Parents, Educators Challenge Over-Reliance on Testing

Amy Frogge vs. Testing

For more on education politics and policy in Tennessee, follow @TNEdReport

 

Replacing TCAP

Measurement, Inc. has been hired by the State of Tennessee to design new assessments in ELA and Math.

The contract came about because the General Assembly passed legislation calling on the state to open bidding for new assessments rather than continue as planned with administration of the PARCC tests.

Here’s an email sent to educators today explaining the upcoming changes:

Over the past several months, Gov. Haslam and his administration, including the state department of education, have participated in a number of ongoing conversations with you and your colleagues about K-12 education in Tennessee. These conversations have reflected both the historic progress Tennessee has made through your work as well as your concerns and recommendations for improvement. One emerging theme from these discussions has been the challenges experienced by educators due to the uncertainty of the state’s assessments in English language arts (ELA) and math and the impact of administering the existing TCAP exams while meeting the current ELA and math academic standards.
We are excited to report to you that this week the state of Tennessee completed the process to replace the state’s current TCAP assessments in ELA and math. The new measurements of learning for ELA and math will be called Tennessee Ready (TNReady). These assessments, to be administered by Measurement Inc., were selected through a fair, thorough and transparent process established by the General Assembly and administered by the state’s Central Procurement Office.
TNReady will be administered beginning in the 2015-16 school year and will assess our state standards in ELA and math. These standards are located on the department of education website (ELA is here and math is here).
You’ll find additional information about TNReady below:

  • By Tennessee, For Tennessee: Tennessee educators – both at the K-12 and higher education levels – were significantly involved in the selection process and chose an assessment that is both fully aligned to the state’s academic standards but also adaptable to future improvements. Tennessee will make decisions about item selection, test length and composition, and scoring. In the future, Tennessee will decide on changes to the test based on changes to standards, and Tennesseans will be engaged in item development and review.
  • Higher Expectations and Critical Thinking:  TNReady will expand beyond just multiple choice questions to include: writing that requires students to cite text evidence at all grade levels; questions that measure math fluency without a calculator; and questions that ask students to show their work in math with partial credit available.
  • Resources for Parents and Teachers:  Online tools will be available for schools and teachers to develop practice tests that can provide students, teachers, and parents with valuable and immediate feedback. These resources will be available before the end of the 2014-15 school year.
  • Comparability:  While the assessments will be unique to Tennessee, TNReady will allow Tennesseans to compare our student progress to that of other states. Through a partnership between Measurement Inc. and American Institutes for Research, TNReady will offer Tennessee a comparison of student performance with other states, likely to include Florida and Utah.
  • Training:  The Tennessee Department of Education will provide training for educators across the state during the summer of 2015.
  • Test Administration & Scoring: TNReady will have two parts. The first portion, which will replace the state’s current comprehensive writing assessment, will require extended written responses in ELA and math and will be administered in February/March. The second portion will include selected responses, such as multiple choice and drag-and-drop items, and will be administered in April/May.
  • Technology: TNReady will be administered online and available for use on multiple devices with minimal bandwidth. As most states move their tests for all grade levels online, we must ensure Tennessee students do not fall behind their peers in other states. However, all districts will have the option of administering paper-pencil exams.

We look forward to sharing additional details about the new assessments in the coming months.  Additional information will be posted on the new TNReady page of our website.
Finally, as previously noted, Tennessee will make appropriate revisions to assessments in the future to reflect any change in the academic standards. Recently, Gov. Haslam and the State Board of Education announced a public review process in which all Tennesseans will have an opportunity to provide input on our ELA and math standards. These public comments will then be reviewed by committees of Tennessee educators, which will make recommendations to the state board. We encourage all of you to be engaged in this process in an effort to ensure our academic standards continue to reflect higher expectations for our students. To participate in the standards review process, visit https://apps.tn.gov/tcas/. We want to thank you for your patience and acknowledge the tremendous dedication you have shown in improving the life outcomes for Tennessee students and their families. Thank you for what you do every day.

For more on education policy and politics in Tennessee, follow @TNEdReport