Tag Archives: Measurement Inc

Posted on by

A Modification

As we reported yesterday, Phase II of TNReady is not so ready. In fact, Grace Tatter reports that the problem is statewide, impacting grades 3-8.

Tatter cites an email from the Department of Education indicating the state is not sure when the Phase II tests will be delivered to districts.

The email also says:

“…Districts may modify their testing schedules as needed, without any prior approval or notice to the (state),”

The thing is, some districts have already been trying to modify their schedules by not giving the test at all. The idea of refusing to administer Phase II surfaced in Murfreesboro in late March and early April. The state responded by issuing a vague threat regarding withholding BEP funds.

Tullahoma City Schools on Monday approved a resolution unanimously calling on the state to cancel testing for the remainder of this year.

All of this was before the realization that Phase II tests would not make it to Tennessee districts on time. Now, though, the Department of Education’s own words suggest that districts may modify as they see fit without consulting the state. One possible modification would be to not administer the test at all. Another would be to schedule it for a time in June when students aren’t in school. Districts could say they offered the test, but no one showed up to take it.

The state has also made a big fuss about what happens to students/districts if students simply refuse to take the test. Trouble is, the state’s memo is based on some pretty fuzzy reasoning.

As this piece was being written, the Department of Education announced it will not ask districts to reschedule tests beyond the current testing window, which expires on May 10th. That means if materials are not received in time for administration by that date, districts don’t have to administer the tests. The Department also indicated it would provide additional flexibility to districts.

From Jason Gonzales:

The Tennessee Department of Education announced to districts Friday it won’t reschedule the TNReady testing window again this year and for those districts that don’t receive tests on time, will provide flexibility.

“We will not ask districts to reschedule again beyond what has been communicated to date, and we will not extend the testing window beyond May 10,” according to a statement sent to districts Friday.

So, what’s next? Will the state cancel the contract with testing vendor Measurement, Inc.? Will Commissioner McQueen assume responsibility for the failed transition to a new test?

Only time will tell, and there’s not much time left.

 

For more on education politics and policy in Tennessee, follow @TNEdReport

Posted on by

Back to the Future

After yesterday’s opening day fiasco with the new TNReady test, Commissioner Candice McQueen announced that all TNReady tests for this year will now shift to paper and pencil tests and a new testing window will be created. No testing will happen before February 22nd.

Here’s the letter Commissioner McQueen sent to Directors of Schools about the shift:

Thank you for your patience as we faced technical challenges with the MIST platform this morning. At 8:25 a.m. CST the state’s vendor for TNReady, Measurement Incorporated, experienced a severe network outage, causing significant problems with the MIST platform. Like you, we are incredibly disappointed that the MIST platform was not accessible to schools across the state as the Part I testing window opened.

 

Shortly after learning about the issue, we advised that schools experiencing problems with the test discontinue testing, and return to their normal classes.

 

Throughout the 2015-16 school year, the department has continuously worked with Measurement Incorporated to strengthen the online testing platform. As a result of district feedback and through our efforts to collaborate, we have mitigated and eliminated many technical issues. The online platform has undergone many capacity tests, yielding actionable information to drive improvements. Following Break MIST Day last October, we’ve made significant investments in server capacity. As a follow up to our Jan. 12 capacity test, the department’s technology team also spent multiple weeks in the field visiting select districts around the state to reproduce system errors in a real-world, real-time situation to gather better diagnostic information. As a result of this continued analysis, we offered districts the option to move to paper testing as we saw continuing issues with how the platform interacted with districts’ infrastructure.

 

Unfortunately, issues have continued to arise with the online platform. The new nature of the issue this morning has highlighted the uncertainly around the stability of Measurement Inc.’s testing platform, MIST. Despite the many improvements the department has helped to make to the system in recent months and based on the events of this morning, we are not confident in the system’s ability to perform consistently. In the best interest of our students and to protect instructional time, we cannot continue with Measurement Incoporated’s online testing platform in its current state. Moving forward, during the 2015-16 school year TNReady will be administered via paper and pencil (both Part I and Part II).

 

We thank districts, schools, and teachers for their commitment and perseverance to move our students to a 21st century learning environment. We know this is what the real world requires. We understand and appreciate the investment of time, money, and effort it has taken to attain readiness.

 

As a result of a statewide shift to paper and pencil, we will delay and extend the Part I testing window. Measurement Incorporated is currently scheduling the printing and shipping of the paper tests, and the department will share the revised testing window with districts by Thursday of this week. We understand that the shift to paper and pencil testing has many scheduling implications for your schools, teachers, and students. We thank you for your patience and cooperation as we transition to a test medium that we are confident will allow all students to show what they know.

 

TNReady is designed to assess true student understanding and problem-solving abilities, not just basic memorization skills. Regardless of the medium of assessment, this new and improved test will provide schools, teachers, and parents with valuable information about our students college and career readiness.

 

Warning Signs

Prior to Monday’s scheduled test administration, some educators across the state were raising concerns about the testing system and its ability to handle the load of student all across the state.

Amanda Haggard reports:

In a letter sent to the Tennessee Department of Education Commissioner Candice McQueen on Jan. 31, RePublic CEO Ravi Gupta outlines exactly what happened when the school made its attempt at the test. From the letter:

Our experience on January 28, however, raised substantial concerns about the technical capacity of MIST [Measurement Incorporated Secure Testing platform] to support state-wide testing. RePublic has only 1,200 kids — a tiny fraction of the State’s 500,000. On January 28, we attempted to administer the Math practice test on MIST as a step toward preparing kids for the first round of state exams. More than half of our kids were unable to log on, were kicked off the platform after logging on, or could not submit a completed test. The critical issue, confirmed by MIST representatives, was an error or series of errors on MIST’s own servers.

Haggard goes on to detail other concerns raised ahead of Monday’s test administration.

A Call for a Pause

In response to the challenges presented by the TNReady test administration, some legislators are now calling for a pause on test-based accountability for students, teachers, and schools. The tests would still be administered, and results reported, but they would not impact student grades, teacher evaluations, or the state’s priority schools list.

What Happens Now?

The state now has asked its vendor, Measurement, Inc. to provide paper and pencil tests. These will not start before February 22nd. Districts and schools will have to reschedule testing based on the availability of tests and guidance from the Department of Education.

Now that the tests have shifted to pencil and paper, some are asking how they will be graded. Human graders were always a part of the equation due to the constructed-response nature of the tests, but they will now be assessing handwritten responses.

A Pattern?

This is the third consecutive year the state has had problems with its testing regimen. In 2014, quick scores were not ready in time to be factored into student grades. Last year, there was a change in quick score calculation that was not clearly communicated to districts and which resulted in confusion when results were posted.

 

For more on education politics and policy in Tennessee, follow @TNEdReport

 

 

Posted on by

Testing Time

While Tennessee teachers are raising concerns about the amount of time spent on testing and test preparation, the Department of Education is lauding the new TNReady tests as an improvement for Tennessee students.

According to an AP story:

However, the survey of nearly 37,000 teachers showed 60 percent say they spend too much time helping students prepare for statewide exams, and seven out of ten believe their students spend too much time taking exams.

“What teachers recognize is the unfortunate fact that standardized testing is the only thing valued by the state,” said Jim Wrye, assistant executive director of the Tennessee Education Association, the state’s largest teachers’ union.

“Teachers and parents know there are so many things that affect future student success that are not measured by these tests, like social and emotional skills, cooperative behaviors, and academic abilities that do not lend themselves to be measured this way.”

Despite teacher concerns, the Department of Education says the new tests will be better indicators of student performance, noting that it will be harder for students to “game” the tests. That’s because the tests will include some open-ended questions.

What they don’t mention is that the company administering the tests, Measurement, Inc., is seeking test graders on Craigslist. And, according to a recent New York Times story, graders of tests like TNReady have, “…the possibility of small bonuses if they hit daily quality and volume targets.”  The more you grade, the more you earn, in other words.

Chalkbeat summarizes the move to TNReady like this:

The state was supposed to move in 2015 to the PARCC, a Common Core-aligned assessment shared by several states, but the legislature voted in 2014 to stick to its multiple-choice TCAP test while state education leaders searched for a test similar to the PARCC but designed exclusively for Tennessee students.

Except the test is not exactly exclusive to Tennessee.  That’s because Measurement, Inc. has a contract with AIR to use test questions already in use in Utah for tests in Florida, Arizona, and Tennessee.

And, for those concerned that students already spend too much time taking standardized tests, the DOE offers this reassurance about TNReady:

The estimated time for TNReady includes 25-50 percent more time per question than on the prior TCAP for English and math. This ensures that all students have plenty of time to answer each test question, while also keeping each TNReady test short enough to fit into a school’s regular daily schedule.

According to the schedule, the first phase of testing will start in February/March and the second phase in April/May. That means the tests are not only longer, but they also start earlier and consume more instructional time.

For teachers, that means it is critical to get as much curriculum covered as possible by February. This is because teachers are evaluated in part based on TVAAS — Tennessee Value-Added Assessment System — a particularly problematic statistical formula that purports to measure teacher impact on student learning.

So, if you want Tennessee students to spend more time preparing for and taking tests that will be graded by people recruited on Craigslist and paid bonuses based on how quickly they grade, TNReady is for you. And, you’re in luck, because testing time will start earlier than ever this year.

Interestingly, the opt-out movement hasn’t gotten much traction in Tennessee yet. TNReady may be just the catalyst it needs.

For more on education politics and policy in Tennessee, follow @TNEdReport

Posted on by

Is John Oliver Reading TN Ed Report?

John Oliver recently took on the issue of standardized testing and it sounds like he’s been reading Tennessee Education Report. In 18 brilliant minutes, he hits on a number of topics covered here time and again.

Oliver discussed teacher merit pay, the recruiting tactics of testing companies, value-added assessment, and testing transparency.

Back in 2013, Tennessee’s State Board of Education moved toward merit pay based on value-added data.

This year, while adding nearly $100 million to the pot for teacher compensation, Governor Haslam continued a push for merit pay.

While Oliver noted that Pearson recruits test scorers on Craigslist, Tennessee’s new testing vendor, Measurement, Inc. uses the same practice.

And of course, there’s the issue of value-added assessment — in Tennessee, called TVAAS. While it yields some interesting information, it’s not a reliable predictor of teacher performance and it’s going to be even more unreliable going forward, due to the shift from TCAP to TNReady. Here’s what we’ve learned from TVAAS in Tennessee:

In fact, this analysis demonstrates that the difference between a value-added identified “great” teacher and a value-added identified “average” teacher is about $300 in earnings per year per student.  So, not that much at all.  Statistically speaking, we’d call that insignificant.  That’s not to say that teachers don’t impact students.  It IS to say that TVAAS data tells us very little about HOW teachers impact students.

Surprisingly, Tennessee has spent roughly $326 million on TVAAS and attendant assessment over the past 20 years. That’s $16 million a year on a system that is not yielding much useful information.

And then there’s testing transparency. Oliver points out that it’s difficult if not impossible to get access to the actual test questions. In fact, Tennessee’s testing vendor, Measurement, Inc., has a contract with Utah’s testing vendor that involves a fine if test questions are revealed — $5000 per question:

The contract further notes that any release of the questions either by accident or as required by law, will result in a fee of $5000 per test item released. That means if Tennessee wants to release a bank of questions generated from the Utah test and used for Tennessee’s assessment, the state would pay $5000 per question.

Here’s the clip from John Oliver:

 

For more on education politics and policy in Tennessee, follow @TNEdReport

 

Posted on by

TNReady … Already?

Back in November, the State of Tennessee awarded a contract to Measurement Inc. to develop the new assessment that would replace TCAP.

This assessment is to be aligned to state standards (largely based on Common Core State Standards) and should take into account feedback from Tennesseans.

Measurement Inc. will be paid $108 million for the contract.

Chalkbeat noted at the time the contract was awarded:

Measurement Inc. is subcontracting to AIR, a much larger player in the country’s testing market. AIR already has contracts with Utah and Florida, so Tennessee educators will be able to compare scores of Tennessee students with students from those states “with certainty and immediately.” AIR is also working with Smarter Balanced, one of two federally funded consortia charged with developing Common Core-aligned exams. That means that educators in Tennessee will also likely be able to measure their students’ progress with students in the 16 states in the Smarter Balanced Consortium.

The Department of Education notes on its website:

Comparability: While the assessments will be unique to Tennessee, TNReady will allow Tennesseans to compare our student progress to that of other states. Through a partnership between Measurement Inc. and American Institutes for Research, TNReady will offer Tennessee a comparison of student performance with other states, likely to include Florida and Utah.

While Measurement Inc. has an interesting approach to recruiting test graders, another item about the contract is also noteworthy.

The Department and Chalkbeat both noted the ability to compare Tennessee test scores with other states, including Utah and Florida.

Here’s why that’s possible. On December 5th, the Utah Board of Education approved the use of revenue from test licensing agreements with Florida, Arizona, and Tennessee based on contracts with AIR, the organization with which Measurement Inc. has a contract, as noted by Chalkbeat.

The contract notes that Utah’s expected arrangement in Tennessee is worth $2.3 million per year (running from 2015-2017) and that Tennessee will use questions licensed for the Utah assessment in Math and ELA in its 2015-16 assessment.

So, Tennessee’s new test will use questions developed for Utah’s assessment and also licensed to Florida and Arizona.

The contract further notes that any release of the questions either by accident or as required by law, will result in a fee of $5000 per test item released. That means if Tennessee wants to release a bank of questions generated from the Utah test and used for Tennessee’s assessment, the state would pay $5000 per question.

While Tennessee has said it may change or adapt the test going forward, it seems that the 2016 edition of the test may be well underway in terms of its development.

For more on education politics and policy in Tennessee, follow @TNEdReport

 

Posted on by

Ready to Grade?

Measurement, Inc. has been hired by the State of Tennessee to design new standardized tests to replace TCAP. The new test is to be aligned to Tennessee’s new standards and will include constructed-response questions in addition to multiple choice. This means students will write answers or demonstrate work as part of the test. The idea is to demonstrate understanding of a subject, rather than simply guessing on a multiple choice test. Typically, grading a constructed response test is costly, because evaluators have to read and consider the answers and then rate them based on a rubric. Fortunately for Tennessee taxpayers, Measurement, Inc. has found a way to keep these costs low.

Here’s an ad from Measurement seeking Evaluators/Readers for tests:

Thank you for your interest in employment with Measurement Incorporated. We are a diverse company engaged in educational research, test development, and the scoring of tests administered throughout the world. Our company has grown to be the largest of its kind by providing consistent and reliable results to our clients. We are able to do so through the efforts of a professional and flexible staff, and we welcome your interest in becoming a member. Measurement Incorporated Reader/Evaluator Position Recruiting for projects starting in March of 2015 for both day and evening shift at the Ypsilanti Scoring Center. If you qualify as a reader/evaluator, you will be eligible to work on a number of our projects. Many projects require readers to score essays for content, organization, grammatical convention, and/or the student’s ability to communicate and to respond to a specific directive. Other projects involve scoring test items in reading, math, science, social studies, or other subject areas. The tests you will score come from many different states and from students at all grade levels, elementary through college, depending on the project.

LOCATION Measurement Incorporated Ypsilanti Scoring Center 1057 Emerick Ypsilanti, MI 48198 (734) 544-7686

REQUIREMENTS Bachelor’s degree in any field Ability to perform adequately on a placement assessment Completion of a successful interview Access to a home computer with high speed internet in a secure work area for telecommuters

HOURS Readers are hired on a temporary basis by project but are expected to work five days per week, Monday through Friday. Hours vary by shift. Attendance during training (usually the first few days of a project) is mandatory. PAY The starting pay is $10.70 per hour. After successful completion of three major scoring projects (or a minimum of 450 hours), readers who meet the minimum standards of production, accuracy and attendance will receive an increase to $11.45 per hour.

APPLICATION PROCEDURE To apply, please go to http://www.measurementinc.com/Employment/ and select the Reader/Evaluator position. Select Ypsilanti as your location and click on the “Apply Online” tab. Qualified applicants will be contacted to complete an online placement assessment, schedule an interview, and provide proof of degree. If invited to work on a scoring project, proof of employment eligibility in order to complete a federal I-9 from will be required within three days of employment.

Apparently, scorers at the Nashville scoring center can earn starting pay of $11.20 an hour.

 

Certainly, quality scorers for TNReady can be found for $10.70-$11.20 an hour via ads posted on Craigslist. I’m sure parents in the state are happy to know this may be the pool of scorers determining their child’s test score. And teachers, whose evaluations are based on growth estimates from these tests, are also sure to be encouraged by the validity of results obtained in this fashion. So, if you have a Bachelor’s degree and want to make around $11 an hour on a temporary, contract basis by all means, get in touch with the developers of Tennessee’s new standardized tests. For more on education politics and policy in Tennessee, follow @TNEdReport

Posted on by

The TN Poverty Test

TCAP is Tennessee’s standardized test for grades 3-8.  At least until next year, when it is replaced with something designed by Measurement, Inc. that meets new Tennessee Standards.

TCAP stand for Tennessee Comprehensive Assessment Program.  But, it could just as easily stand for Tennessee Comprehensive Assessment of Poverty.

Here’s why:

An analysis of TCAP performance over time indicates that those school systems with consistently high levels of poverty tend to have consistently low scores on TCAP. Likewise, those systems with the least amount of poverty tend to have consistently higher scores on TCAP.

Much attention was focused on Tennessee and our “rapid gains” on the NAEP. Less celebrated by state officials was the attendant expansion of the achievement gap between rich and poor students.

One possible explanation for the expanding achievement gap is the investment gap among districts. That is, those districts with lower levels of poverty (the ones scoring higher on TCAP) also tend to invest funds in their schools well above what the state funding formula (BEP) generates. The top ten districts on TCAP performance spend 20% or more above what the BEP formula generates. By contrast, the bottom 10 districts spend 5% or less above the formula dollars.

It’s no accident that the districts that spend more are those with less poverty while the districts with less investment above the BEP have higher poverty levels. And, I’ve written recently about the flaws in the present BEP system that signal it is well past time to reform the formula and increase investment.

Of further interest is an analysis of 3-year ACT averages. Here again, 9 of the top 10 districts on ACT performance spend well above the state average in per pupil spending. The top 10 districts in ACT average spend an average of $900 more per student than the state’s average per pupil expenditure.

And, on ACT scores again, those districts with the highest poverty rates make the least investment above BEP dollars and typically see results below the state average ACT score.

While Tennessee may be moving to a new test in 2016, it’s not clear yet whether that test will do more than identify the poverty level and education investment of the state’s school districts.

For more on education politics and policy in Tennessee, follow @TNEdReport