Pearson's Standardized Tests Are Flawed And Useless

Bombshell report in the NY Times on standardized testing:

In 2006, a math pilot program for middle school students in a Dallas-area district returned surprising results.

The students’ improved grasp of mathematical concepts stunned Walter Stroup, the University of Texas at Austin professor behind the program. But at the end of the year, students’ scores had increased only marginally on state standardized TAKS tests, unlike what Mr. Stroup had seen in the classroom.

A similar dynamic showed up in a comparison of the students’ scores on midyear benchmark tests and what they received on their end-of-year exams. Standardized test scores the previous year were better predictors of their scores the next year than the benchmark test they had taken a few months earlier.

Now, in studies that threaten to shake the foundation of high-stakes test-based accountability, Mr. Stroup and two other researchers said they believe they have found the reason: a glitch embedded in the DNA of the state exams that, as a result of a statistical method used to assemble them, suggests they are virtually useless at measuring the effects of classroom instruction.

Pearson, which has a five-year, $468 million contract to create the state’s tests through 2015, uses “item response theory” to devise standardized exams, as other testing companies do. Using I.R.T., developers select questions based on a model that correlates students’ ability with the probability that they will get a question right.

That produces a test that Mr. Stroup said is more sensitive to how it ranks students than to measuring what they have learned. That design flaw also explains why Richardson students’ scores on the previous year’s TAKS test were a better predictor of performance on the next year’s TAKS test than the benchmark exams were, he said. The benchmark exams were developed by the district, the TAKS by the testing company.

Mr. Stroup, who is preparing to submit the findings to multiple research journals, presented them in June at a meeting of the Texas House Public Education Committee. He said he was aware of their implications for a widely used and accepted method of developing tests, and for how the state evaluates public schools.

“I’ve thought about being wrong,” Mr. Stroup said. “I’d love if everyone could say, ‘You are wrong, everything’s fine,’ ” he said. “But these are hundreds and hundreds of numbers that we’ve run now.”

...

Mr. Stroup’s research comes as opposition to high-stakes standardized testing in Texas is creating an alliance between parents, educators and school leaders who wonder how the tests affect classroom instruction and small-government conservatives who question the expense and bureaucracy they impose.

...

State Representative Scott Hochberg, Democrat of Houston, led the charge against the measure and has since proposed legislation aimed at reforming the role of standardized testing because of data showing that a student’s test score on the first year highly predicted it for the next.

“I have for a long time said that the accountability system doesn’t give us all the information that the numbers are used to generate,” Mr. Hochberg said, adding that basing accountability “more on the kid’s history than the specifics of what happened in the classroom that year may make us feel good but it doesn’t give us any true information.”

Great - the tests are flawed, but they're using them to make high stakes decisions on students, teachers and schools in states all over the nation, thanks to the Obama administration and the corporate education reform movement, led by the Gates Foundation.

Fantastic!

And how much money will Pearson and all the vendors who provide testing materials and test prep make off these flawed measurements and tests?

Somewhere between $20 billion and $50 billion.

This isn't just a travesty - it's criminal.

And yet, here in NY State Governor Cuomo and his education brain trust have decided that standardized tests are the only measure of teacher performance in the classroom and plan to add dozens of standardized tests a year not to measure student performance but rather to measure teacher performance.

Again, fantastic!

One wonders just how much money Pearson is throwing around to get these deals with the states for all this testing.

We know they bribed the former NYSED Commissioner David Steiner.

We know too that Regents Chancellor Merryl Tisch has family in the test prep business, so she stands to benefit from all this testing too (We also know the parent company of her brother-in-law's test prep company is being investigated for securities fraud.)

As for current NYSED Commissioner John King, hard to know if he's taken any bribes from Pearson, but we do know that he has navigated quite a career from teacher to charter school leader to NYSED bureaucrat to youngest NYSED commissioner ever.

It doesn't hurt John King's career trajectory that he is such a promoter of standardized testing. In fact, one might think that's exactly how he's risen so fast in education leadership circles - he's happy to shill a corporate education reform agenda, including a heavy emphasis on test-based accountability systems.

I suspect that this study won't change anything here in NY State because the standardized testing train has already left the station and is barreling down the track at too fast a speed to be stopped.

We will spend a few years closing more schools, firing more teachers and leaving more kids back based on these flawed tests and measurements, all the while feeding Pearson scarce resources to do the crappy job they do with these things.

Too bad the AFT didn't jump off the standardized testing train years before, when we could have stopped these plans from being implemented (though it's nice to have them on record now acknowledging the harms that high stakes testing does.)

Unfortunately, I think it is now too late.

We will just have to pick up the pieces after it becomes apparent to the public the harm that is being done by the high stakes accountability system.

If it becomes apparent.

There is a lot of money to be made by the testing companies, the tech companies, the politicians, the education "consultants," and the rest of the education infrastructure with the Common Core Federal Standards, the new federal tests that will be rolled out as part of the CCFS, and the state and district tests that have been mandated as part of new teacher evaluation systems in many states.

You can bet these stakeholders will try and keep the high stakes standardized testing gravy train going for as long as they can - and they'll get lots of help from the corporate media, many of whom are stakeholders themselves.

Remember, the Washington Post owns Kaplan and News Corporation is in the for profit education game as well.

As I like to write all the time when talking about the financial markets, it's all rigged.

It's all rigged.

And there's so much money involved in this game that the boys and girls making it aren't going to want to unrig it any time soon.