Time For An Investigation Into New York State's Test Score Results, Data Tracking

Two stories from over the weekend show that the statistics and data issued by NYSED Commissioner John King and his merry men and women in reform at SED cannot be trusted.

First, as I posted yesterday, the NY Post covered how NYSED lowered the cut score levels on this year's state Common Core exams, an act which resulted in slightly higher test scores:

State officials touted increases in scores on tough Common Core exams this year but failed to reveal that they had lowered the number of right answers needed to pass half the exams.

The state Education Department dropped the number of raw points needed to hit proficiency levels in six of the 12 English and math exams given to students in grades 3 to 8, officials acknowledged.

“The reason that occurs is because the tests are slightly harder,” Deputy Education Commissioner Ken Wagner told The Post.

Student scores plunged on last year’s statewide 3-8 tests — the first based on the new Common Core standards. Before the 2013 exams, a panel of 95 educators decided how many points, or correct answers, students had to get to demonstrate proficiency.

But the point cutoffs were tweaked after this year’s tests. The state and its testing vendor, Pearson, found six tests were harder and four easier this year than in 2013, Wagner said.

They did so by comparing how students performed on “anchor” test questions — identical items used in both 2013 and 2014. A report on the scoring process will be released in December or January, Wagner said.

The changes raise questions about the validity of the results.

“The information given out about the test questions does not provide a complete picture, making it hard to judge how much progress students made last year,” said Fred Smith, a former testing analyst for city schools.

Score manipulation has erupted in scandal before. Between 2006 and 2009, the state reduced the number of raw points students needed to pass. Then-state Education Commissioner Richard Mills insisted the questions got harder, justifying the lower passing scores. But experts found the test items got easier, inflating scores hailed by then-Mayor Mike Bloomberg, among others, as proof of great progress.

In addition to the Post report on the lowered cut scores, Stephen Rex Brown at the Daily News reported that NYSED couldn't account for a whole bunch of kids they said opted out of the state tests this year in New York City that the city said didn't:

State Education officials were scrambling to determine Friday why test data appeared to show more than 20,000 city students did not take math and English exams.

The perplexing numbers, which the city disputed, revealed 26,949 kids were no-shows for state math tests and 22,656 skipped the English Language Arts exam. The figures were more than triple the previous year’s numbers. State officials suspect there was an error in the way a large group of city students were coded in the state database of third- through eighth-graders who took the tests.

The city Education Department said only 1,925 students formally opted out of the exam — still double the estimate from critics of the April tests.

Leonie Haimson at NYC Public School Parents blog picked up that story, noting that while she would love to believe that the number of students opting out of the state exams in the city was as high as NYSED said it was, she didn't think this was so:

As much as I would have liked to believe the opt out figures were this high, I expressed skepticism to Stephen– and explained that I thought the numbers of students opting out had been far higher on Long Island and Westchester than in NYC. In the suburbs, in general, parents are more organized, enjoy well-funded public schools with high college-going and graduation rates, and have erupted in justified incredulity when the state tried to convince them their schools were failing and their kids were not “college and career ready.” 

Leonie goes on to note that SED's data should not be considered reliable:

My response to all this: with such erratic and unreliable information, how can anyone trust any of the test score data from NYSED?

...

Before the new Common Core tests, we had ten years of state test score inflation in NY that was obvious to anyone paying close attention, but year after year was ignored by the powers that be, because it was politically convenient. Each year Mayor Bloomberg and Chancellor Joel Klein, sometimes accompanied by Randi Weingarten, would ritualistically bow down to the supposedly infallible test score gods and celebrate the results as showing that their reforms were working. And then the entire imaginary edifice came tumbling down in 2010, when the educrats finally admitted that an enormous test score inflation had occurred, somehow without their knowledge and complicity.

It is too early to assume that the small rise in test scores this year were due to similar manipulations , but a decade of experience should teach us to be open to the possibility. Merryl Tisch predicted that more kids would pass this year – and they did. In any event, we have overwhelming evidence from teachers and principals that the tests were poor quality and a lousy judge of real learning.

The state’s release of data showing thousands of opt outs in NYC is just one more piece of evidence showing how skeptical everyone should be about any data our government officials supply. 

The lowered cut scores that allowed SED to claim "progress" on their Common Core agenda and the disputed opt-out's here in the city are two examples for why there needs to be scrutiny into NYSED's testing operations from an outside entity unaligned with the Board of Regents, the State Education Department or the "non-profit" education reform groups that bolster both the Regents and SED (i.e., the Gates Foundation, the Broad Foundation, etc.)

As Leonie notes there has been test score manipulation and inflation in the state before and it happened when our current Regents Chancellor Merryl Tisch was on the Board of Regents.

In fact, Tisch defended former SED Commissioner Mills over the test score inflation, claiming there was none, when it was quite clear there was.

Michael Winerip, the former NY Times columnist, wrote up a timeline of test score inflation here in New York State that is worth revisiting at this time:

In the last decade, we have emerged from the Education Stone Age. No longer must we rely on primitive tools like teachers and principals to assess children’s academic progress. Thanks to the best education minds in Washington, Albany and Lower Manhattan, we now have finely calibrated state tests aligned with the highest academic standards. What follows is a look back at New York’s long march to a new age of accountability.

DECEMBER 2002 The state’s education commissioner, Richard P. Mills, reports to the state Regents: “Students are learning more than ever. Student achievement has improved in relation to the standards over recent years and continues to do so.”

JANUARY 2003 New York becomes one of the first five states to have its testing system approved by federal officials under the new No Child Left Behind law. The Princeton Review rates New York’s assessment program No. 1 in the country.

SPRING 2003 Teachers from around New York complain that the state’s scoring of newly developed high school tests is out of whack, with biology and earth science tests being too easy and the physics test too hard. The state Council of School Superintendents finds the physics scores so unreliable, it sends a letter to colleges for the first time in its history urging them to disregard the test result. Dr. Mills does not flinch, calling the tests “statistically sound” and “in accordance with nationally accepted standards.”

JUNE 2003 Scores on the state algebra test are so poorly calibrated that 70 percent of seniors fail. After a statewide outcry, officials agree to throw out the results. The Princeton Review says that ranking New York first was a mistake. “We’re going to have to come up with a fiasco index for a state like New York that messes up a lot of people’s lives,” a spokesman says.

OCTOBER 2003 A special panel appointed to investigate the state math fiasco concludes that the test “can’t accurately predict performance,” was created “on the cheap” and was full of exam questions that were “poorly worded” and “confusing.”

DECEMBER 2003 The director of state testing resigns. It was his idea to leave, a spokesman says.

MAY 2004 For the fourth year in a row, scores have risen on elementary and middle school state reading and math tests. Dr. Mills urges the Regents: “Look at the data that shows steadily rising achievement of the standards in school districts of all wealth and categories. More children are learning more now than ever before.”

FEBRUARY 2005 Dr. Mills rebukes those who question whether state scores are inflated. “The exams are not the problem,” he said in a report to the Regents. “It’s past time to turn from obsessive criticism of the exam and solve the real problems — the students who are not educated to the standards.”

SPRING 2005 New York City fourth graders make record gains on the state English test, with 59 percent scoring as proficient, compared with 49 percent the year before. “Amazing results” that “should put a smile on the face of everybody in the city,” says Mayor Michael R. Bloomberg, who happily recites the numbers on his way to re-election.

FALL 2005 The federal tests (the National Assessment of Educational Progress), which are considered more rigorous than the state tests, show a drop in New York City reading scores. On the eighth-grade test, 19 percent are proficient in 2005, compared with 22 percent in 2003. Asked if city and state officials had hyped the state test results, Merryl H. Tisch, a Regent, says, “They have never, ever, ever exaggerated.”

SEPTEMBER 2007 New York’s national assessment test results are again dismal; eighth-grade reading scores are lower than they were in 1998.

DECEMBER 2007 In his report to the Regents, Dr. Mills notes, “A rich, scholarly literature has challenged NAEP validity since the early 1990s.” He announces a plan to develop the first new state learning standards since 1996, to further spur academic excellence.

JUNE 2008 Newly released state test scores show another record year for New York children. Math scores for grades three through eight indicate that 80.7 percent are proficient, up from 72.7 in 2007. “Can we trust these results?” Dr. Mills asks. “Yes, we can. New York’s testing system, including grades three through eight tests, passed a rigorous peer review last year by the U.S. Department of Education. State Education Department assessment experts commission independent parallel analyses to double- and sometimes triple-check the work of our test vendor.”

JUNE 2009 In the previous decade, New York students’ average SAT verbal score has dropped to 484 from 494; the math SAT score has dropped to 499 from 506. The national assessment’s fourth-grade reading scores have been stagnant for four years, and the eighth-grade scores are their lowest in a decade.

But somehow, state test scores again soar to record levels. In New York City, 81 percent of students are deemed proficient in math, and 68.8 percent are proficient in English. “This is a big victory for the city,” the schools chancellor, Joel I. Klein, says, “and we should bask in it.” In November the mayor is elected to a third term, again riding the coattails of sweet city scores.

JULY 2010 Finally someone — Dr. Tisch, the chancellor of the Board of Regents — has the sense to stand up at a news conference and say that the state test scores are so ridiculously inflated that only a fool would take them seriously, thereby unmasking the mayor, the chancellor and the former state commissioner. State scores are to be scaled down immediately, so that the 68.8 percent English proficiency rate at the start of the news conference becomes a 42.4 proficiency rate by the end of the news conference.

Shael Polakow-Suransky, chief accountability officer for the city, offers the new party line: “We know there has been significant progress, and we know we have a long way to go.” Whether there has been any progress at all during the Bloomberg years is questionable. The city’s fourth-grade English proficiency rate for 2010 is no better than it was in February 2001, nine months before the mayor was first elected.

Mr. Polakow-Suransky says that even if city test scores were inflated, he is not aware of any credible research calling the city’s 64 percent graduation rate into question.

FEBRUARY 2011 The city’s 64 percent graduation rate is called into question. The state announces a new accountability measure: the percentage of high school seniors graduating who are ready for college or a career. By this standard, the graduation rate for New York City in 2009 was 23 percent.

MAY 2011 Embracing the latest new tool in the accountability universe, the governor, state chancellor and education commissioner ramrod a measure through the Board of Regents, mandating that up to 40 percent of teachers’ and principals’ evaluations be based on student test scores.

AUGUST 2011 With new, more rigorous state tests, city scores rise slightly. “We are certainly going in the right direction,” the mayor says.

NOVEMBER 2011 New York is one of two states in the nation to post statistically significant declines on the National Assessment tests. John B. King, the education commissioner, says the state is certainly going in the wrong direction, but has a plan to spur students’ achievement. “The new Common Core Learning Standards will help get them there,” he says.

DEC. 19, 2011 Nearly a quarter of the state’s principals — 1,046 — have signed an online letter protesting the plan to evaluate teachers and principals by test scores. Among the reasons cited is New York’s long tradition of creating tests that have little to do with reality.

Let us note that before Regents Chancellor Tisch finally admitted the scores were inflated, she defended the scores and SED Commissioner Mills as well as state and city officials on test score inflation by saying “They have never, ever, ever exaggerated.”

Uh, huh - except the scores were absurdly inflated and the claims Mills, state and city officials made hailing the scores were very much "exaggerated."

So Merryl Tisch's word is worthless here - as worthless as SED's data on the city opt-outs.

I had an exchange that went like this today with Assemblyman Steve McLaughlin:

@SteveMcNY It would be nice to see the legislature look into SED's manipulation of scores.
— realitybasededucator (@perdidostschool) August 18, 2014

@SteveMcNY The cut scores were rigged by SED last year to plummet, this year rigged to show "modest improvement".
— realitybasededucator (@perdidostschool) August 18, 2014

@SteveMcNY This test-score rigging is meant to show how NYSED's Common Core agenda is going swimmingly.
— realitybasededucator (@perdidostschool) August 18, 2014

As Leonie noted in her blog post, Merryl Tisch declared there would be test score improvement before the tests were given and - lo and behold! - there was improvement.

The previous year, both Tisch and NYSED Commissioner King declared scores would plummet on the new Common Core tests and - lo and behold! - they plummeted.

Now they lowered the cut scores on the tests for 2014, got a slight rise in scores overall and are declaring "modest progress" in the scores, noting that this "modest progress" demonstrates why their Common Core reform agenda must be followed through.

But as we can see in the disputed opt-outs, SED's data is suspect at best, and with the Post reporting that raw score manipulation puts the validity of the state tests in question, I say it is high time we get an independent investigation into the state's testing regime.

We know they pulled a fast one all through the 2000's with the scoring.

We know current Regents Chancellor Tisch was part of that deception back in those days.

We know that SED's data is suspect.

We know that King and Tisch call what the scores are going to be long before the tests are actually given, calling into question the validity of the tests.

And we know that they lowered the cut scores this year to show "modest progress" - again, something that calls into question the validity of the tests.

It is time for an investigation into both the State Education Department and the Board of Regents over these matters because they have a track record of deception previous to this, we see now that their data is suspect at best, and we know they have a political agenda they are pushing.