Two stories from over the weekend show that the statistics and data issued by NYSED Commissioner John King and his merry men and women in reform at SED cannot be trusted.
First,
as I posted yesterday, the NY Post covered
how NYSED lowered the cut score levels on this year's state Common Core exams, an act which resulted in slightly higher test scores:
State officials touted increases in scores on tough Common Core exams
this year but failed to reveal that they had lowered the number of
right answers needed to pass half the exams.
The state Education Department dropped the number of raw points
needed to hit proficiency levels in six of the 12 English and math exams
given to students in grades 3 to 8, officials acknowledged.
“The reason that occurs is because the tests are slightly harder,” Deputy Education Commissioner Ken Wagner told The Post.
Student scores plunged on last year’s statewide 3-8 tests — the first
based on the new Common Core standards. Before the 2013 exams, a panel
of 95 educators decided how many points, or correct answers, students
had to get to demonstrate proficiency.
But the point cutoffs were tweaked after this year’s tests. The state
and its testing vendor, Pearson, found six tests were harder and four
easier this year than in 2013, Wagner said.
They did so by comparing how students performed on “anchor” test
questions — identical items used in both 2013 and 2014. A report on the
scoring process will be released in December or January, Wagner said.
The changes raise questions about the validity of the results.
“The information given out about the test questions does not provide a
complete picture, making it hard to judge how much progress students
made last year,” said Fred Smith, a former testing analyst for city
schools.
Score manipulation has erupted in scandal before. Between 2006 and
2009, the state reduced the number of raw points students needed to
pass. Then-state Education Commissioner Richard Mills insisted the
questions got harder, justifying the lower passing scores. But experts
found the test items got easier, inflating scores hailed by then-Mayor
Mike Bloomberg, among others, as proof of great progress.
In addition to the Post report on the lowered cut scores,
Stephen Rex Brown at the Daily News reported that NYSED couldn't account for a whole bunch of kids they said opted out of the state tests this year in New York City that the city said didn't:
State Education officials were scrambling to determine Friday why test
data appeared to show more than 20,000 city students did not take math
and English exams.
The perplexing numbers, which the city disputed, revealed 26,949 kids
were no-shows for state math tests and 22,656 skipped the English
Language Arts exam. The figures were more than triple the previous
year’s numbers. State officials suspect there was an error in the way a
large group of city students were coded in the state database of third-
through eighth-graders who took the tests.
The city Education Department said only 1,925 students formally opted
out of the exam — still double the estimate from critics of the April
tests.
Leonie Haimson at NYC Public School Parents blog picked up that story, noting that while she would love to believe that the number of students opting out of the state exams in the city was as high as NYSED said it was, she didn't think this was so:
As much as I would have liked to believe the opt out figures were this high, I expressed skepticism to Stephen– and explained that I thought the numbers of students opting out had been far higher on Long Island and Westchester than in NYC. In the suburbs, in general, parents are more organized, enjoy well-funded public schools with high college-going and graduation rates, and have erupted in justified incredulity when the state tried to convince them their schools were failing and their kids were not “college and career ready.”
Leonie goes on to note that SED's data should not be considered reliable:
My response to all this: with such erratic and unreliable information, how can anyone trust any of the test score data from NYSED?
...
Before the new Common Core tests, we had ten years of state test score inflation in NY that was obvious to anyone paying close attention, but year after year was ignored by the powers that be, because it was politically convenient. Each year Mayor Bloomberg and Chancellor Joel Klein, sometimes accompanied by Randi Weingarten, would ritualistically bow down to the supposedly infallible test score gods and celebrate the results as showing that their reforms were working. And then the entire imaginary edifice came tumbling down in 2010, when the educrats finally admitted that an enormous test score inflation had occurred, somehow without their knowledge and complicity.
It is too early to assume that the small rise in test scores this year were due to similar manipulations , but a decade of experience should teach us to be open to the possibility. Merryl Tisch predicted that more kids would pass this year – and they did. In any event, we have overwhelming evidence from teachers and principals that the tests were poor quality and a lousy judge of real learning.
The state’s release of data showing thousands of opt outs in NYC is just one more piece of evidence showing how skeptical everyone should be about any data our government officials supply.
The lowered cut scores that allowed SED to claim "progress" on their Common Core agenda and the disputed opt-out's here in the city are two examples for why there needs to be scrutiny into NYSED's testing operations from an outside entity unaligned with the Board of Regents, the State Education Department or the "non-profit" education reform groups that bolster both the Regents and SED (i.e., the Gates Foundation, the Broad Foundation, etc.)
As Leonie notes there has been test score manipulation and inflation in the state before and it happened when our current Regents Chancellor Merryl Tisch was on the Board of Regents.
In fact, Tisch defended former SED Commissioner Mills over the test score inflation, claiming there was none, when it was quite clear there was.
Michael Winerip, the former NY Times columnist,
wrote up a timeline of test score inflation here in New York State that is worth revisiting at this time:
In the last decade, we have emerged from the Education Stone Age. No
longer must we rely on primitive tools like teachers and principals to
assess children’s academic progress. Thanks to the best education minds
in Washington, Albany and Lower Manhattan, we now have finely calibrated
state tests aligned with the highest academic standards. What follows
is a look back at New York’s long march to a new age of accountability.
DECEMBER 2002 The state’s education commissioner, Richard P. Mills, reports to the state Regents:
“Students are learning more than ever. Student achievement has improved
in relation to the standards over recent years and continues to do so.”
JANUARY 2003 New York becomes one of the first five states to have its testing system approved by federal officials under the new No Child Left Behind law. The Princeton Review rates New York’s assessment program No. 1 in the country.
SPRING 2003 Teachers from around New York complain that
the state’s scoring of newly developed high school tests is out of
whack, with biology and earth science tests being too easy and the
physics test too hard. The state Council of School Superintendents finds the physics scores so unreliable, it sends a letter to colleges for the first time in its history urging them to disregard the test result. Dr. Mills does not flinch, calling the tests “statistically sound” and “in accordance with nationally accepted standards.”
JUNE 2003 Scores on the state algebra test are so
poorly calibrated that 70 percent of seniors fail. After a statewide
outcry, officials agree to throw out the results. The Princeton Review
says that ranking New York first was a mistake. “We’re going to have to
come up with a fiasco index for a state like New York that messes up a
lot of people’s lives,” a spokesman says.
OCTOBER 2003 A special panel appointed to investigate
the state math fiasco concludes that the test “can’t accurately predict
performance,” was created “on the cheap” and was full of exam questions
that were “poorly worded” and “confusing.”
DECEMBER 2003 The director of state testing resigns. It was his idea to leave, a spokesman says.
MAY 2004 For the fourth year in a row, scores have risen on elementary and middle school state reading and math tests. Dr. Mills urges the Regents:
“Look at the data that shows steadily rising achievement of the
standards in school districts of all wealth and categories. More
children are learning more now than ever before.”
FEBRUARY 2005 Dr. Mills rebukes those who question whether state scores are inflated. “The exams are not the problem,”
he said in a report to the Regents. “It’s past time to turn from
obsessive criticism of the exam and solve the real problems — the
students who are not educated to the standards.”
SPRING 2005 New York City fourth graders make record
gains on the state English test, with 59 percent scoring as proficient,
compared with 49 percent the year before. “Amazing results” that “should
put a smile on the face of everybody in the city,” says Mayor Michael R. Bloomberg, who happily recites the numbers on his way to re-election.
FALL 2005 The federal tests (the National Assessment of Educational Progress),
which are considered more rigorous than the state tests, show a drop in
New York City reading scores. On the eighth-grade test, 19 percent are
proficient in 2005, compared with 22 percent in 2003. Asked if city and
state officials had hyped the state test results, Merryl H. Tisch, a Regent, says, “They have never, ever, ever exaggerated.”
SEPTEMBER 2007 New York’s national assessment test
results are again dismal; eighth-grade reading scores are lower than
they were in 1998.
DECEMBER 2007 In his report to the Regents,
Dr. Mills notes, “A rich, scholarly literature has challenged NAEP
validity since the early 1990s.” He announces a plan to develop the
first new state learning standards since 1996, to further spur academic
excellence.
JUNE 2008 Newly released state test scores show another
record year for New York children. Math scores for grades three through
eight indicate that 80.7 percent are proficient, up from 72.7 in 2007.
“Can we trust these results?” Dr. Mills asks. “Yes, we can. New York’s
testing system, including grades three through eight tests, passed a
rigorous peer review last year by the U.S. Department of Education.
State Education Department assessment experts commission independent
parallel analyses to double- and sometimes triple-check the work of our
test vendor.”
JUNE 2009 In the previous decade, New York students’
average SAT verbal score has dropped to 484 from 494; the math SAT score
has dropped to 499 from 506. The national assessment’s fourth-grade
reading scores have been stagnant for four years, and the eighth-grade
scores are their lowest in a decade.
But somehow, state test scores again soar to record levels. In New York
City, 81 percent of students are deemed proficient in math, and 68.8
percent are proficient in English. “This is a big victory for the city,”
the schools chancellor, Joel I. Klein,
says, “and we should bask in it.” In November the mayor is elected to a
third term, again riding the coattails of sweet city scores.
JULY 2010 Finally someone — Dr. Tisch, the chancellor
of the Board of Regents — has the sense to stand up at a news conference
and say that the state test scores are so ridiculously inflated that
only a fool would take them seriously, thereby unmasking the mayor, the
chancellor and the former state commissioner. State scores are to be scaled down
immediately, so that the 68.8 percent English proficiency rate at the
start of the news conference becomes a 42.4 proficiency rate by the end
of the news conference.
Shael Polakow-Suransky,
chief accountability officer for the city, offers the new party line:
“We know there has been significant progress, and we know we have a long
way to go.” Whether there has been any progress at all during the
Bloomberg years is questionable. The city’s fourth-grade English
proficiency rate for 2010 is no better than it was in February 2001,
nine months before the mayor was first elected.
Mr. Polakow-Suransky says that even if city test scores were inflated,
he is not aware of any credible research calling the city’s 64 percent
graduation rate into question.
FEBRUARY 2011 The city’s 64 percent graduation rate is called into question.
The state announces a new accountability measure: the percentage of
high school seniors graduating who are ready for college or a career. By
this standard, the graduation rate for New York City in 2009 was 23
percent.
MAY 2011 Embracing the latest new tool in the
accountability universe, the governor, state chancellor and education
commissioner ramrod a measure through the Board of Regents, mandating
that up to 40 percent of teachers’ and principals’ evaluations be based on student test scores.
AUGUST 2011 With new, more rigorous state tests, city
scores rise slightly. “We are certainly going in the right direction,”
the mayor says.
NOVEMBER 2011 New York is one of two states in the nation to post statistically significant declines on the National Assessment tests.
John B. King, the education commissioner, says the state is certainly
going in the wrong direction, but has a plan to spur students’
achievement. “The new Common Core Learning Standards will help get them
there,” he says.
DEC. 19, 2011 Nearly a quarter of the state’s
principals — 1,046 — have signed an online letter protesting the plan to
evaluate teachers and principals by test scores. Among the reasons
cited is New York’s long tradition of creating tests that have little to
do with reality.
Let us note that before Regents Chancellor Tisch finally admitted the scores were inflated, she defended the scores and SED Commissioner Mills as well as state and city officials on test score inflation by saying “They have never, ever, ever exaggerated.”
Uh, huh - except the scores were absurdly inflated and the claims Mills, state and city officials made hailing the scores were very much "exaggerated."
So Merryl Tisch's word is worthless here - as worthless as SED's data on the city opt-outs.
I had an exchange that went like this today with Assemblyman Steve McLaughlin:
As Leonie noted in her blog post, Merryl Tisch declared there would be test score improvement before the tests were given and - lo and behold! - there was improvement.
The previous year, both Tisch and NYSED Commissioner King declared scores would plummet on the new Common Core tests and - lo and behold! - they plummeted.
Now they lowered the cut scores on the tests for 2014, got a slight rise in scores overall and are declaring "modest progress" in the scores, noting that this "modest progress" demonstrates why their Common Core reform agenda must be followed through.
But as we can see in the disputed opt-outs, SED's data is suspect at best, and with the Post reporting that raw score manipulation puts the validity of the state tests in question, I say it is high time we get an independent investigation into the state's testing regime.
We know they pulled a fast one all through the 2000's with the scoring.
We know current Regents Chancellor Tisch was part of that deception back in those days.
We know that SED's data is suspect.
We know that King and Tisch call what the scores are going to be long before the tests are actually given, calling into question the validity of the tests.
And we know that they lowered the cut scores this year to show "modest progress" - again, something that calls into question the validity of the tests.
It is time for an investigation into both the State Education Department and the Board of Regents over these matters because they have a track record of deception previous to this, we see now that their data is suspect at best, and we know they have a political agenda they are pushing.
