Saturday, August 15, 2015

The Irrationality Of New York State's VAM Scores

Carol Burris has a piece in the Washington Post about Sheri Lederman's case against NYSED over her VAM score, which went from 14 out of 20 one year to 1 out of 20 the next despite her students' test scores remaining stable.

Burris points out Lederman is not the only irrational VAM score around the state:

Lederman was not the only teacher in the school to get a poor score. In 2014, 21 percent of the staff at E.M. Baker School received a score of “ineffective,” 21 percent “developing” and 57 percent were “effective.”  Just the year before, not one teacher received an “ineffective” score.

The irrationality was not limited to the teachers of Sheri Lederman’s school, one of the highest performing elementary schools in the state. In 2014, 44 percent of the teachers of the Fox Meadow School in Scarsdale received growth scores that said they were not “effective” teachers with 22 percent rated “ineffective.”  Yet 61 percent of the school’s students were proficient in English Language Arts, and 75 percent were proficient in math—more that double the state’s proficiency rate.   Similar results were found at the high-achieving Harbour Hill School in Roslyn, where 36 percent of its teachers received growth scores that labeled them “ineffective.”

The Lincoln School in Rochester, is a school designated as a priority/failing school by the state. Its proficiency rate was less than 3 percent. In 2014, 100 percent of its teachers received “effective” state scores, with 7 percent being rated “highly effective.”  At another school facing receivership, The Martin Luther King Jr School in Utica, New York, 60 percent of the teachers received “effective” VAM scores and 40 percent were given VAM scores of “highly effective .”

NYSED never showed cause for why Lederman received the 1 out of 20 on her VAM score the year after she received 14 out of 20.  They instead tried to have the Lederman case dismissed twice (here and here), arguing there was no harm to Lederman because she was rated "effective" overall despite the "ineffective" VAM score.

However the case did go to trial and the judge didn't sound like he was buying the "Because We Said So" bell curve argument NYSED showed up with for why Lederman was deemed "ineffective" on her VAM:

The more Ms. Galligan tried to defend the bell curve of growth scores as science, the more the judge pushed back with common sense. It was clear that he did his homework. He understood that the New York State Education Department’s VAM system artificially set the percentage of “ineffective” teachers at 7 percent. That arbitrary decision clearly troubled him. “Doesn’t the bell curve make it subjective? There has to be failures,” he asked.
...
“Did her students learn nothing?” Justice McDonough asked. “How could it be that she went from 14 out of 20 points to 1 out of 20 points in one year?” He noted that the students’ scores were quite good and not that different from the year before.

Back behind the bell curve Ms. Galligan ran. As she tried to explain once again, the judge said, “Therein lies the imprecise nature of this measure.”

Burris writes that a decision is expected in two to three months - the Lederman's have argued that not only Sheri Lederman's rating should be overturned but the court should  “declare the measure an abuse of discretion” because “the State Education Department does not get a pass on unreasonable and irrational actions."

Of course the evaluation system that the Lerdermans have out on trial has been made that much worse by Governor Cuomo and the Legislature this year, with test scores now increased to about 50% of a teacher's evaluation rating, so the "irrationality" of the state's APPR VAM matters greatly and a decision by the judge to declare the measure "an abuse of discretion" will have serious consequences for the governor, the Legislature, the Board of Regents, NYSED and teachers and schools all across the state.

11 comments:

  1. What a great post, RBE. As always, I keep my fingers crossed when reforms look like they might take a hit. Most of the time, poor decisions stand, (and of course are seen as a victory by our union).

    This case would be a good one to win. Let's hope the judge gets no calls from the Cuomo residence in the middle of the night.

    ReplyDelete
    Replies
    1. Cuomo and his asshole minions will find a way and Mulgrew will proclaim IF YOU DO NOT LIKE YOUR VAM I WILL PUNCH YOU IN THE FACE!

      I hope I'm wrong. However, this is NY where corruption reigns supreme.

      Justice may be blind but can be paid off nicely.

      Delete
  2. There are way more things wrong with VAM.
    It is good news though that the judge is onto them.
    The VAM is also not allowing the teacher to earn a good rating based on their own scores. They are rated based on how other teachers scored.
    For that reason you can't know in advance what it is you have to do. The criteria that your score is based on is shifting and not concrete.
    VAM is also not transparent so that a teacher can understand in advance how they will be rated as is required by law.
    VAM is also purported to be a growth model, however, it is not capable of showing the kind of vertical growth that needs to be measured from year to year.
    And let's not forget that the Cut Scores are decided AFTER the kids take the test. And each year the scale changes. Not very scientific if you ask me.
    It sounds like the Ledermans case is going well. Finally the charade is being exposed publicly for what it really is: a convoluted SCAM.

    ReplyDelete
    Replies
    1. You made great points, I built another post around your comments. VAM is harmful on so many levels, not just the ones under attack in the Lederman case:

      http://perdidostreetschool.blogspot.com/2015/08/more-things-wrong-with-vam.html

      Delete
  3. This is indeed a great post. I am an assistant principal in the Bronx on the HS level. Unfortunately, we cannot attract teachers to our school due to our graduation rate, deriving from low scores (international school). I wanted to recruit an Earth Science teacher. She told me she'd never work at my school because 50% of her overall rating would basically give her a Developing as the HIGHEST rating. You see, the schools are taking a hit too with recruitment. Great teachers go to specific schools. You want to talk about segregation? Here it is at its finest. There are other HS in the campus that are premium. We are in the same campus but can't offer our kids a vibrant Science education, but the other school are. This is terrible and should additionally be reported from an administrators viewpoint.

    ReplyDelete
    Replies
    1. I built a post around your comment because it's so important to get it out there. The very system that reformers claim will help the most vulnerable students actually harms them:

      http://perdidostreetschool.blogspot.com/2015/08/how-andrew-cuomos-reformy-teacher.html

      Thanks for leaving the comment. It's important to get feedback from administrators on the system too - it shows a different perspective but the same end result: APPR is damaging.

      Delete
  4. Let's hope Justice McDonough is capable of disregarding the politics of the situation. Fingers crossed, but even if his decision declares the use of VAM irrational and an abuse of discretion (which it clearly should), NYS can appeal the decision to the Appellate Division and, I believe, continue applying VAM in the interim. Next there's the Court of Appeals. I hate to say it, but Foundation Aid came about after 10 years of litigation in which the Courts of Appeals found NYS to be underfunding public schools. The Governor and Legislature froze the formula in year two (?!), and well, we all know how well things went from there. The politicians need to be convinced to do the right thing. How can that happen until they are compelled to subject themselves to campaign finance reform, and therefore, lose their reliance on wealthy donors, otherwise known as private policy makers? It always comes back to the money (that the masses are not seeing) . . .

    ReplyDelete
    Replies
    1. No one NEEDS to rely on wealthy donors. This is done intentionally.

      Delete
    2. There will be political pressure put onto the heavy hearts in Albany if the judge rules against NYSED in the Lederman case, so even though there will be appeals, a lower level court win will have some political consequences. I agree that this will likely be tied up for a while, but I bet Cuomo is a lot less likely to publicly defend it if a judge tears it apart in a decision.

      Delete
  5. There's no doubt that this will be appealed. What Im hoping is that the Lederman's can request a Stay. With a Stay in place, the State would have to suspend VAM until the appeal is finished.
    Since peoples livelihoods are at stake, and since children education is being significantly affected, a judge might see the rationale for this.

    ReplyDelete
    Replies
    1. See comment above - if the ruling pulls apart the irrationality of the VAM, Cuomo will not have the cover to say that his system help kids. It will tear off his political cover for APPR and expose him as doing it to harm teachers rather than help kids.

      Delete