Pearson Screws Up State Test Scoring Guides Too

Start lawyering up, folks, because if you get declared "ineffective" based upon these Pearson state tests, you are going to have yourself a hell of a case:

Not only are the state tests confusing for the kids. Now teachers say they can't figure out how to score them.

Long-time principals and teachers say they have never witnessed the level of confusion that has broken out in scoring sessions for the state math and English tests this year. Principals have received several emails with corrections to the scoring guide created by the testing company Pearson. More than 5,000 city teachers have been pulled out of their classes to grade the exams, a job which is scheduled to wrap up Wednesday.

"From what I am hearing, there are huge issues with the scoring and the scoring rubrics from the state, and I have heard this from suburban as well as urban principals," said Elizabeth Phillips, principal at PS 321 in Brooklyn. "What is being reported by teachers who have scored -- who are bound by confidentiality regulations and so are not revealing specifics -- is that the rubrics are extremely confusing and don't align with what experienced teachers believe would make sense."

Some teachers, who also asked to remain anonymous since they had agreed not to reveal details of the exams, said that they believed problems with scoring were as significant as the errors in the exam itself. This year's tests have been riddled with mistakes, including a nonsense story about a talking pineapple.

"I am so disgusted by the afternoon session today," said a math content specialist last week, who is in charge of training other teachers. "The 'expert' from Pearson knew less about the test and rubric than most people in the room. He wasn't able to explain the test items clearly, let alone answer concerns or questions."

"I sat next to a DOE supervisor who seemed to have connections to top people at Pearson, making calls throughout the session," the teacher continued. "She was equally disgusted and made several attempts to manage the chaos in the room."

Other teachers reported widespread confusion about grading English essays. "Scorers differed in their opinions about correct scores for students regularly -- meaning every few minutes at my table and the tables around me," said one teacher. "Supervisors were unable to explain all the reasoning around debated answers, and were just able to say, "This is what the State says." [Schoolbook also reported that teachers are confused about how to grade the tests.]

State Education Department officials said that there would be a statewide audit of the scoring, as there has been in past years.

"While scoring was taking place, SED was in close contact with scoring leaders across the state," said spokesman Tom Dunn.

Technical reports showing the results of the audits will be produced and posted on the Department's web site, he said.

All third through eighth graders took the exams this spring, and the results help to determine where a student gets in middle and high school, if a teacher gets fired and whether a school closes.

Other problems teachers scoring the English essays reported included:

1. It was possible to get a 0 out of 4, even if you comprehended the text, because only certain details were accepted. It was also very possible to get a 2 with no clear understanding of the text if the student copied the correct sentences or paragraphs from the text.

2. Some students had poor handwriting or spelling and were scored down because the reader was not familiar with typical unconventional spelling that elementary students often have due to language issues or developmental issues.

3. The two short answers for one selection asked for virtually the same answers. While you could use slightly different information to answer the question correctly, if you recopied your first answer exactly, you would get the highest score.

4. The guidelines for scoring the 5th and 6th grade essays are the same, even though most schools expect a higher level of performance from 6th graders.

In addition, a listening passage about a kid who loved music asked students to write about how the child in the passage is like and unlike a "typical 6th grader." Teachers debated what would lead to a high score: does a typical 6th graders really like music? Does a typical 6th grader attend after-school? Take the bus? There was not consensus on what details would be considered "meaningful and relevant examples," as dictated by the scoring guide.

Math teachers were also alarmed to see that there was no "reference page" on the 8th grade math exams. In the past, the tests included a page with conversions that many students are not asked to memorize, such as how many pints are in a gallon. The first two of eight questions on the exam demanded conversions. "Students had written down things like, 'I could do this problem, but I can't start it because I don't know the conversion,'" said one educator. "It was kind of heartbreaking. If you can't do two out of the eight problems, that can really break your confidence."

Other teachers described the scoring guide as "vague" or "rigid," and said they believed some student scores depended more on a teacher's understanding of the scoring guide rather than the actual answers children had given.

"There is a great deal of subjectivity to the scoring," said Ms. Phillips. "If this were not so high stakes, it wouldn't matter, but when teachers' jobs depend on a couple of points on a couple of students' exams, this is completely unfair."

A total mess.

And yet, both Governor Cuomo and Regents Chancellor Tisch maintain the teacher evaluation system built upon these state tests is going to be "scientific" and "objective" and the best in the nation.

Sure it will.