Bar exam committee latest to face public wrath of botched test

Both test takers and law professors lambasted Kaplan Test Prep for adding subject sections to the new test without going through the proper notice and process. Some commenters also said some of the multiple-choice questions had two correct answers, while others had none.

Those were among the issues raised by a group of law school faculty in a letter to Committee of Bar Examiners earlier this month. At least two signers of that letter spoke at Friday's meeting: Katie Moran, co-director of the Academic and Bar Exam Success program at the University of San Francisco School of Law, and Mary Basick, assistant dean for academic skills at UC Irvine Law.

"Those of us in academia were very concerned because it was not enough time to properly vet the questions," Basick said when reached by phone during a break in the all-day meeting. "Typically, the questions, particularly the multiple-choice questions, they vet those over several years of testing. To create a whole new bank of questions within couple of months was wildly ambitious."

It also may have been illegal. Basick said that the California Business & Professions Code puts notice and process requirements on the bar that may have been violated. She later said the meeting concluded "without addressing the significant concerns raised about the quality of the Kaplan questions."

"They have to give two years' notice to the law schools if they make a substantial change in the exam because the law schools have to have time to readjust their curriculum in order to properly prepare their students," she said. "We were all promised that it would not be a substantial deviation from the existing multiple choice question bank was which used to be provided by the MBE," the Multistate Bar Exam.

"Our questions were developed by a robust and experienced team," said Kaplan spokesman Russell Schaffer in an email. "We are pleased that the State Bar of California has confirmed that the exam questions are valid and reliable, reflecting the rigorous process we undertook."

However, according to the letter, the faculty and student guides Kaplan released last fall "unilaterally added at least 34 new areas that are now eligible for testing." These issues, with new sections and individual sample questions, persisted when Kaplan issued a revised student guide on Feb. 6, less than three weeks before the exam.

Committee Chair Alex H. Chan began the meeting with an extended apology to test takers and discussed some of the steps being taken to make amends. He also repeatedly adjusted the meeting schedule so all public commenters could be heard.

But just like during a State Bar Board of Trustees meeting the prior week, dozens of frustrated test takers and law school instructors lambasted the members for hours. These comments hit several other themes mentioned in the earlier meeting, including unnecessary interruptions by proctors, disruptions by upset students with technical issues, and a lack of accommodation for test takers with disabilities. Several also said they were "blindsided by new topics," or encountered questions with factual and spelling errors.

During the afternoon, Chad Buckendahl, founder of the test consulting company ACS Ventures, gave an extended presentation on the questions in the exam. He said that the test questions performed well under some traditional criteria for standardized testing. Buckendahl presented statistics suggesting the exam had a good mix of hard, medium and easy questions. He also said there was a good correlation between how well students did on each question and their overall score.

Donna S. Hershkowitz, chief of mission advancement and accountability at the bar, spoke about some of the remedial measures that had been suggested by disgruntled test takers. These include offering provisional licenses or changing scores, for instance ensuring the number of passing scores reaches the typical 35% number for a February bar exam.

"I think these are things that the Supreme Court has talked about before," Hershkowitz said. "Obviously, the committee can make whatever recommendation the committee deems appropriate. We just wanted to make it clear that these are things that the committee itself does not have the power to do."

This story was originally published on the Daily Journal.