State Bar used AI to write exam questions, suggests lower pass score

The California State Bar used artificial intelligence to help write some of the questions on the February bar exam. The disclosure came on Monday night when the bar announced it was lowering the score needed to pass after a variety of problems plagued the most recent exam.

"Many examinees reported after taking the exam, without disclosing the content of the questions, that they felt there were a lot of typos, and the language felt strange," Mary Basick, assistant dean for academic skills at UC Irvine School of Law, said when reached on Tuesday. "Many suspected the questions were AI generated. Honestly, I defended the bar. I thought there's no way they would do that."

On Monday evening, the bar issued a news release stating that the Committee of Bar of Examiners met on April 18 and recommended "That the raw passing score for the exam be set at 534. This represents two standard errors of measurement below the psychometrician-recommended raw passing score of 560." This was "in recognition of the reported experiences of examinees, performance data, and the challenges associated with fairly and accurately categorizing the level of disruption experienced by individual examinees," the release stated.

"After thoughtful deliberation and hearing directly from many applicants, we believe our recommendations to the Supreme Court mark the beginning of a fair and meaningful path forward for all February bar exam test takers," Board Chair Alex Chan said in the news release.

The State Bar did not say when the California Supreme Court might take up the matter.

According to its website, the bar will release the exam results on May 2. It was not immediately clear how many additional test takers will pass because of the change, but it could easily be hundreds, based on past exam results.

But some academics and test takers focused on a different paragraph near the end of the news release: "Also discussed during the CBE meeting was the performance of the different types of questions on the bar exam, including the multiple-choice questions (MCQs) and their sources. A majority of the MCQs were developed by Kaplan, a subset of questions came from the First-Year Law Students' Exam, and some questions were developed by ACS Ventures, the State Bar's independent psychometrician. The ACS questions were developed with the assistance of AI and subsequently reviewed by content validation panels and a subject matter expert in advance of the exam."

There are several problems with this admission, Basick said.

"They were all supposed to be developed by Kaplan," she said.

She added that her "holy crap" moment came when she saw that questions were developed by ACS Ventures. Many of the problematic questions, and the ones she suspects were written using AI, were in the criminal justice portion of the exam, Basick said, and appeared to have been written by "non-lawyers."

"The questions were developed with the assistance of AI and subsequently reviewed by content validation panels," Basick said. "I would like to know who was on those."

"Who is responsible for this decision?" she added. "Because it appears that the Board of Trustees and the Committee of Bar Examiners and the Supreme Court did not approve this."

The board is scheduled to meet again on May 5, three days after the release of the exam results. According to the Monday news release, they will take up "non-scoring related remedies for February bar exam takers, including provisional licensure, a supervised practice pathway to licensure, and special admission status for attorneys licensed in other states."

A State Bar spokesperson issued a statement late Tuesday that read: "The decision to use ACS Ventures to draft some of the questions using AI was made by staff within the Admissions Division and not clearly communicated to State Bar leadership. This was a breakdown, and structural changes have been made within Admissions to address it, with a new Chief-level role overseeing Admissions directly reporting to the Executive Director, a new team structure to strengthen accountability and effectiveness, and other personnel changes."

This story was originally published on the Daily Journal