Bar exam blunders and the crisis the State Bar continues to ignore

The California State Bar made swift and sweeping changes to the bar exam. In February, those changes resulted in utter disaster. The Committee of Bar Examiners (CBE) and the Board of Trustees (BOT) have listened to hours of public comments detailing the many failures of that exam, including complaints about the Bar's inconsistent and inaccurate communications, the inept electronic administration by Meazure Learning, and the poor quality of the multiple-choice questions. But buried in the hours of meetings following the exam is another story: our State Bar's complete lack of curiosity about whether the brand-new multiple-choice questions they are using - questions authored quickly by a bar preparation company with no experience in writing actual bar exam questions - might be flawed. Because the State Bar's leadership seems unwilling to even consider whether there might be a problem with the quality of the hastily drafted questions and ask appropriate follow-up questions to gain clarity, the July bar exam will also be plagued with issues of fairness and lack of exam integrity. 

When inking its contract with Kaplan in August of 2024, the State Bar promised it would not expand testable areas of law, as mandated by Business and Professions code section 6046.6, which requires two years of notice of a substantial change in the training or preparation required to pass the exam. At the same time, the Bar had to proceed cautiously because the National Conference of Bar Examiners (NCBE), who authored the Multistate-Bar-Exam (MBE) questions used in almost every jurisdiction and previously used in California, had warned Kaplan not to infringe on any of its intellectual property. In asking Kaplan to author questions that achieved both goals, the resulting multiple-choice questions changed the test in two fundamental ways: they added dozens of new rules eligible for testing, and they produced questions that are deeply flawed because of poor drafting and errors in the law. Consequently, the Bar failed to create an exam capable of fairly assessing an applicant's minimum competency to be an attorney.

We'll say it again more plainly: we believe the California State Bar has violated Business and Professions section 6046.6 because the newly created multiple-choice questions have substantially modified the training and preparation required for passage of the exam.

That is a bold statement. The natural next question is: How do we know? And, importantly, why have neither the CBE nor the BOT asked this question publicly despite the many hours of public meetings that have been devoted to the February bar exam debacle?

By the way, here's how we know. First, we looked at NCBE's Multistate Bar Exam (MBE) Subject Matter Outline. That provides an outline of each of the seven subjects tested on the multiple-choice portion of the exam. It details the topics and subtopics eligible for testing. Examinees, law schools, and bar preparation companies use that outline to identify the testable areas of law. We then turned to California's new multiple-choice Content Map, which was re-released Feb. 6, 2025, after undergoing substantial modifications. The Content Map is designed to replace the MBE Subject Matter Outline. We then made a side-by-side comparison of each listed topic and subtopic. Immediately, we noted 55 rules California added to its Content Map. 

For example, the Content Map contains the category of "crimes against a person," which includes the crimes of false imprisonment and mayhem. The MBE Subject Matter Outline lists no such crimes. Thus, alerting us to two potential areas of expanded law. 

Then we analyzed the 1,991 released MBE questions - questions bar applicants use to learn and practice the approximately 3,000 rules of black letter law already eligible for testing on the bar exam - to see if those topics had ever been the subject of a released question. Criminal false imprisonment has never been tested. Mayhem has never been tested, either. Neither have the other 55 rules unilaterally added to the Content Map.

We then asked if it is possible that Kaplan made a mistake and accidentally included this rule on the Content Map? Unfortunately, no. One of Kaplan's 50 released bar multiple-choice practice questions tests the elements of criminal false imprisonment.

By unilaterally adding this topic to CA's Content Map for crimes and testing it, and, of course, in doing the same for all 55 added rules, the testable subjects have been impermissibly expanded without notice. The aggregate impact of adding dozens of rules that have never previously been tested is especially troubling. The rules newly eligible for testing seem to have disproportionate importance since of the 50 released Kaplan multiple-choice questions, approximately 20% test applicants on these 55 newly added rules, with the remainder covering the other 3,000+ rules. Because law schools did not have notice, these rules were likely not covered in the curriculum. These rules would not be covered in a typical commercial bar course because they were not eligible for testing previously. Further, applicants would also not have the opportunity to see these rules covered in their practice questions.   

This prompted us to ask: how does testing bar applicants on rules they did not learn assess their minimum competence to practice law?

Of course, it does not. So, in advance of the BOT's April 2 meeting, we submitted a seven-page letter with a nine-page appendix detailing the expanded areas of law listed on California's Content Map. We also identified many examples of questions containing incorrect statements of law and/or questions with poor or imprecise language in the 50 published Kaplan questions. Was there a single comment or question on this topic?

Nope.

In fact, when faced with repeated public comments made by law professors at CBE and BOT meetings expressing concern about the published Kaplan materials, the members did not ask a single question. At the March 14, April 2, and April 8 CBE and BOT meetings, after reviewing written comments and listening to oral public comments, not a single member of either body asked a single question about whether the questions expanded testable areas of law. 

Zero.

Forget the assertions by the law professors for a moment. How about the statements from examinees? At the April 2 Board of Trustees meeting, multiple February examinees said the multiple-choice questions looked nothing like the questions they practiced with to prepare for the exam. 

Not a single comment or follow-up question.

Instead, someone from the Office of General Counsel delivered a PowerPoint presentation that summarized their psychometrician's conclusion that 175 of the multiple-choice questions gave him a confidence score above what he predicted. Was there a single question about whether there were issues identified with wrong law, as the professors stated they believed there would be? Did anyone ask about whether the exam tested topics beyond the testable scope? 

No and no again.

The PowerPoint also described their question validation process, in which the questions were drafted, reviewed and validated in a few short months. In our public comment, we detailed the NCBE's much more thorough multi-step question validation process, which takes many years to complete. Did a single member of the BOT ask for details about the California Bar's review process and whether it could indeed be flawed? Was there a question about whether the members of the volunteer review team have the necessary subject matter expertise on the questions they reviewed? Whether reviewers were allowed to bring outlines with rules to check black letter law and look up case holdings? Did they ask whether the reviewers had expertise in multiple-choice question formulation? Did they ask whether this validation process they confidently lauded was used to review and validate the 50 published Kaplan questions that were so error-filled they had to be substantially revised and republished in February and yet they are still rife with errors? 

No to all of the above.

In the TV show Ted Lasso, Rupert, the disgruntled former owner of AFC Richmond, challenges Coach Lasso to a high-stakes game of darts. While sealing victory with his final winning three shots, Ted cautions Rupert to "Be curious, not judgmental." Prior to challenging Ted, did Rupert ask whether Ted knew how to play darts? No. Did he ask whether Ted played a lot of darts? No again. Instead, he blindly challenged a better player he had misjudged. 

A little curiosity goes a long way.

Why, at these meetings, does not a single member of the BOT or CBE ask whether a question goes outside testable law? Whether the Business and Professions code is being followed? The State Bar should be the paragon of upholding the law - not covering their ears, closing their eyes, and hiding from it. And when in doubt, they should ask questions. Questions like whether things are being done correctly and how they can be better.

Curiosity and asking questions are not a sign of weakness - they show sincere effort to get things right.