Generative AI Passes the Legal Ethics Exam

In an exciting development within the field of legal technology, our team at LegalOn Technologies has conducted a study with significant implications for the future of AI in law. We found that leading generative AI models, including OpenAI's GPT-4 and Anthropic's Claude 2, have the capability to pass the legal ethics exam, a requirement for lawyers in the United States alongside the bar exam. This achievement highlights the potential of AI in assisting with legal work and showcases the advanced capabilities of large language models in the field of law.

About the Study

In our research, we tested several generative AI models, including GPT-4, GPT-3.5, Anthropic's Claude 2, and Google's PaLM 2 Bison, focusing on their ability to answer questions modeled for the Multistate Professional Responsibility Exam (MPRE). The results were noteworthy: GPT-4 answered 74% of questions correctly, outperforming the average human test-taker by an estimated 6%. Both GPT-4 and Claude 2 scored above the passing threshold for the MPRE in every state where it is required.

Insights and Implications

Daniel Lewis, US CEO of LegalOn, emphasizes that while this research doesn't suggest AI knows right from wrong, it indicates AI's potential in supporting ethical decision-making in law. He stated that “this research advances our understanding of how AI can assist lawyers and helps us assess its current strengths and limitations. We are not suggesting that AI knows right from wrong or that its behavior is guided by moral principles, but these findings do indicate that AI has potential to support ethical decision-making.”

The MPRE covers a broad range of topics such as client confidentiality, conflicts of interest, and malpractice. For the study, each AI model was tested against 100 simulated exams composed of questions created by Professor Dru Stevenson, an expert in professional responsibility. The models were tested using a “zero shot” approach, which involves no prior training about legal ethics.

Professor Stevenson regards this achievement as a turning point for both legal technology and the practice of law. He asserts that “the responsibility for ethical decisions will always remain firmly with legal professionals, but this study shows the potential for technology to assist the legal community with consistently meeting high ethical standards.”

Looking Forward

The performance of AI models like GPT-4 varied by subject area, indicating areas for improvement and potential growth with more domain-specific knowledge and lawyer-led training. This aligns with LegalOn’s approach of integrating expert legal content and AI systems to build professional-grade tools.

For more detailed insights into this study and its implications, we invite you to download the full report.

‍