Generative AI Passes the Legal Ethics Exam in Study by LegalOn Technologies

San Francisco, USA; November 16, 2023 - In a groundbreaking development, researchers at LegalOn Technologies have demonstrated that both OpenAI’s GPT-4 and Anthropic’s Claude 2 can pass the legal ethics exam, a test nearly all US lawyers are required to pass, alongside the bar exam. This milestone underscores the potential for AI to assist lawyers in legal work and demonstrates the increasingly advanced capabilities of large language models applied to law.
‍
Earlier this year, research found that generative AI models such as GPT-4 could surpass law students in passing the Uniform Bar Examination. LegalOn’s study extends this discovery, revealing that these models can also navigate complex rules and fact patterns around professional responsibility. Ethics are part of many professional certifications, but their importance in law is highlighted by the standalone ethics exam that lawyers must pass.

LegalOn’s research tested several leading generative AI models, including OpenAI's GPT-4, GPT-3.5, Anthropic's Claude 2, and Google's PaLM 2 Bison, on their ability to correctly answer questions modeled for the legal ethics exam, known as the Multistate Professional Responsibility Exam (MPRE). GPT-4 performed best, answering 74% of questions correctly, outperforming the average human test-taker by an estimated 6%. GPT-4 and Claude 2 both score above the approximate passing threshold for the MPRE in every state where it is required, a threshold estimated to range between 56-64% depending on the jurisdiction.

“This research advances our understanding of how AI can assist lawyers and helps us assess its current strengths and limitations,” stated Daniel Lewis, US CEO of LegalOn. “We are not suggesting that AI knows right from wrong or that its behavior is guided by moral principles, but these findings do indicate that AI has potential to support ethical decision-making.”

The MPRE consists of 60 multiple choice questions covering a broad range of topic areas, such as client confidentiality, conflicts of interest, and malpractice. For its study, LegalOn tested each large language model against 100 simulated exams composed of questions created by Professor Dru Stevenson, who teaches professional responsibility at South Texas College of Law Houston. Each large language model was tested using a “zero shot” approach, which involves no prior training about legal ethics.

GPT-4 exhibited remarkable proficiency, particularly in areas such as conflicts of interest and client relationships, although the research also highlighted opportunities for enhancement.

"That AI can pass the legal ethics exam marks a turning point not only for legal technology but also for the practice of law," said Professor Stevenson. “The responsibility for ethical decisions will always remain firmly with legal professionals, but this study shows the potential for technology to assist the legal community with consistently meeting high ethical standards.”

“Generative AI applied to legal work continues to surprise and impress. Today it is capable of tasks that, this time last year, seemed years away,” said co-author Gabor Melli, VP of Artificial Intelligence at LegalOn Technologies. "This research indicates that it may be possible to develop AI to assist lawyers with ethical compliance and operate, where relevant, in alignment with lawyers’ professional responsibilities.”

GPT-4’s performance varied by subject area, and there are opportunities for improvement. It performed particularly well in areas such as conflicts of interest and client relationships, and less well on topics such as the safekeeping of funds. These findings indicate that performance may improve with more domain-specific knowledge and lawyer-led training and validation. This research reaffirms LegalOn’s belief in the importance of integrating expert legal content and knowledge with AI systems to build professional-grade tools. Earlier this year, LegalOn launched AI Revise, the first AI contract editor enhanced by expert legal knowledge, enabling legal teams to make precise, context-aware contract revisions with one click.

For a deeper insight into this study and its implications, download a copy of the report here.

About LegalOn Technologies

LegalOn Technologies is the leading AI contract review software for legal teams, serving innovative lawyers and legal professionals at over 4,000 companies and firms globally. The company is backed by leading investors and has raised over $130M. Companies and firms interested in our technology can find more information and sign up for a demo at LegalOnTech.com. LegalOn’s US headquarters are in San Francisco, and its global headquarters are in Tokyo. Follow LegalOn on LinkedIn to stay up to date on the latest news and developments. 

Contact

Corey Longhurst | pr@legalontech.com