Artificial intelligence can summarise documents, assist with research and accelerate legal drafting.
It can also invent facts.
A recent case before the New Mexico Supreme Court demonstrates just how serious the consequences can become when lawyers fail to distinguish between the two.
A lawyer was fined US$5,000 after filing an appellate brief in a murder case containing fabricated material generated through ChatGPT, including invented witnesses and purported police testimony that did not appear in the trial record.
The lawyer reportedly used ChatGPT to help summarise the proceedings but failed to verify the resulting content before incorporating it into a court filing.
The court found the conduct serious enough to impose a financial sanction, refer the matter to the disciplinary authorities and require the appeal to continue under different representation.
The case is another warning about AI hallucinations in law, but it raises an even more fundamental issue:
A lawyer remains responsible for every factual assertion submitted to a court, regardless of how that assertion was produced.
The Problem Was Not Simply Using ChatGPT
The case should not be reduced to the proposition that lawyers should not use artificial intelligence.
That would miss the point.
AI tools can be useful for:
- organising large volumes of information;
- producing initial summaries;
- identifying issues for further research;
- assisting with drafting;
- comparing documents; and
- improving workflow efficiency.
The professional risk arises when the output is treated as authoritative without independent verification.
In this case, the alleged problem was not that ChatGPT had been used.
It was that AI-generated factual material was incorporated into a court filing without being checked against the actual record.
That distinction matters.
Fabricated Facts Can Be More Serious Than Fake Citations
Much attention has already been given to lawyers filing briefs containing fictitious cases or authorities generated by AI.
This case goes further.
The reported errors concerned not merely legal authorities, but the evidence itself.
Invented witnesses and fabricated testimony can fundamentally alter how a court understands a case.
A false citation may misstate the law.
A fabricated factual assertion may misrepresent what actually happened at trial.
In litigation, that can affect:
- credibility;
- the interpretation of evidence;
- appellate review;
- the client’s position;
- and confidence in the administration of justice.
That is why factual verification should be treated as just as important as checking legal authorities.
AI Cannot Review a Record the Way a Lawyer Must
A lawyer reviewing a trial record is not simply looking for words to summarise.
Professional judgment is required.
The lawyer must understand:
- which evidence was actually admitted;
- who gave particular testimony;
- whether evidence was challenged;
- what objections were made;
- what findings were reached;
- and which parts of the record are legally relevant to the appeal.
An AI system may generate a coherent narrative that sounds convincing without reliably distinguishing between material actually contained in the record and content it has inferred or invented.
Fluent language can therefore create a dangerous illusion of accuracy.
The more polished the output sounds, the easier it may be to overlook the need for verification.
The Lawyer’s Duty Does Not Transfer to the Tool
Professional responsibility cannot be delegated to software.
A lawyer signing or filing a submission remains accountable for its contents.
That principle applies whether the material was produced by:
- a junior lawyer;
- a paralegal;
- an external researcher;
- automated software; or
- a generative AI system.
Technology may assist in producing the work.
It does not become professionally responsible for the result.
A lawyer cannot therefore rely on the fact that an AI tool generated an incorrect statement as an answer to why that statement appeared in a court filing.
The obligation to verify remains human.
The Client Also Bears the Consequences
AI mistakes in litigation do not affect only the lawyer.
They can damage the client’s case.
A court confronted with fabricated material may begin questioning the reliability of the wider submission.
Time may be lost correcting the record.
Additional costs may arise.
The client’s appeal or claim may be delayed.
In serious cases, new representation may become necessary.
This is why AI governance within law firms should not be treated solely as an internal technology issue.
It is also a client-protection issue.
Confidentiality Creates a Separate Risk
Accuracy is not the only concern.
Lawyers using generative AI must also consider what information is being entered into the system.
Legal work frequently involves:
- confidential client communications;
- commercially sensitive material;
- witness evidence;
- medical records;
- personal information;
- privileged documents; and
- litigation strategy.
Uploading such material into an AI platform without understanding how the system handles, stores or processes that information can create separate confidentiality and data-protection risks.
A law firm’s AI policy should therefore deal with both what comes out of the system and what goes into it.
What Should Law Firms Be Doing?
The practical response is not necessarily to prohibit AI entirely.
It is to control its use.
A sensible legal AI framework should require:
Independent verification
Every authority, quotation, factual assertion and reference generated through AI should be checked against the original source.
Clear limits on sensitive information
Staff should understand what client or case information may and may not be uploaded into external AI tools.
Human review before filing
No court document should be filed simply because AI-generated drafting appears complete or persuasive.
Training
Lawyers and support staff should understand that generative AI can produce confident but false output.
Responsibility
The lawyer supervising the matter should remain clearly accountable for the final work product.
These controls are not complicated.
The difficulty lies in ensuring they are followed consistently.
The UAE Relevance
The New Mexico ruling does not govern legal practice in the UAE.
Its underlying lesson, however, is directly relevant.
Lawyers practising in any jurisdiction are expected to act with professional care, protect client interests and avoid misleading courts.
As AI tools become more common in legal research and drafting, UAE firms will increasingly need internal rules governing:
- verification;
- confidentiality;
- supervision;
- appropriate use;
- and accountability.
The professional standard should not depend on whether a mistake originated from a person or an algorithm.
What matters is that the lawyer chose to rely on it.
AI Is an Assistant, Not an Authority
Generative AI can be remarkably useful when used within proper controls.
It can reduce time spent on repetitive work and help lawyers process information more efficiently.
But the technology has an important limitation: it is designed to generate plausible language, not to guarantee truth.
In legal work, plausibility is not enough.
A witness either testified or did not.
A document either exists or does not.
A case either says something or it does not.
Those questions require verification against authoritative material.
Conclusion
The New Mexico case is not simply another story about an AI hallucination.
It is a reminder that technology does not dilute professional responsibility.
A lawyer may use AI to assist with research, summarisation or drafting.
But once that material is placed before a court, responsibility for its accuracy belongs to the lawyer.
The central lesson is straightforward:
Using AI is not the professional failure. Filing unverified AI output can be.
Al Kabban & Associates
For businesses seeking guidance, Al Kabban & Associates, with over 30 years of experience in UAE law and recognition by Legal 500, stands ready to help corporations build resilience against legal risks while ensuring compliance with local and international standards. For more information or to schedule a consultation, contact us at +971 4 453 9090 or visit www.alkabban.com. You can also follow us on social media for more updates on everything law related in the UAE: @Alkabban_Law
ALSO READ –
