Skip to content

Limitations & Risks

This may be the most important page in this section. Coding agents are powerful tools, but they carry risks that are particularly acute in legal work. Understanding these limitations is not optional — it is a professional obligation.

Hallucinated Citations

This is the single most dangerous failure mode for legal professionals.

AI models, including the one powering Claude Code, can and do fabricate citations. They will invent case names, generate plausible-sounding docket numbers, cite law review articles that do not exist, and reference statutes with incorrect section numbers — all with complete confidence and perfect formatting.

This is not a rare edge case. It happens regularly, and it happens most often precisely when we are relying on the AI for research — when we may not already know the correct answer.

What this means in practice:

  • Never include an AI-generated citation in any document without independently verifying it in a primary source (Westlaw, Lexis, the actual statutory text, the actual journal).
  • Never rely on an AI's description of a case holding without reading the case yourself.
  • Treat every factual claim from an AI the way we would treat a claim from an unverified source: potentially useful as a lead, but not trustworthy until confirmed.

Real consequences

Lawyers have been sanctioned for filing briefs with AI-fabricated citations. In Mata v. Avianca, Inc. (S.D.N.Y. 2023), counsel faced sanctions after submitting a brief containing fictitious case citations generated by ChatGPT. This is not a hypothetical risk. It is a documented professional hazard.

Confidentiality and Client Data

When we use Claude Code — or any cloud-based AI tool — our input is transmitted to the AI provider's servers for processing. This raises serious concerns under professional responsibility rules.

ABA Model Rule 1.6: Confidentiality

Rule 1.6 requires that we not reveal information relating to the representation of a client unless the client gives informed consent. Sending client documents, case details, or privileged communications to a third-party AI service is a disclosure, and we must treat it as one.

Practical considerations:

  • API vs. consumer products. Anthropic's API and Claude Code operate under different data policies than the free consumer chatbot. Under current API terms, Anthropic states that it does not train on user data submitted through the API. However, data is still transmitted to and processed on Anthropic's servers.
  • Anonymization is not foolproof. We might think we can redact client names and identifying details before submitting documents. In practice, legal documents contain numerous identifying details beyond proper names — case numbers, jurisdictions, factual patterns, dates, and transaction amounts that can make re-identification straightforward.
  • Institutional policies. Many law firms, legal aid organizations, and law schools have their own policies regarding AI tool use and client data. We should know and follow our institution's policy in addition to the Rules of Professional Conduct.

What We Can and Cannot Safely Process

Generally Safer Generally Riskier
Publicly available legal documents (published opinions, statutes, regulations) Client communications and privileged materials
Our own drafts and notes that do not contain client-identifying information Documents containing personally identifiable information (PII)
Hypothetical scenarios for teaching purposes Sealed or confidential court filings
Course materials and syllabi Student records (protected under FERPA)
Published scholarship and research Unpublished research involving human subjects

Competence: ABA Model Rule 1.1

Rule 1.1 requires competent representation, which the comments define as including staying abreast of changes in technology relevant to the practice of law. This cuts both ways:

  1. We have an obligation to understand tools we use. If we use a coding agent to assist with legal work, we must understand enough about how it works to supervise its output effectively. "The AI did it" is not a defense for inadequate work product.
  2. We may have an obligation to know these tools exist. As AI tools become standard in legal practice, failing to understand what they can do may itself become a competence issue — not because we must use them, but because we should be able to make an informed decision about when they are and are not appropriate.

The practical standard: We should be able to explain, to a client or a disciplinary board, what the AI tool did, what we verified, and why we are confident in the final work product. If we cannot do that, we are not supervising the tool adequately.

Bias and Accuracy

AI models are trained on large datasets that reflect existing biases in legal systems, scholarship, and society. This can manifest in several ways:

  • Overrepresentation of majority perspectives. The model may treat the dominant legal framework as the only one, underweighting minority viewpoints, alternative legal traditions, or emerging doctrines.
  • Historical bias in legal reasoning. If trained on decades of case law, the model absorbs the biases embedded in that law — including biases around race, gender, socioeconomic status, and other protected characteristics.
  • Jurisdiction blindness. The model may default to federal law or the law of large jurisdictions (New York, California) when the relevant jurisdiction is elsewhere.
  • Recency problems. Training data has a cutoff date. The model may not know about recent statutory amendments, new case law, or regulatory changes.

The mitigation: We treat AI output the way we would treat a first draft from a new research assistant — as a starting point that requires careful review, not as an authoritative source.

Malpractice Risk

Using AI in legal work does not create new duties, but it creates new ways to breach existing ones:

  • Failure to verify. Submitting AI-generated work product without adequate review is the most direct path to a malpractice claim or disciplinary action.
  • Overreliance on AI judgment. The agent may produce a confident analysis that misses a critical issue — an exception to a rule, a recent amendment, a jurisdictional variation. If we accept that analysis without independent evaluation, we bear responsibility for the oversight.
  • Unauthorized practice concerns. If we build AI-powered tools for clients or students to use, we need to consider whether those tools cross the line from information to legal advice.

Data Privacy: API vs. Consumer Products

Not all AI tools handle data the same way:

Policy Area Consumer Chatbot (Free Tier) API / Professional Tier
Training on user data Often yes, unless opted out Typically no
Data retention Varies; often retained for improvement Typically shorter retention or no retention
Data sharing May be shared with third parties for improvement Typically not shared
Compliance certifications Rarely Sometimes (SOC 2, etc.)

Key point: Claude Code, when used through the API or a Pro/Max subscription, operates under Anthropic's commercial terms, which currently state that user data is not used for model training. However, policies change, and we should periodically verify the current terms. The Privacy page provides additional detail on data handling for the tools covered in this guide.

When NOT to Use AI

Some situations call for keeping AI tools out of the workflow entirely:

  • Privileged attorney-client communications that have not been appropriately de-identified (and where de-identification may not be practically achievable).
  • Any task requiring current case law or statutory text without independent verification in a primary legal database.
  • Matters involving sealed records, confidential settlements, or trade secrets where any external transmission creates risk.
  • High-stakes factual determinations where the consequences of an error are severe and the AI's confidence might lead to insufficient independent verification.
  • Work where your institution's policy prohibits AI use. Some courts, agencies, and academic institutions have explicit restrictions.

A Responsible Use Framework

We recommend a four-part framework for any legal professional using coding agents:

1. Verify

Every factual claim, citation, and legal conclusion produced by an AI must be independently verified before it is relied upon. This is non-negotiable. The time saved by using an agent is time we reinvest in verification, not time we pocket.

2. Attribute

When AI tools contribute to our work, transparency is appropriate. Several jurisdictions now require or encourage disclosure of AI use in court filings. Even where not required, honest attribution builds trust and protects us if something goes wrong.

3. Maintain Human Judgment

The agent handles mechanics. We handle judgment. This distinction must be maintained rigorously. If we find ourselves accepting AI output without critical evaluation — because it looks polished, because it is convenient, because we are busy — we have crossed a line.

4. Know the Boundaries

Before using an AI tool on any matter, we should ask: What are the confidentiality implications? What are the accuracy requirements? What would happen if the AI got this wrong? If the answers give us pause, the tool stays in the drawer for that task.

The Bottom Line

Coding agents are tools — powerful ones, but tools nonetheless. Like Westlaw, like email, like the photocopier before them, they create new capabilities and new risks. The lawyers who thrive with these tools will be the ones who understand both sides of that equation and maintain the professional judgment that no AI can replace.