top of page

When Lawyers Feed the Machine: Why ChatGPT Is Quietly Becoming the Biggest Leak in Legal History

When Lawyers Feed the Machine: Why ChatGPT Is Quietly Becoming the Biggest Leak in Legal History
When Lawyers Feed the Machine: Why ChatGPT Is Quietly Becoming the Biggest Leak in Legal History

The taboo that vanished overnight

Five years ago, the idea of pasting client work product into a third-party website would have been unthinkable in most firms, the kind of thing that could cost you your job. Today, it’s routine. Not because lawyers suddenly stopped caring about privilege, but because the way we create legal work has quietly, radically changed.


The International Legal Technology Association (ILTA) surveys show a profession that has crossed the AI Rubicon. In 2024, over a third of firms said they were already using generative AI for business tasks - rising steeply with firm size.


In ILTA-related coverage this year, 84% of respondents said AI would be used to summarize complex documents in the next 12 months, precisely the workflow where confidential content is most likely to be copied into prompts. and crucially, about one in five firms is using the public version of ChatGPT (rising to 33% in smaller firms), which typically means consumer-tier terms, limited auditability, and opt-in/opt-out toggles rather than true DPAs.


That’s the mind shift: to realize AI’s value, many lawyers have adopted a new habit, “paste then ask”, that moves sensitive text outside the firm’s controlled environment. No alarms go off. The output looks great. But at the network layer, you’ve just exported client data to a third-party model.


This post builds a plausible yet balanced case that law firms are now extremely likely to be leaking sensitive data, sometimes trivially, sometimes in ways that are hard to see, even when no headlines appear. We’ll ground the argument in industry data, ethics guidance, and real incidents, then map practical controls that preserve AI’s productivity without torching privilege.


The adoption reality (and why summarization is the risk epicenter)

Let’s separate hype from signal.

  • Usage is widespread and growing. ILTA’s 2024 results showed 37% of firms using GenAI for business tasks (20% of small firms, 74% of the largest). That’s a giant step change from 2023.

  • Summarization sits at the center of legal workflows. ILTA-related 2025 coverage reports 84% expecting AI to summarize complex documents in the coming year, exactly the task that tempts lawyers to paste pages of depositions, contracts, and DD reports.

  • Public ChatGPT is commonly in play. “About one in five firms” are using the public version (and one in three smaller firms). That use pattern matters because consumer tiers rely on individual settings and generic privacy controls rather than enterprise-grade data isolation.

The adoption curve explains the behavioral change. When deadlines compress and GenAI is two clicks away, copy-paste becomes muscle memory.

What “public” actually means and why it matters

“Public ChatGPT” isn’t a pejorative; it’s a product tier. On consumer accounts, OpenAI’s own policy states they may use your content to improve models—unless you opt out or use features like Temporary Chat. Enterprise/Business/API tiers invert this by default (no training on your data, better controls, auditability), but many lawyers aren’t there yet.


Implications:

  • Data egress: Text leaves your network and lands on third-party infrastructure.

  • Retention and access: Even when not used for training, logs and telemetry can persist for operational/safety reasons. (Controls vary by tier, product, and settings.)

  • Jurisdictional exposure: Without enterprise agreements, you typically lack contracted data residency and right-to-audit.

  • Privilege ambiguity: Disclosing client confidences to a third party without adequate confidentiality commitments can jeopardize privilege—ethics bodies now flag this risk explicitly.

None of this means “don’t use AI.” It means consumer chatbots aren’t a safe container for privileged content.

“Are we leaking?” A probability argument, not a panic one

We don’t need a breach headline to make a sober assessment.

  1. Usage is high (and growing).

  2. Summarization—the most leak-prone task—is the dominant use case.

  3. A non-trivial share of firms use public ChatGPT.

  4. Human factors drive most security incidents; ILTA trend analyses still rank user behavior as the top risk.

  5. We have precedents (e.g., Samsung’s 2023 ban after staff pasted sensitive code into ChatGPT). Different industry, same behavior pattern.

Put together, the likelihood of inadvertent legal-data leakage is extremely high. Not because lawyers are careless, but because the default workflow (paste → ask → iterate) is leak-friendly unless you put controls in the path.

The nuances your skeptics will raise (and how to handle them)


“We’re only using non-client examples.”

Sometimes true. But in practice, associates under time pressure paste “lightly anonymized” text that still contains deal sizes, dates, counterparties, or drafting idiosyncrasies that can re-identify clients. Redaction by find/replace ≠ anonymization.


“Enterprise LLMs fix this.”

Enterprise-tier ChatGPT/official APIs do improve the risk posture (no training by default, better controls). But leakage still happens if users bypass official channels or if the firm’s proxy/DLP doesn’t see prompts. Shadow usage remains the gap to close. OpenAI


“Privilege isn’t automatically waived.”

Correct, if a vendor is under adequate confidentiality obligations, courts often treat them as an extension of the firm. But consumer chatbots typically don’t supply the same contractual assurances, audit rights, or data handling specificity bar guidance expects. ABA Formal Opinion 512 tells lawyers to know how the tool uses data and implement safeguards before use.


“We’ve seen no incidents.”

Many leaks produce no visible artifact (e.g., retention of prompts in provider logs). The absence of a headline is not assurance. Other sectors learned this the hard way (Samsung).


“Isn’t this just hallucinations?”

Separate issues. Hallucinations affect output reliability (and sanctions). Leakage concerns input handling and data flows. Both are live risks; both require controls.


Ethics & regulatory context—what actually changed

Ethics guidance didn’t lower the bar; it made obligations explicit:

  • ABA Formal Opinion 512 (2024): competence, confidentiality, supervision, communication, and billing transparency apply to GenAI; firms must understand data usage and implement safeguards.

  • SRA/UK guidance emphasizes protecting confidentiality and legal privilege when adopting AI; ICO principles still apply.

The difference is cultural. What was once a sackable taboo is now a normalized step in everyday drafting. Policies haven’t caught up with muscle memory.

How leaks actually occur in 2025 (six common patterns)

  1. Copy-paste to consumer chat (the classic): Deposition pages, client emails, deal schedules.

  2. Drag-and-drop files into web UIs (many tools now accept PDFs/Word).

  3. Browser extensions that “help” summarize current pages—including privileged content in web-based DMS or VDRs.

  4. Bring-your-own-AI on unmanaged devices: private laptops/phones at home.

  5. GPTs/“custom assistants” built by individuals—often defaulting to consumer privacy controls unless carefully configured.

  6. Agentic automations chaining multiple tools (RAG to chat to email), creating untracked egress hops.


Each path is preventable—if you insert controls at the network/API boundary.

A balanced position: keep the gains, cut the risk

GenAI is genuinely valuable to lawyers. Studies and industry reporting show lawyers use it to summarize case law and draft materials, and when used thoughtfully, it boosts productivity without replacing legal judgment.


The goal isn’t to shame usage; it’s to replace risky routes (consumer chat + paste) with safe routes (enterprise/API + policy + pseudonymization).

Practical controls that work (no culture war required)

1) Route all LLM traffic through a network-layer proxy: Make the only way to reach external models pass through a control plane that can classify content, apply policy, and log/audit. (Blocking alone pushes usage off-network.)


2) Pseudonymize at the boundary: Replace sensitive entities (parties, identifiers, amounts) with realistic surrogates before an external model sees them; rehydrate only for authorized users on return. This preserves answer quality while protecting the original data.


3) Auto-detour sensitive prompts to private LLMs: If the prompt crosses a policy threshold (client names, PII, deal data), route to an internal/contracted private model with appropriate DPAs instead of a public endpoint.


4) Make consumer settings a last line, not the first: Turning on “Temporary Chat” or “do not train” helps, but they’re user-controlled toggles and not substitutes for firm-level guarantees.


5) Log the prompts (safely): You can’t govern what you can’t see. Keep prompt-level telemetry (hashed/redacted as needed) for audit and to help partners understand real usage patterns.


Where AI DataFireWall™ fits (briefly)

This is exactly the class of problem AI DataFireWall™ is designed for:

  • API-boundary enforcement (not desktop policing): we live where data leaves, LLM APIs, org APIs, and Contextul APIs.

  • Pseudonymization + rehydration by default at the egress point - zero user friction, context preserved, consistent surrogates.

  • Policy evaluate → deny/allow/detour: block external egress when content is policy-sensitive and optionally reroute to a private LLM sidecar inside your environment.

  • Full audit of the decision trail without exposing the sensitive text to public models.

Net effect: keep the summarization boost, kill the leak vector—without asking associates to change how they type.

The contrarian gotchas (so you’re ready in committee)

“We have a DPA with Vendor X; are we fine?”

Maybe, but check model-improvement use, telemetry retention, subprocessors, data residency, right-to-audit, and fine-grained deletion workflows. Many DPAs don’t address prompt-level data lifecycles.

“We’ll train an internal model and avoid all risks.”

Private models mitigate egress, but shadow usage will persist unless the public path is both policed and replaced with a safer, equally convenient path.

“We’ll just ban AI.”

Bans create BYO-AI and client frustration. The firms that win channel usage into safe defaults—and prove it to clients, will out-compete.


Bottom line

Given adoption rates, the dominance of summarization use cases, and the prevalence of public-tier usage, it is extremely likely that law firms today are leaking client-sensitive data in non-obvious ways, not through hacks, but through habits.


The fix is practical: move from copy-paste anarchy to governed AI, with network-layer controls that preserve speed and privilege. If 2020’s taboo has become 2025’s reflex, it’s time to make safe the default reflex.

Sources

 
 
 

Comments

©2025 Contextul Holdings Limited

bottom of page