The Anatomy of Leaked Data into an LLM.

Data exfiltration to Large Language Model (LLMs)

How harmless prompts become expensive problems in financial services

Every paste to an assistant is an outbound API call. That single action can set off a chain where sensitive details leave your tenant, get copied for product operations or safety analysis, and later reappear in ways that drive up costs, enable fraud, or erode trust. Even when a provider does not train its foundation models on your enterprise content, other pathways still create exposure, such as logs, examples for safety, plug in telemetry, and link preview caches. Using Barclays as an example in the financial services sector, the real damage tends to arrive weeks later as price drift in tenders, targeted spear phishing, client churn in wealth, or operational disruption.

Microsoft states that Microsoft 365 Copilot does not use your organisational prompts or Graph data to train foundation models, which is important (there is evidence, however, that this is changing as training data starts to dry up from public sources) e.g. Linkedin had the same promise, but recently updated their T&C's to reflect a change in strategic approach to access users' activity to train their models). Irrespective, that does not remove the need for boundary controls at the API edge, because the initial outbound hop is still where sensitive content can escape to multiple processing surfaces and vendors.

A simple flow that explains most incidents

User Action  ->  Outbound Path  ->  Capture and Retention  ->  Public Model Spillover  ->  Resurfacing  ->  Impact

• User Action: a colleague pastes content or drags a file into an assistant. Hidden columns, comments, EXIF or DICOM tags, and thread history can ride along.

• Outbound Path: the content leaves the tenant through an API call, a plug-in or extension, or a shared link with history.

• Capture and Retention: providers may store copies for abuse detection, quality support, safety evaluation or debugging. Plug ins and annotation vendors can retain snippets. Link previews and embeddings may be cached.

• Public Model Spillover: even where core training is not allowed, snippets or patterns may still influence examples, safety tests or autocomplete behaviour if improvement features are enabled.

• Resurfacing: information reappears as higher supplier prices, spear phishing with real amounts and dates, local re identification of clients, or leaked draft narratives.

• Impact: contract value erosion, cash losses, regulatory scrutiny, team disruption, and reputation damage.

  
  

Five hypothetical Barclays scenarios that look harmless, then boomerang

Each scenario shows who did what, what actually leaked, why it felt safe, how it resurfaces, and how others likely got hold of it.

1) ExCo pack to Copilot

• Action: an executive assistant pastes a confidential ExCo deck into Copilot to create a one pager.

• What leaked: hidden tabs and speaker notes reveal supplier discount ladders, price ceilings, and programme milestones.

• Why it felt safe: the deck was marked internal only and felt administrative. Metadata was not visible, and unknown.

• Resurfacing: next RFP cycle returns oddly harmonised bids that sit just below the known ceiling. Procurement sees a 2 to 3 percent drift across a multi year framework.

• Likely path: outbound API call, provider logs for quality or support, then knowledge of ceilings circulates through people and vendors who see or infer that data.

Note: Microsoft documents that Copilot for Microsoft 365 does not use your organisational prompts or Graph data to train foundation models. That reduces one class of risk and is welcome. It does not eliminate exposure from logs, plug ins, previews or downstream vendors where your data may still be processed for operations if allowed by settings and contracts.

2) KYC and wealth narratives

• Action: a wealth associate uploads KYC PDFs to summarise complex family structures and beneficiary links.

• What leaked: names, trusts, addresses, holdings and relationship notes. Even if a document seems redacted, free text often contains quasi identifiers.

• Why it felt safe: the aim was only to rewrite the narrative in plain English.

• Resurfacing: a targeted spear phish uses highly specific family and asset details. Competitors approach clients with strikingly accurate offers.

• Likely path: a plug in or extension with telemetry, an annotation vendor used for quality or safety, and cached previews. Link preview research shows how unfurling can leak data through caches and background fetches if not implemented carefully.

  
  

3) Payments CSV for categorisation

• Action: finance uploads a month end CSV to classify vendors and refunds.

• What leaked: sort codes, partial account numbers, rare vendor names, refund narratives and exact amounts.

• Why it felt safe: it looks like admin data, not client data.

• Resurfacing: invoice redirection emails reference the exact refund narrative and date. Losses occur before controls catch up.

• Likely path: a shared link with history enabled, then link previews and CDN caches index titles and snippets that include vendor names and amounts.

  
  

4) HR compensation planning

• Action: HR shares a comp spreadsheet to draft personalised communications.

• What leaked: names, bands, equity, and informal notes that create discrimination risk.

• Why it felt safe: this sits inside a private chat.

• Resurfacing: grievance and media friction when draft and final narratives appear misaligned. Attrition rises in a critical team.

  
  

5) Trading and infrastructure runbooks

• Action: an engineer pastes a runbook to refactor a script that contains keys in comments.

• What leaked: API keys, endpoints and internal dataset URIs.

• Why it felt safe: it is only code, and the keys are in comments.

• Resurfacing: credential abuse leads to lateral movement and a weekend outage. Regulated incident reporting and client communications follow.

• Memorisation risk context: research has shown that large language models can memorise rare strings and that trained models can be coaxed to produce verbatim training data under specific conditions. Controls must assume rare values like keys and UUIDs are high risk.

  
  

Public model spillover, explained without drama

There are two truths that must be held together:

1. Enterprise assurances are improving. Microsoft states that for Microsoft 365 Copilot, prompts, responses and data accessed through Microsoft Graph are not used to train foundation models. Copilot customer data is not available to OpenAI, even when OpenAI models are used within Azure, and model provider choice is becoming more flexible.

   
   

2. Spillover can still occur through other channels. Providers and their vendors may handle copies for abuse detection, security testing or quality evaluation. Some products allow optional improvement settings. Plug ins and extensions may retain telemetry. Link previews and caches can store snippets. These are not the same as core training, but they can create public facing echoes, such as realistic synthetic examples, suggestion bias or highly specific autocompletions. OpenAI states that business offerings like the API and enterprise tiers do not train on your data by default, which is good, yet the same caution applies when integrations and plug-ins are added.

Implication for Barclays: The bank should treat every outbound AI call as a data sharing event with layered processing and retention. Foundation model training bans reduce one class of risk. Boundary controls are still required to remove sensitive specifics before the first hop.

Where regulation bites: DORA, PRA and FCA

DORA applies in the EU from 17 January 2025 and sets uniform rules on ICT risk management, incident reporting, resilience testing and oversight of critical third party ICT providers. For EU subsidiaries and branches, DORA drives stronger contractual and technical controls for any AI related external processing, including logging, testing or annotation vendors. It also elevates incident classification and reporting when AI assisted processes contribute to an operational disruption.

PRA SS2/21 in the UK sets expectations for outsourcing and third party risk management. It requires robust governance, data security, audit rights, exit strategies and clarity on sub processors. If a model provider, a plug in, or an annotation vendor handles bank data, the firm must be able to evidence appropriate security, audit and termination terms. This includes controls on data location, retention, incident response and access.

Together with the FCA and PRA operational resilience policy, the direction of travel is clear. You are expected to understand and control your third party AI supply chain, to limit the blast radius of any failure, and to ensure rapid recovery when something goes wrong. DORA makes the EU angle more prescriptive with common incident taxonomy and oversight of critical providers. The UK regime expects equivalent outcomes through supervisory statements and resilience policy.

Pinch point for Barclays: EU entities must meet DORA’s explicit ICT third party requirements and testing expectations. UK entities must meet PRA SS2/21 and operational resilience outcomes. The weakest link is often the informal plug in or analytics vendor that sits between the user and the model. These create data flows that are difficult to evidence and hard to terminate quickly.

The adversary playbook, shortened

• Compromised extensions or plug-ins capture prompts and files for analytics.

• Annotation and safety vendors hold snippets for labelling or evaluation.

• Link previews and caches persist sensitive titles and snippets. Research shows these mechanisms can leak data or fetch content in the background in ways users do not expect.

  
  

• Prompt injection on a browsable agent coerces the assistant to exfiltrate secrets.

• Data brokerage aggregates leaked details into lists for fraud or price intelligence.

A realistic timeline looks like this: Day 0 paste. Day 3 caches and telemetry contain useful snippets. Day 14 invoice redirection attempts begin using real narratives. Day 45 suppliers return with curiously aligned bids.

Quantifying the risk

• Price drift: a 2 to 3 percent rise on a multi year procurement framework can cost seven to eight figures over the life of the contract.

• Fraud: a small number of successful invoice redirections can wipe out savings from automation.

• Governance overhead: internal investigations, legal advice, vendor audits and regulator liaison consume senior time.

• Trust: client retention in wealth is acutely sensitive to perceived confidentiality failures.

These are illustrative estimates. Your own telemetry will give better numbers once you measure the boundary.

Controls that work, with trade offs

Ban public LLMs entirely

• Strong containment.

• Trade off is heavy productivity loss and shadow IT as staff go outside the boundary.

Private only LLM

• Keeps data on premises or within a strict boundary.

• Trade off is loss of frontier model capability and ecosystem features, so staff may still look for external tools.

Training and policy only

• Necessary foundation.

• Not sufficient against metadata, quasi identifiers, plug in telemetry and link previews.

Boundary control at the API edge

• Intercept the first hop.

• Transform or block sensitive data before it leaves.

• Detour tasks to a private sidecar so work continues inside the tenant.

  
  

What AI DataFireWall changes, in plain terms

Scope: AI DataFireWall sits on the API boundary. It protects LLM APIs and organisational APIs, not static data at rest.

Default action:

1. Pseudonymise on egress. Names, client identifiers, account numbers, amounts and dates are replaced with realistic, format true surrogates that preserve analytical value.

2. Block external egress when policy demands. The user gets a clear reason.

3. Detour to a private LLM sidecar hosted inside your environment so the task completes without the data leaving.

4. Rehydrate on return for authorised users only, with no manual toggles. Others see safe surrogates.

5. Full audit to SIEM for compliance and internal audit.

Effect: staff keep the power of public models without exposing live values. Shadow AI pressure reduces because there is no friction. This aligns cleanly with DORA, PRA SS2/21 and operational resilience objectives because it places a verifiable control at the boundary and limits third party data spread.

Quick checklist for Barclays

Before enabling any assistant in production

• Confirm whether prompts and responses are used to train foundation models. For Microsoft 365 Copilot, Microsoft states they are not. Record the citation.

• List every plug in, extension or connector. Require a data flow diagram and retention statement.

• Disable or control link unfurling for sensitive workspaces. Educate staff that previews can leak.

• Negotiate audit rights, subprocessor lists, data location and deletion timelines. Align to DORA for EU entities and PRA SS2/21 in the UK.

  
  

What to transform or block by default

• Client names and identifiers, account numbers, amounts, dates.

• Code secrets and endpoints.

• HR and compensation fields.

• Supplier prices and discount ladders.

• KYC narratives, trust structures and addresses.

  
  

Where to detour to a private sidecar

• ExCo packs and procurement.

• KYC and wealth documentation.

• Payments and reconciliations.

• HR compensation and sensitive employee data.

• Trading and infrastructure runbooks.

  
  

How to measure success

• Percentage of outbound calls transformed, detoured or blocked.

• Observed reduction in sensitive strings leaving the tenant.

• Fraud attempts detected that contain previously exposed narratives.

• Procurement price variance on comparable tenders.

  
  

Close: turn the risk into an advantage

Today’s paste is tomorrow’s price rise, phish or headline. Foundation model training bans protect against one category of risk, which is valuable. The hidden exposure sits in the first outbound hop and in the operational plumbing that surrounds modern assistants: logs, safety examples, plug ins, previews and caches.

AI DataFireWall makes the risky hop safe by default. It pseudonymises live details at the boundary, blocks what should not leave, detours work to a private sidecar when needed, and rehydrates originals only for authorised users. Your staff keep the speed of public models. Your regulators get a clean, auditable control that matches the intent of DORA and PRA SS2/21. Your clients and suppliers see a bank that treats confidentiality as a first class product feature.

DISCLAIMER: This article is an educational analysis of potential risks associated with large language model use in financial services. References to Barclays Bank are illustrative case studies only and do not describe actual events, incidents, or internal practices at Barclays. The scenarios are fictionalised examples created to help readers understand how sensitive data can leak into AI systems and the possible downstream impacts. Nothing in this article should be taken as a statement of fact about Barclays or its affiliates.

Readers should seek their own professional advice when assessing regulatory obligations under DORA, PRA, FCA or other applicable frameworks.

Sources

• Microsoft 365 Copilot: prompts, responses and Microsoft Graph data are not used to train foundation models. Microsoft Learn

• Microsoft 365 Copilot: customer data is not available to OpenAI and is not used to train OpenAI models. TECHCOMMUNITY.MICROSOFT.COM

• OpenAI enterprise privacy: API and enterprise tiers do not train on your data by default. OpenAI

• DORA overview and application from 17 January 2025. EIOPA

• DORA official text, Regulation (EU) 2022/2554. EUR-Lex

• PRA SS2/21 Outsourcing and third party risk management, updated November 2024. Bank of England

• Link preview privacy risks and background data fetching. mysk.blog

• Memorisation and training data extraction in large language models.