Safeguarding Trust in the Age of AI: Our Position on LLMs within NHS Platforms

By Dr. Marc Farr, the Chair of the CDAON

As Chief Data and Analytics Officers across the NHS, we strongly support the responsible adoption of artificial intelligence to enhance insight, productivity and patient care. The integration of advanced analytics and large language models (LLMs) into enterprise platforms represents a significant opportunity for our health and care system.

However, recent discussions within our Network reflect a growing concern among NHS analysts about how LLM functionality is being embedded into operational environments — particularly within the Federated Data Platform (FDP).

We do not raise these issues to slow progress. We raise them to ensure that progress is sustainable.

Secure Interface Does Not Automatically Mean Secure Processing

We recognise that there can be an implicit assumption that if an AI capability sits “inside” an NHS platform, then all associated processing remains within NHS-controlled infrastructure. In practice, LLM integrations can involve more complex technical arrangements.

Depending on configuration, elements such as prompts, contextual inputs, system logs or generated outputs may interact with external model providers. Without clear technical documentation, local organisations and analysts are left uncertain about:

• Whether any component of processing leaves the platform boundary
• Which organisations act as processors or sub-processors
• Where infrastructure is geographically located
• How long information is retained
• Whether data contributes to model logging or improvement processes

In the absence of transparent assurance, it becomes difficult for organisations to confidently demonstrate compliance with UK GDPR, the Data Protection and Digital Information framework, and NHS information governance requirements.

Clarity is not optional; it is foundational.

Accountability Must Sit at the Appropriate Level

We are aware of situations where analysts are being encouraged to use embedded AI tools to support delivery, without accompanying clarity on governance boundaries.

We believe that individual analysts cannot reasonably be expected to assess cloud architecture, cross-border processing arrangements, supplier contracts or residual legal risk. Those responsibilities sit with platform owners, national bodies and accountable organisations.

Where AI functionality is introduced at scale, governance, DPIAs, and risk ownership must also be established at scale.

Professional data staff must never be placed in a position where they feel personally exposed for using centrally provided tools.

Data Sovereignty and International Processing

Many leading LLM providers operate global cloud environments. Where processing involves infrastructure outside the UK — particularly in jurisdictions subject to different surveillance or access regimes — we must have explicit assurance regarding safeguards and contractual protections.

Even if the practical risk is assessed as low, ambiguity itself carries reputational and public trust implications. If NHS data — whether direct content, contextual information or metadata — could be processed outside UK sovereign control, this must be clearly articulated, lawfully justified and transparently governed.

Public trust depends on demonstrable stewardship.

Equity and Analytical Integrity

Beyond information governance, we also recognise the analytical risks.

AI-enabled decision-support tools built on partial or uneven datasets risk embedding structural blind spots. For example, if outputs rely predominantly on secondary care or prescribing data without sufficient contextualisation, there is a possibility of reinforcing inequities in population health management.

We believe equality and health inequalities impact assessment must form an integral part of AI-enabled analytical development. AI systems scale patterns rapidly; without safeguards, they may also scale bias.

Innovation must reduce inequalities, not widen them.

This Is a System-Wide Question

Although these concerns have surfaced in relation to the FDP, we recognise that the underlying issues extend beyond any single platform. As AI becomes embedded across multiple NHS systems, the same questions will arise elsewhere.

We therefore view this as a system-level governance issue requiring consistent national clarity.

Our Position

As the Chief Data and Analytics Officers Network, we:

• Support the responsible and proportionate use of AI to enhance NHS capability.
• Expect full transparency regarding technical data flows and processing arrangements.
• Believe accountability for AI-related risk must sit with platform owners and commissioning bodies, not individual analysts.
• Consider formal DPIAs and IG sign-off essential before widespread deployment.
• Regard equity impact assessment as a core requirement for AI-enabled analytical tools.

Raising these issues is not resistance to change. It is part of our professional obligation to ensure that data use in the NHS remains lawful, ethical and defensible.

Immediate Areas Where Assurance Is Required

To enable confident adoption of AI capabilities, we believe the system now requires:

1. Authoritative Information Governance guidance confirming whether current AI integrations are approved for defined NHS analytical use cases.
2. A clear and accessible description of technical data flows, including any involvement of third-party processors or overseas infrastructure.
3. Explicit clarification of organisational accountability and liability arrangements.
4. Interim guidance where governance review is ongoing.
5. Formal review and endorsement by appropriate Data Protection Officers and programme leadership.

These are proportionate and reasonable expectations. They align with existing NHS governance principles and with the professional standards expected of senior data leaders.

Protecting the Conditions for Sustainable Innovation

The NHS is entrusted with uniquely sensitive data. Our responsibility is not only to unlock its value, but to safeguard the trust that underpins its use.

AI offers extraordinary potential. But its adoption must be demonstrably lawful, transparent, equitable and accountable.

Where uncertainty exists, we believe it is our duty to surface it constructively and collectively. By doing so, we can ensure that AI strengthens public confidence in NHS data stewardship rather than testing it — and that innovation proceeds on foundations that are secure as well as ambitious.