A reference architecture for
HIPAA-aligned voice agents.

Why "wire up an LLM" isn't enough

A patient-facing voice agent touches Protected Health Information (PHI) the moment the caller says their name. Under the HHS HIPAA Security Rule, every service that processes that PHI must be a covered entity or a Business Associate — including your speech-to-text vendor, your LLM provider, your text-to-speech vendor, and any tool the agent can call.

The wall most pilots hit isn't the model quality — it's the answer to the question "who else did our patient's voice pass through, and did they sign a BAA?" Get that architecture right on day one and the rest is engineering.

The reference architecture, at a glance

   Caller (PSTN / SIP / WebRTC)
        │
        ▼
   ┌──────────────────────────┐
   │  Telephony ingress       │  ← BAA required
   │  (SIP trunk, WebRTC gw)  │
   └────────────┬─────────────┘
                │  raw audio + call metadata
                ▼
   ┌──────────────────────────┐
   │  PHI-aware guardrail     │  ← in-VPC
   │  input filter            │
   └────────────┬─────────────┘
                │
                ▼
   ┌──────────────────────────┐
   │  Speech-to-text (STT)    │  ← BAA required
   │  streaming               │
   └────────────┬─────────────┘
                │  transcript
                ▼
   ┌──────────────────────────┐
   │  Agent orchestrator      │
   │  · policy schema         │
   │  · retrieval (RAG)       │
   │  · tool router (allowlist)│
   │  · human-escalation trap │
   └────────────┬─────────────┘
                │  prompt + context
                ▼
   ┌──────────────────────────┐
   │  LLM (BAA-covered)       │  ← Bedrock / Azure OpenAI / Vertex
   └────────────┬─────────────┘
                │  candidate response
                ▼
   ┌──────────────────────────┐
   │  Output validator        │  ← in-VPC
   │  · groundedness check    │
   │  · PHI-leak check        │
   │  · policy conformance    │
   └────────────┬─────────────┘
                │  approved text
                ▼
   ┌──────────────────────────┐
   │  Text-to-speech (TTS)    │  ← BAA required
   └────────────┬─────────────┘
                │  audio
                ▼
           back to caller

   Every arrow is logged to an append-only audit store.
        

The six controls that make it defensible

1. Only BAA-covered providers in the audio path

Amazon Bedrock, Azure OpenAI, and Google Vertex AI all publish HIPAA-eligible service lists and will sign BAAs. For STT/TTS, the same is true of Amazon Transcribe Medical, Azure Speech, and Google Cloud Speech-to-Text — but only when configured for HIPAA-eligible workflows. Consumer OpenAI, Deepgram's default tier, ElevenLabs' default tier — none of these are BAA-eligible off the shelf.

The rule: if the vendor's BAA doesn't cover the specific endpoint and region you're calling, don't call it with PHI. Period.

2. PHI-aware guardrail on both input and output

An input filter that detects and structures PHI (name, MRN, DOB, address) before it reaches the LLM lets you decide, per prompt, whether to redact, tokenize, or pass through. An output validator that checks whether the response contains PHI the caller didn't originate (i.e. the model leaking from context) is the second half of the same control.

This is where the OWASP LLM Top 10 maps directly onto healthcare: LLM06 (Sensitive Information Disclosure) is exactly this failure mode.

3. Retrieval, not recall

The LLM should never be the source of truth for clinical or billing facts — it composes what a retrieval layer surfaces. This does two things at once: it makes hallucinations detectable (every claim can be traced back to a document) and it makes the audit trail meaningful (you can prove which record the answer was based on).

Practical hint: every response the agent gives should be linkable to a document ID + version + retrieval score. The audit log stores that link, not just the text.

4. Tool-use allowlist, not tool-use permission

If the agent can call scheduleAppointment(patientId, slotId), the orchestrator must enforce three checks before the call: (a) the patient's identity was verified in-call, (b) the requested slot belongs to the patient's own provider, and (c) the call originated from the LLM in response to an explicit caller request. The LLM never gets to decide "yes, call this tool." The orchestrator decides.

5. Human-in-the-loop escalation trap

Every voice agent needs a documented, deterministic list of triggers that hand the call to a human: caller says "let me talk to a person"; symptom-severity keyword hit; billing-dispute keywords; low confidence score on identity verification; three failed clarifications in a row; anything the caller marks as urgent.

The trap must be adversarial-safe: the LLM cannot be prompted or persuaded out of it. Escalation is enforced by the orchestrator on rules, not by the model on judgement.

6. Audit log that a compliance officer can read

HHS §164.312(b) requires the ability to reconstruct exactly what happened to any PHI. In a voice-agent context, that means per-call: caller identity (as verified), consent captured, audio recording (encrypted, retention-bounded), full transcript, every LLM prompt and response with prompt hash, every retrieval hit with document version, every tool call attempted (allowed or blocked, with reason), and the final disposition.

Write it to an append-only store. Not a mutable database. Not "we can query CloudWatch." An actual immutable ledger with retention aligned to the client's record-retention policy (often 6–10 years for healthcare).

What the deployment topology usually looks like

• Region: AWS us-east-1 or us-east-2 for U.S. clients with U.S. patients; keep audio storage in-region.
• Networking: LLM and STT/TTS calls from within a client-owned VPC, using PrivateLink where available so nothing traverses the public internet.
• Encryption: KMS-managed keys per tenant. Audio at rest is encrypted with a customer-managed key; access to the key is separately audited.
• Access: Break-glass only. No standing human access to raw audio. Every access request is logged and reviewed.
• Retention: Aligned to the client's HIPAA retention policy; automatic tombstoning after that window.

What to test before you go live

1. Prompt-injection replay: Feed the STT layer 30 known jailbreak strings via text-to-speech. Confirm zero of them cause the agent to break policy.
2. Cross-tenant PHI test: Ensure retrieval from tenant A can never surface data from tenant B, even when the identifiers collide.
3. Consent-first flow: Every call must capture consent before any tool call. Prove it in the audit log.
4. Escalation reachability: Verify every escalation trigger reaches a human within the SLA — including the "adversarial persuasion" test where the LLM is prompted to avoid escalating.
5. Groundedness threshold: Measure per-response groundedness on a curated 200-question eval set. Set a live-fire threshold. If a response falls below it, the orchestrator refuses to speak and escalates.

What this doesn't solve

This architecture is the technical floor for HIPAA alignment. It doesn't replace:

• A written risk analysis (§164.308(a)(1)(ii)(A)) — you still need to do one.
• Workforce training — everyone with access to the system needs HIPAA training on record.
• Business Associate Agreements with your own clients — not just with your sub-processors.
• State-specific rules (e.g. Texas HB 300, California CMIA) that layer on top of federal HIPAA.

HIPAA Voice agents Healthcare AI Reference architecture