HIPAA AI Inference: Running LLMs on Patient Data Without a BAA Nightmare

A teleneurology startup called us last month. Twelve clinicians, 800 video visits per week, and a board meeting in nine days. Their problem was simple: they had bolted GPT-4 onto their EHR for SOAP-note drafting, then learned during a SOC 2 readiness audit that OpenAI's standard contract is not a HIPAA Business Associate Agreement. The auditor was not amused. Neither was their general counsel.

This is not an edge case. It is the default story for any healthcare team trying to put large language models near patient data. The infrastructure that makes LLMs cheap and fast is the same infrastructure that makes them HIPAA-incompatible: shared GPUs, shared hypervisors, and a cloud operator who can technically read every byte that crosses VRAM.

The BAA Trap That Nobody Wants to Explain

HIPAA requires a Business Associate Agreement with anyone who handles Protected Health Information on your behalf. The Privacy Rule is unambiguous (45 CFR § 164.502(e)): no BAA, no PHI. So when a covered entity wants to send a doctor's dictation through a cloud LLM, every party in the data path needs to sign.

Here is the awkward part most vendors will not say out loud:

• OpenAI offers a BAA only on a specific Enterprise tier. The default API contract excludes PHI.
• Anthropic publicly stated in 2024 that PHI through the Claude API is out of scope without a separate enterprise agreement.
• Most Bittensor-style and "decentralized" GPU networks cannot sign a BAA at all — there is no single legal entity to indemnify.
• AWS, GCP, Azure sign BAAs, but only for specific HIPAA-eligible services. Run a vanilla EC2 with an A100 and you are out of scope.

So clinics either pay enterprise rates ($60K+/year minimums are typical) or they self-host on a HIPAA-eligible GPU instance and pay 2–4× the going market rate. The BAA premium is real, and it has nothing to do with technology — it is a contractual surcharge for a guarantee the cloud cannot actually verify.

Why Traditional Clouds Cannot Fix This

Even with a BAA, traditional cloud GPU instances have a structural problem. PHI lives in plaintext at three layers during inference:

1. System RAM while the inference server marshals tokens.
2. GPU VRAM while the model processes them.
3. PCIe bus on every host↔device transfer.

A privileged hypervisor process, a compromised orchestrator, or a sufficiently determined physical attacker can read all three. The BAA gives you legal recourse after a breach. It does not prevent one. HIPAA's technical safeguards rule (45 CFR § 164.312) asks you to encrypt PHI — and traditional inference hardware decrypts it the moment work begins.

What Intel TDX Actually Does

Intel Trust Domain Extensions (TDX) is the third generation of confidential-computing hardware to ship in production servers. It creates a Trust Domain (TD): a VM whose memory is encrypted with a per-TD AES-256-XTS key managed by the CPU itself. The hypervisor sees ciphertext. The host kernel sees ciphertext. Even VoltageGPU sees ciphertext.

For GPU workloads, the missing piece used to be the bus. Encrypted RAM is useless if the H100 receives plaintext over PCIe. Intel TEE-IO closes that gap: data traveling between the CPU and an attested H100/H200/B200 is encrypted end-to-end. The attacker model collapses to "the silicon is lying," which is a much harder bar than "someone has root on the host."

How a HIPAA-Safe PHI Flow Looks in Practice

Here is the deployment shape for a typical clinical-summarization workload on VoltageGPU:

1. Provision an attested H200 TD. The pod ships with a measured boot, and the TD's identity (MRTD) is recorded.
2. Verify the attestation quote from your application before sending any PHI. If the quote does not match the expected MRTD — fail closed.
3. Send your prompt to a TLS endpoint that terminates inside the enclave. The inference server is the only entity holding the cert key.
4. Set confidential: true on the request to disable any residual logging and request VRAM scrubbing on completion.
5. Tear down the pod at end-of-shift. VRAM is wiped before the next TD is provisioned.

Verifying the attestation quote is two dozen lines of Python:

import requests, json

# Verify the TDX attestation BEFORE you ever send PHI.
# If the quote does not match expected MRTD, fail closed.
quote = requests.get(
    "https://api.voltagegpu.com/v1/pods/POD_ID/attestation",
    headers={"Authorization": "Bearer vgpu_YOUR_KEY"},
).json()

assert quote["tdx_version"] == "1.5"
assert quote["measurement_valid"] is True
assert quote["mr_td"] == EXPECTED_MR_TD  # pinned at provisioning
print("Enclave verified. Safe to transmit PHI.")

And here is the actual inference call. The SDK is OpenAI-compatible:

from openai import OpenAI

# Endpoint terminates inside an Intel TDX enclave on H200.
# PHI never leaves the encrypted boundary; even VoltageGPU
# cannot read prompts, completions, or VRAM contents.
client = OpenAI(
    base_url="https://api.voltagegpu.com/v1/confidential",
    api_key="vgpu_YOUR_KEY",
)

resp = client.chat.completions.create(
    model="meta-llama/Llama-3.1-70B-Instruct-TEE",
    messages=[
        {"role": "system", "content": "You are a clinical summarizer. Output: SOAP."},
        {"role": "user", "content": "Patient note: 62F, dyspnea on exertion..."},
    ],
    temperature=0.2,
    max_tokens=600,
    extra_body={"confidential": True, "no_log": True},
)
print(resp.choices[0].message.content)

Real Numbers: VoltageGPU TDX vs HIPAA-Eligible Hyperscaler GPUs

Two notes on the table. First, Azure's confidential VM SKUs change quarterly — we last cross-checked these prices on April 22, 2026; treat them as a snapshot. Second, Azure adds a HIPAA premium on top of the base CC rate when you negotiate a BAA through their Microsoft Cloud for Healthcare contract; the gap above understates the real TCO difference.

Where VoltageGPU Loses (Pratfall, Honest Edition)

I am not going to pretend we are strictly better than Azure on every axis. We are not. Three honest limitations:

• No SOC 2 Type II yet. We rely on Intel TDX attestation and GDPR Article 25 by-design controls. Some procurement teams require SOC 2 by default; if you do, talk to us — we are mid-audit, due Q3 2026.
• PDF OCR is not native. If your PHI lives in scanned faxes, you need to run an OCR step yourself before our inference endpoint. Azure's Document Intelligence is more turnkey here.
• Cold start on Starter is 30–60 seconds. If you are building a patient-facing chatbot with sub-second SLAs, you want the Pro plan or a reserved pod.

These are tradeoffs, not show-stoppers. For a typical teleclinic or behavioral-health practice, you trade a certification logo for a 60-70% cost reduction and stronger technical safeguards than the cloud they came from.

Who Should Care

• Telehealth platforms running clinical summarization, intake triage, or coding assist on dictated visit notes.
• Behavioral health and substance-use clinics where session transcripts are extraordinarily sensitive (42 CFR Part 2 territory).
• Medical imaging startups running inference on DICOM that they cannot legally pool with another tenant's data.
• Hospital IT teams piloting LLM-assisted EHR features and trying to keep their CISO from having a stroke.

If you are in any of those buckets, two starting points:

• The confidential computing primer for the architecture story.
• The healthcare landing page for HIPAA-specific terms, pricing, and the BAA template.

FAQ