Back to Insights

HIPAA + AI isn't just a BAA. It's architecture.

Isometric illustration of a secure cloud computing architecture with a shield motif
ShareLinkedInEmail

By Bennett Malbon, Chief Technology Officer · July 16, 2026

A signed BAA alone doesn't equal HIPAA compliance. This gap is where clinical AI roadmaps stall: in a 2025 JAMIA survey, seven in ten health systems named data privacy and security a major barrier to AI adoption. Closing the gap is a solvable engineering problem.

Even savvy teams trip on configuration and governance after the BAA. Here's the plan: Pick your cloud infrastructure, close the configuration gap, govern the outputs, and earn autonomy with data.

Background: Three shapes of cloud AI infrastructure

  1. Frontier models through managed AI services. GPT-, Claude-, and Gemini-class models behind an API on Azure, AWS, or Google Cloud. Fastest start, strongest capability, no infrastructure to run. Your data protections live entirely in the platform configuration, which makes configuration the key to compliance.
  2. Open-weight models through managed AI services. Llama, Qwen, Mistral, and DeepSeek, open weight models, served serverless in Bedrock, Foundry, and Vertex, and fine-tunable on your own data. Often the right size: extraction, classification, and structuring tasks rarely need a frontier model, and a small model is cheaper, faster, and simpler to defend in review.
  3. Open weights on private cloud GPU compute. An inference server and a model checkpoint on EC2, EKS, GKE, or AKS, all HIPAA-eligible under the standard BAAs. No model provider in the loop at all: zero third-party retention by construction, because nothing leaves your boundary. The trade-off: you take on the operational work a managed service would otherwise handle, patching servers, scaling capacity, and running evals yourself.

Wherever a workload sits, auditable compliance is decided in the same place: the configuration.

The configuration gap: What to do after the BAA

Platform defaults are tuned for convenience, not HIPAA. Teams that fail their first compliance review almost never fail because cloud AI is off-limits. They fail because they assumed the defaults were compliant. Here's what to pressure test in your configuration:

  • Retention and monitoring. Azure OpenAI's abuse monitoring can store flagged prompts for human review unless you're approved for modified monitoring, and the approval is verifiable in the resource configuration itself. Gemini caches prompts for up to 24 hours unless you disable it at the project level. On Bedrock, Anthropic's newest Claude models are unavailable until you explicitly opt your account into a provider-data-share mode (prompts and outputs go to Anthropic for safety review, retained up to 30 days) unless AWS grants you zero-data-retention access to that model. Zero retention is achievable on all three platforms, but only Bedrock defaults to it, and even there the newest frontier models are the carve-out. Verify ZDR, don't assume it.
  • The sub-processor chain. When a platform hosts someone else's model, know exactly where the BAA stops. That Bedrock data-share mode sends prompts to Anthropic, and Anthropic is not a party to your AWS BAA. Azure's catalog draws the same line contractually: "models sold directly by Azure" sit under Microsoft's terms and BAA, while "models from partners and community," including some serverless Llama endpoints, are formally not Microsoft products. Same console, different contract. A model isn't covered just because the platform hosting it is.
  • Features that step outside the ring. Individual features can silently exit your control envelope. Turn on Vertex's Grounding with Google Search and certain security controls, including customer keys and VPC Service Controls, no longer apply to that traffic. Read the per-feature compliance matrix, not just the service page.
  • Network path and keys. By default your prompts travel the public internet to a public endpoint. All three platforms support private connectivity (PrivateLink on AWS, Private Link on Azure, Private Service Connect on Google Cloud) so inference traffic never leaves your network fabric, and all three support customer-managed encryption keys for what does get stored.
  • Region and deployment type. Azure's "Global" deployments can process prompts in any geography where the model runs. Data Zone and regional deployments keep processing where you expect. The convenient endpoint isn't always the in-scope one.
  • Your own logs. Bedrock's invocation logging is off by default. Turn it on for auditability and the prompts land in your own S3 buckets and CloudWatch, which makes PHI in logs your problem to govern. CloudTrail records who called the model, never what they sent; prompt and token telemetry is something you build and govern.

Notice the pattern. On every axis that matters (retention, residency, routing, logging, keys), the platforms give you the governance controls you need, but many are not the default. HHS's proposed Security Rule update would make controls like encryption mandatory rather than addressable. The BAA is table stakes, but the configuration is how the AI-specific part of compliance is actually achieved.

Govern the outputs through implementation: Batch first, human in the loop

Configuration governs what happens to data going into the model. It says nothing about the other half of governance: what your application does with what comes out. No console setting reviews a model's clinical claim before a patient sees it. That control lives in your application, which is why the choice of workload matters as much as the platform settings. Pick one that makes every control easy to prove:

  1. Batch and asynchronous, not real-time. An overnight or event-driven job gives you room to stage data deliberately, scope exactly what leaves your boundary, and log every call for audit. Every configuration control above is easier to demonstrate when nothing is time-critical. An eligibility gate that decides which records even warrant inference belongs here too; inference is expensive.
  2. Right-sized on the dial. Match the model to the task. A small open model for extraction or classification, a frontier model only where the reasoning earns it. This is where the three shapes stop being background and become line items in your architecture and your budget.
  3. Human-reviewed on the way out. Batch output lands in a review queue for a clinician, not in front of a patient. That's the output-side control, and it turns "the AI made a clinical claim" into "the AI drafted something a clinician approved," a fundamentally easier conversation with compliance. The pattern is already mainstream: every ambient-documentation product that made it into production, like Abridge, works exactly this way: the model drafts and the clinician signs. This creates labeled data.
  4. Close the loop with your own data. Every clinician review is a labeled example. Feed them into an eval harness that measures accuracy and drift, and use them to improve the system: better prompts, a fine-tuned open-weight model, reinforcement learning where it earns its complexity. Measured trust is what eventually relaxes the human checkpoint. Take the training wheels off slowly.

Get one valuable, non-critical-path workload running this way (BAA signed, configuration audited, outputs reviewed) and you've converted an abstract fear into a reference architecture a HIPAA security team can approve. The next workload is an incremental improvement, not uncharted territory.

What this looks like in practice

That reference architecture isn't hypothetical; it's how we're building ours. At EZCARE, the clinical-commerce module of the T2 Ignite platform, our pipeline reads a visit summary, extracts the clinical concepts, and recommends supportive-care supplies aligned with what the clinician actually documented, each recommendation carrying a rationale traced back to the note. Inference over clinical text used to require hosting your own models. Not anymore.

We don't. It's an asynchronous pipeline. Data lands in a staged, layered store, so we control exactly what reaches the model and every hop is auditable. Each pipeline step runs the smallest model that does the job. And recommendations reach the patient with a clinician's endorsement. The human checkpoint is built into the domain model, not a policy memo. Each endorsement is also a labeled example, the seed of the eval loop that earns the pipeline more autonomy over time. All of it designed to run under the same class of agreement that already covers the rest of a health system's cloud footprint. No rack of GPUs, and no excuse not to ship.

The takeaway

Health systems that get AI into production won't be the ones with the biggest GPU budget or the boldest legal team. They'll be the ones that treat the space between the BAA and the audit as an engineering backlog: retention verified, routing private, outputs reviewed, trust earned with data. That work is unglamorous, but it's already within reach, and it compounds.

Six questions for your AI vendor, or your own team, before you sign or before you assume you can't:

  1. Will you sign a BAA, and exactly which services, endpoints, and features does it cover?
  2. Is our data ever used to train your models, and is that in writing?
  3. What's the retention default, and what configuration gets us to zero retention while staying in scope?
  4. Which regions and deployment types are covered, and is the default endpoint one of them?
  5. If you host a third party's model, does coverage reach all the way down the sub-processor chain?
  6. For each model in your catalog: is it your product under your BAA, or a third party's product under theirs?

If you've taken a clinical AI workload from "blocked" to "running," I'd genuinely like to hear where the hard part actually was. My guess: it wasn't the regulation.

None of this is legal advice; all of it is verifiable in the documentation below.

BM

Bennett Malbon

Chief Technology Officer, T2 Group

Bennett Malbon leads the engineering architecture behind T2 Ignite and EZCARE, building clinical-AI pipelines that ship under HIPAA. His focus is the engineering backlog between a signed BAA and a production-grade, audit-ready workload.

Sources

Official documentation for the technical claims above, current as of July 2026:

  • Adoption-barrier survey: JAMIA 2025 health-system survey
  • HIPAA/BAA eligibility: Microsoft HIPAA/HITECH offering · AWS HIPAA-eligible services reference · Google Cloud HIPAA compliance
  • Azure OpenAI abuse monitoring, retention, and deployment types: Data, privacy, and security for Azure OpenAI · Abuse monitoring
  • Bedrock data retention modes, provider data sharing, and ZDR access: Amazon Bedrock data retention · Data protection in Amazon Bedrock
  • Gemini prompt caching default: Generative AI on Vertex AI data governance
  • Private connectivity: Bedrock via AWS PrivateLink · Azure AI services virtual networks / Private Link · Vertex AI Private Service Connect
  • Customer-managed keys: Bedrock encryption · Azure OpenAI CMK · Vertex AI CMEK
  • Prompt-level audit logging: Bedrock model invocation logging
  • Features outside the control envelope: Google generative AI compliance and security controls
  • Azure Foundry catalog split: Models sold directly by Azure · Models from partners and community
  • Bring-your-own open weights: Bedrock Custom Model Import
  • Proposed HIPAA Security Rule update (NPRM, not yet final): HHS Security Rule NPRM

Ready to explore a partnership?

Let's Connect

We use cookies. View our Privacy Policy