Most healthcare product teams start with the same question: is this model HIPAA compliant? It is the wrong question, and the answer is always no. HIPAA does not certify software. It sets obligations for covered entities and the business associates that handle protected health information (PHI) on their behalf. A model API can be part of a compliant system, or part of a breach. Which one depends on contracts, configuration and design decisions that sit almost entirely with you.
This guide walks through those decisions in the order we would make them when adding a language model to a patient portal, a clinician tool or a care-coordination app. It is written for founders, CTOs and product leads, not lawyers, so treat it as an engineering checklist to bring to your compliance counsel, not a substitute for them.
Step 1: Decide whether PHI needs to reach the model at all
The cheapest compliance work is the work you avoid. Before choosing a vendor, sort every planned AI feature into one of three tiers.
| Tier | What the model sees | Examples | Compliance load |
|---|---|---|---|
| 1. No patient data | Public or internal content only | Drafting patient education leaflets for clinician review, internal policy search, summarising published guidelines | Low. Normal vendor security review, no PHI controls needed if you enforce the boundary. |
| 2. De-identified data | Data that meets a HIPAA de-identification standard | Population analytics, operational reporting, model evaluation on de-identified cases | Medium. The hard part is proving the data is really de-identified. |
| 3. Identifiable PHI | Individual records, notes, messages, images | Visit summaries, inbox triage, prior authorisation drafts, intake chat | High. BAAs across the chain, minimum-necessary design, audit logging, review workflows. |
Our opinion: plenty of teams jump straight to tier 3 because the demo is impressive, then spend months on controls before learning whether anyone uses the feature. If a tier 1 feature can prove the value of AI in your product, ship it first. When you do move to tier 3, pick one workflow with a clear owner on the clinical side, not five.
Also be honest about scope. HIPAA applies to covered entities (health plans, clearinghouses and most providers) and their business associates. A purely consumer wellness app may sit outside it, though FTC and state health privacy rules can still apply. If you sell to providers or payers, assume HIPAA applies.
Step 2: Map every hop PHI will touch
An AI feature has more moving parts than a typical CRUD screen, and each one is a place PHI can land. Draw the data flow before you write code. A realistic tier 3 feature looks something like this:
- The client app sends a request (for example, "summarise this patient's last three visits").
- Your backend fetches records from the EHR or your own database.
- A retrieval layer pulls relevant chunks from a vector database or search index.
- A prompt is assembled and sent, possibly through an LLM gateway or proxy, to the model provider.
- The response is post-processed, shown to the user and maybe written back to the record.
- Along the way, logs, traces, error reports, analytics events and evaluation datasets capture some or all of the above.
Every vendor in that chain that creates, receives, maintains or transmits PHI needs a business associate agreement. Encryption does not get you out of it. The HHS Office for Civil Rights says a cloud service provider is a business associate "even if the CSP processes or stores only encrypted ePHI and lacks an encryption key for the data". The same guidance notes that business associates are directly liable for safeguarding ePHI under the Security Rule and for reporting breaches.
The hops teams forget are rarely the model API. They are the vector database hosted on a startup plan without a BAA, the LLM observability tool that stores full prompts, the error tracker that captures request bodies, and the product analytics script that records what users typed. Put each one in a table with three columns: does it see PHI, is there a signed BAA, and what is its retention period. Any row with "yes, no" is a blocker.
Step 3: Choose a model provider path and read the eligible-features list
Major providers now offer BAAs, but each agreement covers named services and features. The contract and the feature list have to be read together. Here is what the three most common paths say today, based on their own documentation. These are vendor statements and they change, so re-check them before you sign.
| Path | BAA available | Notable limits stated by the vendor |
|---|---|---|
| Anthropic first-party API | Yes, for HIPAA-ready organizations (Anthropic privacy center) | Messages API covered, including prompt caching and structured outputs. Batch API, Files API, Skills API, code execution and web fetch are listed as not covered and not accessible. Data sent to third parties via external MCP is not covered. Covered Models require 30-day retention and are unavailable with zero data retention unless expressly authorized. Several Messages API features are covered only under BAA versions accepted after 1 April 2026. |
| OpenAI API | Yes, for API healthcare customers (OpenAI business data page) | Eligible organizations accept the standard BAA in the API Platform (OpenAI help center). OpenAI points customers to a separate list of HIPAA-eligible products and configuration requirements before processing PHI. Qualifying organizations can configure retention, including zero data retention. |
| Amazon Bedrock | Yes, under the AWS BAA (AWS HIPAA eligible services) | Bedrock is on AWS's HIPAA eligible services list. AWS requires a signed BAA before using eligible services with PHI and states that customers must configure services consistent with HIPAA under a shared responsibility model. |
Three practical consequences follow. First, the feature list shapes your architecture. If a provider excludes its batch endpoint, your nightly summarisation job runs through the synchronous API and costs more; if it excludes file uploads, documents go into the prompt as text. Second, agentic features deserve extra care: a tool that fetches web pages or calls a third-party MCP server can move PHI outside the BAA boundary even when the model call itself is covered. Third, the date you signed matters. A BAA accepted two years ago may not cover features added since.
Which path should you choose? If your infrastructure and other BAAs already live on AWS, Bedrock keeps the vendor list short. If you need a specific model or feature that is only available first-party, sign directly with that provider. Either way, choose based on the eligible features you actually need, not on benchmark scores.
Working on something like this? See our Claude Agent Development →
Step 4: De-identify where you can, and be honest about what that means
De-identification is the main way to keep data out of tier 3, but it is frequently overclaimed. HHS recognises two methods:
- Safe Harbor. Remove 18 categories of identifiers and have no actual knowledge that the remaining information could identify the person. The list goes well beyond names and SSNs: all elements of dates except the year, geographic units smaller than a state, phone and fax numbers, emails, medical record and account numbers, device identifiers, URLs, IP addresses, biometric identifiers, full-face photos and "any other unique identifying number, characteristic, or code".
- Expert Determination. A qualified expert applies statistical or scientific methods and documents that the risk of re-identification is very small for the intended use.
The trap for AI products is unstructured text. HHS is explicit that the standard makes no distinction between structured fields and free text: an identifier must be removed wherever it appears. A clinical note that says "patient's daughter, a nurse at the county hospital, visited on the 14th" contains a date element and potentially identifying details that a name-and-SSN redactor will miss.
So use automated redaction as a risk-reduction layer, not as a compliance claim. Pre-model redaction is still worth doing in tier 3, because it lowers the impact if something goes wrong downstream. But unless you have followed one of the two HHS methods and documented it, treat the output as PHI and keep it inside the BAA boundary.
Step 5: Build prompts on the minimum necessary data
HIPAA's minimum necessary principle maps neatly onto good prompt engineering: send the model only what the task requires. In practice that means a few concrete design rules.
- Assemble context on the server. The client sends an intent and an encounter reference, never a raw record. Your backend decides which fields go into the prompt.
- Scope retrieval to the patient and the user. A vector index that mixes all patients is a cross-patient disclosure waiting to happen. Filter by patient and by the requesting user's permissions before similarity search, not after.
- Strip what the task does not need. A medication reconciliation prompt does not need the patient's address or insurance ID. Drop those fields before the prompt is built.
- Constrain outputs. Use structured outputs where your provider's BAA covers them, so the model returns fields your code validates rather than free text that may echo extra identifiers.
- Treat the model as a user with limited rights. If the model can call tools, give it read-only, patient-scoped tools and log every call. Prompt injection inside a patient message should never be able to fetch another patient's chart.
Step 6: Keep PHI out of logs, traces and evals
This is where we would spend the most review time. AI products generate far more telemetry than traditional apps, because teams need to inspect prompts and outputs to improve quality. That telemetry is full of PHI by default.
- Use tokenized references in general-purpose logs. Log a request ID, a pseudonymous patient or encounter token, user role, prompt version, model, latency, token counts and the reviewer's decision. Not the note text.
- Keep full prompts and responses in one controlled store, if at all. It should be covered by a BAA, access-controlled by role, audit-logged and subject to a written retention period.
- Scrub error trackers and analytics. Disable request body capture, mask input fields in session replay and check that no AI response text is sent as an analytics event property.
- Build evaluation sets from synthetic or properly de-identified cases. Copying a hundred real conversations into a spreadsheet for prompt testing is a disclosure. Synthetic cases written by clinicians, plus a small, access-controlled set of real cases inside your covered environment, is a safer pattern.
- Check your gateway. If you route traffic through an LLM proxy for cost tracking or failover, it sees every prompt. It needs the same treatment as the model provider.
Step 7: Design for clinical risk, not just privacy
HIPAA is about privacy and security. It says nothing about whether an AI summary left out an allergy. That risk sits with product design, and in some cases with other regulation.
If you are a covered entity under Section 1557 of the Affordable Care Act and you use AI to support patient care decisions, 45 CFR 92.210 applies. It prohibits discrimination on the basis of race, color, national origin, sex, age or disability through patient care decision support tools, creates an ongoing duty to make reasonable efforts to identify tools that use those variables, and requires reasonable efforts to mitigate the resulting discrimination risk. That is an inventory and governance task, and it is much easier if each AI feature is documented from the start.
On the product side, the patterns that work are the same ones we described in our guide to designing UX for AI features users trust: show the source records behind every generated statement, make the clinician's accept, edit or reject decision explicit, never write AI output into the chart without a human action, and design the failure state as carefully as the success state. Clinicians will forgive a tool that says "I could not find a medication list" much faster than one that invents one.
Step 8: Test the boundary before launch
Before any real PHI flows, run a short, deliberate test cycle focused on the edges rather than the happy path.
| Test | What you are checking | Pass condition |
|---|---|---|
| Data flow audit | Every service that received test PHI | Each one has a signed BAA and a known retention period |
| Feature allow-list | Calls to provider features outside the BAA | Blocked in code or rejected by the provider, never silently allowed |
| Cross-patient retrieval | Prompts that ask about other patients | Retrieval returns nothing outside the current patient and user scope |
| Prompt injection | Instructions hidden in patient messages or documents | Tools cannot be steered to fetch or send data outside scope |
| Telemetry scan | Logs, traces, error reports and analytics after a test run | No clinical text or direct identifiers outside the controlled store |
| Clinical review | A clinician-written set of hard cases | Known failure modes are documented and shown in the UI |
Then confirm the operational pieces: who reviews flagged outputs, how a suspected disclosure is escalated, and how your incident process connects to breach notification obligations under your BAAs. None of this is AI-specific, but AI features change the inventory of systems the process has to cover.
What this means for budget and timeline
We will not quote a number here, because it depends far more on your existing stack than on the AI feature. A team that already runs on a HIPAA-configured cloud with BAAs, audit logging and a working security review will add an LLM feature with modest extra effort. A team that has never mapped its PHI flows will find that the AI feature forces that work, and the compliance groundwork can take longer than the feature itself.
Two things reliably keep cost down. Start with a tier 1 or tier 2 feature to prove value. And make the boundary decisions in Step 2 and Step 6 before the first prototype, because retrofitting logging and retrieval scoping into a working demo is slower than building them in.
How Northell can help
Northell designs and builds healthcare software and AI features for product teams that need them to hold up in production. If you are planning an LLM feature for a healthcare product and want a second opinion on the data flow, feature scope or UX before you commit, get in touch. We are happy to start with a short review of what you already have.
