Form Extraction for Healthcare: Azure AI Document Intelligence or a Multimodal Model

Comparison of Azure AI Document Intelligence and a multimodal model for form extraction
On this page

Paper and scanned forms are still how a large share of clinical information arrives: referral letters, medication charts photographed on a ward, consent forms, insurance cards, questionnaires filled in by hand in a waiting room. Getting fields out of them used to mean a document extraction service with trained models. It can now also mean sending the page to a multimodal language model with a JSON schema and asking for the fields.

Both work on a demo. They differ in what they return beyond the values, and that difference decides how a clinical workflow can check the result. This article compares Azure AI Document Intelligence with a multimodal model, using Gemini on Vertex AI as the example, and proposes a hybrid design in which the evidence for every extracted value reaches a reviewer.

It is the third design article in a set. The abstraction leaks article covers structured output differences between providers, and the clinical content filters article covers why an empty field must never be read as a negative finding. Both apply here.

ℹ️
TEST SCOPE

This is a design article. No document was sent to either service and nothing was measured. Statements about the services come from the documentation pages listed under References. The forms described are invented and no real patient document was used. The design is untested guidance and is not clinically validated.

The problem

Take three invented examples. A two-page GP referral form with printed labels, typed answers and a handwritten note in the margin. A medication administration chart, photographed at an angle, with a table of drugs, doses and times and initials in each cell. A patient questionnaire with tick boxes and short handwritten answers. The fields are the easy part. The hard parts are the handwriting, the tables, the tick boxes, and knowing which values to trust.

In a clinical workflow the extracted values are not the end product. A person accepts them into a record, and that person needs to see where each value came from and how sure the system is. An extraction approach that returns only values pushes all of that checking back onto the reviewer, who then has to re-read the whole form.

Document Intelligence

Document Intelligence is a set of trained extraction models. The current generally available API version is 2024-11-30 (v4.0). Microsoft groups the models as follows.

  • Read extracts text lines and words with their locations and detected languages.
  • Layout adds tables, including merged cells and column headers, selection marks with their state, paragraphs with roles such as titles and headings, figures and sections. It can return Markdown, with tables rendered as HTML so merged cells survive. An optional key-value pairs feature extracts fields from forms without training.
  • Prebuilt models cover specific document types. The health-related one is a model for US health insurance cards; most others are financial, tax, identity and mortgage documents.
  • Custom models are trained on labelled samples: custom template for fixed layouts, custom neural for structured and semi-structured documents, composed models that combine several, and custom classification to route document types. Microsoft’s comparison lists a minimum of five labelled samples.

What sets it apart is the evidence attached to each value. Every word, line and selection mark comes with a polygon on the page and a confidence score. Fields extracted by models come with a confidence that Microsoft describes as an estimated probability between 0 and 1. From API version 2024-11-30, custom models also return confidence for tables, rows and cells. Layout flags each text line as handwritten or not, with its own confidence, though Microsoft documents that handwriting classification for Latin scripts only.

Microsoft’s own guidance positions Document Intelligence for high-volume, deterministic extraction, and points scenarios that need generative AI or highly varied templates to Azure Content Understanding, which this article does not cover.

Multimodal model

A multimodal model reads the page as an image or a PDF and answers a prompt. With a response schema, it returns the fields as JSON. Gemini on Vertex AI accepts PDF documents up to 50 MB through the API, with page and file limits that vary by model. Google’s guidance is to place a single PDF before the text of the prompt, split long documents, and prefer PDFs with machine-readable text to scanned images.

Google is also direct about the limits. Its document understanding page says the models are not precise at locating text or objects in PDFs, and that they might hallucinate when interpreting handwritten text. The page read does not describe bounding boxes or confidence scores for extracted values. The schema constraints and their differences between providers, covered in the abstraction article, apply as well.

What a multimodal model offers in return is flexibility. It needs no labelled samples, it can follow instructions about context, and it can read a free-text paragraph and answer a question about it, which a field extractor is not built to do.

Comparison

AspectDocument IntelligenceMultimodal model (Gemini on Vertex AI)
SetupPrebuilt models, or custom models trained on labelled samplesA prompt and a response schema; no training
Evidence per valuePolygon on the page and a confidence scoreNone described on the page read
Words and textEach word and line with location and confidenceText as part of the answer
TablesRows, columns, merged cells, headers; cell confidence for custom modelsAs far as the schema and the model’s reading allow
Tick boxesSelection marks with state, location and confidenceInferred from the image
HandwritingHandwritten lines flagged with confidence (Latin scripts)Google warns of possible hallucination on handwriting
Variable layoutsCustom neural for semi-structured documents; Content Understanding beyond thatHandles variation through instructions
VersioningAPI version and model ID; a custom model is fixed once trainedA model version that changes when the provider updates it
Billing unitPer 1,000 pages, with separate rates per featureTokens; pages tokenised according to a media resolution setting
Processing locationRegion of the resource; input and results kept 24 hoursDepends on the endpoint; the global endpoint gives no residency guarantee
Comparison of Azure AI Document Intelligence and a multimodal model for form extraction: evidence per value, tables, tick boxes, handwriting, layout variation, versioning, billing and processing location
The difference that matters most is the evidence returned with each value.

Confidence and evidence

A confidence score and a polygon change what a reviewer does. With them, a review screen can show the extracted value next to the highlighted region of the scanned page, and only for the values below a threshold. Without them, the reviewer has to find the value on the page themselves, for every value, or trust it.

Microsoft is careful about what the score means. Field confidence for custom models reflects the model’s confidence in the position of the value, and it recommends checking the underlying word or selection mark confidence for a composite view. It gives no numeric threshold, says confidence can decide between automatic acceptance and human review, and recommends custom model accuracy close to 100% for sensitive cases such as medical records. Any threshold is therefore a local decision, calibrated on a sample of the organisation’s own forms and set more strictly for high-risk fields such as drug, dose and allergy.

A multimodal model can be asked to report how sure it is, but a number the model writes in its own answer is a generated value like any other, not a calibrated probability. It should not be used as a review threshold without evidence that it predicts errors on the forms in question.

Handwriting and tables

Handwriting and tables are where clinical forms differ from invoices, and where the two approaches fail differently. Layout marks a line as handwritten and gives each word a confidence, so a low-confidence handwritten dose stands out. A multimodal model returns a fluent answer whether or not it could read the writing, and Google’s own documentation warns that it might hallucinate there. A plausible but wrong dose is the failure to design against.

For tables, Layout returns the grid with row and column spans, so a medication chart’s structure survives even where a cell is hard to read. A multimodal model can return a table that matches the schema with rows merged, shifted or invented, and nothing in the response says so. Where structure carries meaning, as in drug, dose and time, the structure needs its own check.

Proposed design

The proposal uses each tool for what its documentation says it does well, and routes every uncertain value to a person with the evidence beside it.

%%{init: {'theme':'base','themeVariables':{'primaryColor':'#FFF3E0','primaryTextColor':'#232F3E','primaryBorderColor':'#FF9900','lineColor':'#232F3E','clusterBkg':'#F8FAFC','clusterBorder':'#CBD5E1','edgeLabelBackground':'#FFFFFF','nodeTextColor':'#232F3E'}}}%%
flowchart TD
    IN["Scanned form"] --> CL["Classify document type"]
    CL --> DI["Document Intelligence:
layout or custom model"] DI --> F["Fields with confidence
and page regions"] DI --> FT["Free-text sections"] FT --> MM["Multimodal model:
schema for free-text questions"] F --> R{"Below threshold,
handwritten, or missing?"} MM --> R2["Always reviewed:
no calibrated confidence"] R -- "yes" --> Q["Review queue:
value beside highlighted region"] R -- "no" --> A["Accepted, with source recorded"] R2 --> Q Q --> A A --> REC["Write to the clinical record"] style DI fill:#EBF8FF,stroke:#0073BB,stroke-width:2px,color:#004C74 style MM fill:#F3E8FF,stroke:#7C3AED,stroke-width:2px,color:#4C1D95 style Q fill:#FFF3E0,stroke:#FF9900,stroke-width:2px,color:#232F3E style REC fill:#F0FFF4,stroke:#1D8102,stroke-width:2px,color:#145202
  • Classify first. A custom classification model, or a simple rule where the form carries a code, sends each document type to the extractor trained for it.
  • Extract fields with Document Intelligence. Layout with key-value pairs to start, a custom model once a form type has enough labelled samples. Keep the confidence and the polygon with every value.
  • Use the multimodal model for the questions a field extractor cannot answer. A free-text reason for referral, summarised against a schema, is a good fit. Its output is always reviewed, because it carries no calibrated confidence.
  • Route by evidence. A value goes to review if its confidence is below the threshold for that field, if the line is handwritten, if the field is missing, or if two sources disagree. A missing value is shown as not extracted, never as empty.
  • Show the source. The review screen shows each value next to the highlighted region of the page, so the reviewer checks the value, not the whole form.
  • Record provenance. The accepted record keeps the model ID, API or model version, confidence, and whether a person changed the value. That is the audit trail when a value is later questioned.
Hybrid extraction design: classification, Document Intelligence for fields with confidence and regions, a multimodal model for free-text sections, routing to human review on low confidence, handwriting, missing values or disagreement, and provenance on every accepted value
Every uncertain value reaches a person with its evidence.
%%{init: {'theme':'base','themeVariables':{'primaryColor':'#FFF3E0','primaryTextColor':'#232F3E','primaryBorderColor':'#FF9900','lineColor':'#232F3E','signalColor':'#0073BB','signalTextColor':'#0F172A','actorBkg':'#FFF3E0','actorBorder':'#FF9900','actorTextColor':'#232F3E','noteBkgColor':'#FFF8E7','noteBorderColor':'#FF9900','noteTextColor':'#232F3E'}}}%%
sequenceDiagram
    participant X as Extraction service
    participant R as Reviewer
    participant C as Clinical record
    X->>R: Dose 5 mg, handwritten, confidence 0.62, region on page 2
    Note over R: Sees the value beside the highlighted region
    R->>X: Corrected to 0.5 mg
    X->>C: Dose 0.5 mg, source page 2, changed by reviewer
    X->>C: Name and date of birth, confidence above threshold, accepted

Cost

No prices are quoted here, because both price lists change and nothing was measured. The structures differ. Document Intelligence bills per 1,000 pages, with separate rates for read, for layout and prebuilt models, for custom extraction, for custom generative extraction and for classification, plus charges for add-on features; neural model training is billed per hour after a free allowance, and commitment tiers trade a monthly fee for a page allowance. A free tier covers 500 pages a month. A multimodal model bills tokens: on Vertex AI, the tokens a PDF page consumes depend on a media resolution setting, and the response schema counts toward input tokens. The design above runs the multimodal model only on the sections that need it, which bounds that part of the cost.

Data residency

Microsoft states that Document Intelligence processes input in the region where the resource was created, keeps input and results in that region for 24 hours, then deletes them, and offers an API to delete a result sooner. Custom model training data stays in the customer’s own blob storage. On Vertex AI, data at rest stays in the chosen location, while the endpoint decides where processing happens: Google states that global endpoints give no data residency guarantee, and regional endpoints keep processing within the jurisdiction of that region. For health data, the endpoint is a compliance decision, set once in configuration and not changed for capacity reasons.

Testing the design

  • A synthetic form set. Invented forms for each type, filled in by several people in different handwriting, scanned and photographed, with the correct values recorded. Never real patient documents.
  • Measure per field, not per form. Error rates for drug, dose, allergy and date matter more than an overall score. Calibrate each field’s threshold from these results.
  • Test the routing without the services. Recorded responses with low confidence, handwriting, missing fields and disagreements, run through the routing and the review screen.
  • Rerun on every version change. A new API version, a retrained custom model or a new multimodal model version gets the full form set before it is used.

Production considerations

  • Pin versions. Use a specific Document Intelligence API version and custom model IDs, and a specific multimodal model version. Record both on every extracted value.
  • Delete results promptly. Call the delete API once results are stored, rather than relying on the 24-hour retention.
  • Keep extraction out of logs. Log document identifiers, field names and confidences, not values.
  • Put extraction in the clinical safety case. A wrong dose accepted from a form is a hazard with controls: thresholds, review, provenance and testing.
  • Plan for the forms that will not extract. Poor scans and unusual layouts will always exist. A route to manual entry, visible in the workflow, is part of the design.

Decision framework

Your situationApproachWhy
Fixed form layouts in volumeDocument Intelligence custom template or neural modelConfidence and regions per value, trained on your forms
US health insurance cardsThe prebuilt health insurance card modelA prebuilt model exists for that document
Tables such as medication chartsDocument Intelligence layout or custom modelRows, columns and cells with structure and confidence
Handwritten values that matter clinicallyDocument Intelligence with review of every handwritten valueHandwriting flagged; multimodal models may hallucinate
Free-text sections and varied documentsMultimodal model with a schema, always reviewedFlexible, but no calibrated confidence
Strict residency requirementsA regional resource or endpoint, fixed in configurationThe global endpoint gives no residency guarantee

Key takeaways

  • Document Intelligence returns a page region and a confidence with each value; the multimodal model documentation read describes neither.
  • Google warns that its models might hallucinate on handwriting and are imprecise at locating text, which matters most for doses and tables.
  • Use Document Intelligence for fields, tables and tick boxes, and a multimodal model for free-text questions, with its output always reviewed.
  • Route values to review by confidence, handwriting, absence or disagreement, and show the reviewer the region of the page.
  • Pin versions, record provenance on every value, and choose processing locations as a compliance decision.

If you have run clinical forms through either approach and measured field-level error rates, I would like to hear how they compared.

πŸ“
HOW THIS WAS CHECKED

This is a design article and nothing was run. Statements about the services come from the documentation pages listed below: Document Intelligence model overview, layout model, accuracy and confidence, model selection guidance, data privacy and pricing; Vertex AI document understanding and data residency. Where a page did not cover a point, for example bounding boxes or confidence for Gemini, the article says so rather than filling the gap. No document was sent to either service, no accuracy or cost was measured, and the forms described are invented. Nothing here is clinically validated.

Glossary

  • Document Intelligenceβ€” Azure’s service of trained models for extracting text, layout and fields from documents.
  • Layout modelβ€” The Document Intelligence model that returns text, tables, selection marks and document structure.
  • Custom template and neural modelsβ€” Document Intelligence models trained on labelled samples, for fixed and for semi-structured layouts.
  • Selection markβ€” A tick box or radio button, returned with its state and location.
  • Polygonβ€” The coordinates of a region on the page where a word, line or value was found.
  • Confidence scoreβ€” An estimated probability that an extracted value is correct.
  • Multimodal modelβ€” A language model that accepts images and documents as input as well as text.
  • Provenanceβ€” The record of where a value came from: source page and region, model, version, confidence and any human change.

References & Further Reading


Discover more from C4: Container, Code, Cloud & Context

Subscribe to get the latest posts sent to your email.

Written by

Nithin Mohan TK

Hello Namaste! πŸ‘‹πŸ» I'm an accomplished Enterprise Solution Architect & Engineering Leader with 20+ years of progressive experience architecting, building, and scaling transformative enterprise solutions. ⚜️ Solution Architect / AI & Cloud πŸ“Œ Galway, Ireland

Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.