On this page
Most teams that run on two clouds end up wanting one interface for chat completion: a message list goes in, text or a tool call or a JSON object comes out, and Azure OpenAI and Gemini on Vertex AI become configuration. The interface is easy to write. What it hides is the problem, because the two services agree on the shape of a request and disagree on what happens at the edges: which schemas are honoured, how a blocked request is reported, what a 429 means, and where a setting lives.
This article goes through those edges, from the providers’ own documentation, and proposes a design for an abstraction that admits them instead of hiding them. It is design guidance. Nothing in it was run against either cloud.
This is a design article. Every statement about Azure OpenAI or Vertex AI comes from the documentation pages listed under References, as published when they were read. No request was sent to either service, no latency or cost was measured, and the design has not been built or tested. Treat it as a checklist to verify against your own deployments.
The problem
A provider abstraction is usually judged on the happy path: same prompt, similar answer, one code path. The cost shows up elsewhere. A JSON schema that is enforced on one side and partly ignored on the other produces records that pass on Azure and fail validation on Vertex, or the reverse. A safety block that arrives as an exception from one provider and as a normal response with an empty body from the other ends up retried by one code path and stored as an empty answer by the other. A retry policy tuned for one provider’s throttling misbehaves on the other’s.
None of this is a defect in either service. They are different products with different contracts. The defect is an interface that promises they are the same.
%%{init: {'theme':'base','themeVariables':{'primaryColor':'#FFF3E0','primaryTextColor':'#232F3E','primaryBorderColor':'#FF9900','lineColor':'#232F3E','clusterBkg':'#F8FAFC','clusterBorder':'#CBD5E1','edgeLabelBackground':'#FFFFFF','nodeTextColor':'#232F3E'}}}%%
flowchart TD
APP["Application code"] --> PORT["Chat port:
messages, tools, schema"]
PORT --> AZ["Azure OpenAI adapter"]
PORT --> VX["Vertex AI adapter"]
AZ --> L1["Schema dialect"]
VX --> L1
AZ --> L2["Safety outcome"]
VX --> L2
AZ --> L3["Throttling meaning"]
VX --> L3
AZ --> L4["Where settings live"]
VX --> L4
L1 --> RISK["Same call, different guarantees"]
L2 --> RISK
L3 --> RISK
L4 --> RISK
style PORT fill:#EBF8FF,stroke:#0073BB,stroke-width:2px,color:#004C74
style RISK fill:#FEF2F2,stroke:#D13212,stroke-width:2px,color:#991B1B
style L1 fill:#F3E8FF,stroke:#7C3AED,stroke-width:2px,color:#4C1D95
style L2 fill:#F3E8FF,stroke:#7C3AED,stroke-width:2px,color:#4C1D95
style L3 fill:#F3E8FF,stroke:#7C3AED,stroke-width:2px,color:#4C1D95
style L4 fill:#F3E8FF,stroke:#7C3AED,stroke-width:2px,color:#4C1D95Structured output
Both services can constrain a response to a JSON schema. Both accept a subset of JSON Schema, and the subsets point in different directions.
| Aspect | Azure OpenAI | Vertex AI (Gemini) |
|---|---|---|
| How it is requested | response_format with a JSON schema in Chat Completions; text.format in Responses | responseMimeType: application/json plus responseSchema |
| Required fields | Every property must be listed in required; optional fields are written as a union with null | Fields are optional by default; the model may skip a field not marked required |
| Extra properties | additionalProperties: false on every object | Not a listed field |
| String constraints | minLength, maxLength, pattern, format not supported | format supported for date, date-time, duration and time |
| Number and array constraints | minimum, maximum, minItems, maxItems and others not supported | minimum, maximum, minItems, maxItems supported |
| Unsupported keywords | Listed as unsupported | Ignored; the request still succeeds |
| Size limits | Up to 100 object properties and five levels of nesting | No fixed numbers; a complex schema can fail with HTTP 400 InvalidArgument |
| Key order | Follows the order in the schema | Not strictly followed; propertyOrdering sets it |
| Interaction with tools | Not supported with parallel function calls; set parallel_tool_calls to false | Not stated on the page read |
| Token accounting | Not stated on the page read | The schema counts toward the input token limit |
Three of those rows cause most of the trouble.
- Optionality is inverted. On Azure, a field the model may leave out has to be written as required and nullable. On Vertex, a field you need has to be marked required or it may not appear. A schema written for one and sent unchanged to the other changes meaning, not just syntax.
- Constraints are enforced on one side only. A
maxItemsor aminimumshapes the output on Vertex and is not supported on Azure. Apatternis supported on neither. Any constraint that matters has to be checked after the response arrives, whichever provider produced it. - Failure is silent on one side and loud on the other. Vertex ignores fields it does not support, so a request succeeds with a weaker guarantee than the schema suggests. Azure documents the same keywords as unsupported. A schema that grows over time can cross Vertex’s complexity limit and start returning 400 for a request that worked last month.
Refusals are a further case. Azure’s documentation shows a separate refusal field on the message in Chat Completions, and a content item of type refusal in Responses, to be checked before the parsed object is read. A port that returns “the parsed object” with no room for “the model declined” loses that case.
Content safety
Both providers screen prompts and responses for harmful content. They differ in what they screen for, where the thresholds are set, and above all in how a block reaches the caller.
| Aspect | Azure OpenAI | Vertex AI (Gemini) |
|---|---|---|
| Configurable categories | Hate and fairness, sexual, violence, self-harm | Hate speech, harassment, sexually explicit, dangerous content |
| Levels | Severity: safe, low, medium, high | Probability and severity, each discretized to negligible, low, medium, high; severity method is the default |
| Where thresholds are set | A content filter configuration attached to the deployment | safetySettings on each request |
| Turning filtering off | Only for customers approved through a review form; annotate-only also needs approval | BLOCK_NONE and OFF thresholds in the request |
| Filters that cannot be configured | Not covered on the page read | Filters for child sexual abuse material and sensitive personal data |
| Blocked prompt | HTTP 400, error code content_filter, param prompt | A response with promptFeedback.blockReason set |
| Blocked response | HTTP 200, finish_reason: content_filter, content withheld | finishReason of SAFETY, RECITATION, SPII or PROHIBITED_CONTENT; candidate content cleared |
| Filtering unavailable | HTTP 200, unfiltered, with a content_filter_error inside content_filter_results | Not covered on the page read |
The last three rows are the abstraction leak. A blocked prompt on Azure is an error response, and most HTTP clients and SDKs turn it into an exception. A blocked prompt on Vertex is a successful response with a field set. A blocked completion is a normal response on both, with different field names and different values. If the port maps only exceptions to errors, an Azure prompt block becomes a failure that a retry policy will retry, and a Vertex prompt block becomes an empty answer.
The Azure case of filtering being unavailable deserves its own line in any design. The documentation says the request completes without filtering and reports it inside the response. A port that drops content_filter_results cannot tell a screened answer from an unscreened one.
Both sets of categories were written for general content. Clinical notes discuss self-harm, violence and the body as a matter of course, and a filter that is right for a consumer chat can block a discharge summary. Whether a block is a safety outcome or a false positive is a clinical governance question, and the abstraction should make every block visible and countable rather than turn it into a generic error.
Throttling and capacity
Both services answer HTTP 429 when they cannot take a request. The status code is the same and the meaning is not.
| Aspect | Azure OpenAI (standard deployments) | Vertex AI (Gemini, pay as you go) |
|---|---|---|
| Unit of capacity | Tokens per minute assigned to a deployment, from a per-region, per-model subscription quota | Shared capacity; no reserved throughput |
| Request limit | Requests per minute set in proportion to tokens per minute, evaluated over windows of typically 1 or 10 seconds | Not a separate fixed limit on the page read |
| What a 429 means | The deployment’s limit was reached, or the system is under high demand | Temporary contention for a shared resource, not a fixed quota |
| What the response carries | x-ratelimit-* headers on every response; retry-after-ms on a 429 | Not stated on the page read |
| Documented remedies | Honour retry-after-ms with jitter; more quota; provisioned deployments | Exponential backoff; the global endpoint; smoother traffic; Provisioned Throughput |
| Counted toward the limit | Actual input and output tokens; failed requests count too | Not stated on the page read |
A single retry policy over both is either too eager for one or too patient for the other. Azure tells the caller how long to wait, and a caller that ignores retry-after-ms and retries on its own schedule spends requests that count against the same limit. Vertex gives no fixed limit to plan against, and the documented answer to repeated 429s is a different endpoint or a different purchasing model, not a longer wait.
Retries also interact with agents. A model call retried after a partial stream, or after a tool call it requested has already run, is the duplicate-effect problem from the crash and resume article in another form. The retry belongs below the agent loop, on whole model calls, never on a step whose tools have acted.
Endpoints and authentication
- Model identity. On Azure the request names a deployment, which carries the model, its version, its quota and its content filter. On Vertex the request names a model in a project and a location. The same logical “model” in configuration maps to an operational object on one side and a catalogue entry on the other.
- Credentials. Vertex’s OpenAI-compatible endpoint takes a Google Cloud access token as the API key, and its documentation notes that access tokens live for one hour by default. A client built once at start-up with a token as its key fails an hour later unless the token is refreshed.
- Location. Azure quota is per region and per model. Vertex recommends the
globalendpoint for capacity. Where data residency matters, which endpoint a request is allowed to use is a policy decision that the abstraction must not make implicitly.
Vertex also offers a Chat Completions endpoint that the OpenAI libraries can call, under /projects/{project}/locations/{location}/endpoints/openapi. It is tempting to treat that as the abstraction. It removes the wire-format difference and keeps every other difference in this article: Gemini settings still have to travel in extra_body, the safety outcomes are still Gemini’s, and the throttling is still Vertex’s. It is a reasonable adapter. It is not a reason to skip the design below.
Proposed design
The aim is not to make the providers identical. It is to make the differences explicit in one place, so that application code handles each outcome once and the operations team can see which provider produced it.

An outcome type, not an exception. Every call returns one of a closed set of outcomes. Exceptions are kept for transport failures and bugs.
| Outcome | Azure OpenAI signal | Vertex AI signal | Typical handling |
|---|---|---|---|
| Completed | finish_reason stop or tool calls | finishReason STOP | Validate, then use |
| Truncated | finish_reason length | finishReason MAX_TOKENS | Do not parse as complete JSON |
| Refused | refusal on the message or a refusal output item | Not documented as a separate signal on the pages read | Show or route; do not retry unchanged |
| Blocked on input | HTTP 400 with code content_filter | promptFeedback.blockReason | Count, review, never retry unchanged |
| Blocked on output | finish_reason content_filter | finishReason SAFETY, RECITATION, SPII or PROHIBITED_CONTENT | Count by reason, review |
| Unscreened | content_filter_error in the results | Not documented | Decide by policy: accept, hold or reject |
| Throttled | HTTP 429 with retry-after-ms | HTTP 429 | Provider-specific retry, then fail over or shed |
| Invalid request | Other HTTP 400 | HTTP 400, including schema complexity | Fix the request; alert |
A capability profile per adapter. Each adapter declares what it can do: which schema keywords it enforces, whether structured output and parallel tool calls can be combined, where safety thresholds are set, and whether the outcome set above is fully reported. Application code that needs a guarantee asks for it, and the port refuses a request it cannot honour rather than quietly sending a weaker one.
A schema compiler, not a schema. The application owns one schema in full JSON Schema. Each adapter compiles it to its provider’s dialect and reports what was lost: constraints removed for Azure, fields Vertex will ignore, optional fields rewritten as nullable. The full schema is then validated against every response. The model’s constrained output is a strong hint; the validator is the contract.
%%{init: {'theme':'base','themeVariables':{'primaryColor':'#FFF3E0','primaryTextColor':'#232F3E','primaryBorderColor':'#FF9900','lineColor':'#232F3E','clusterBkg':'#F8FAFC','clusterBorder':'#CBD5E1','edgeLabelBackground':'#FFFFFF','nodeTextColor':'#232F3E'}}}%%
flowchart TD
S["Application schema
full JSON Schema"] --> CA["Compile for Azure:
all required, nullable optionals,
drop unsupported keywords"]
S --> CV["Compile for Vertex:
mark required fields,
keep supported constraints"]
CA --> RA["Report: what was dropped"]
CV --> RV["Report: what will be ignored"]
RA --> CALL["Provider call"]
RV --> CALL
CALL --> VAL{"Validate the response
against the full schema"}
VAL -- "valid" --> USE["Use the result"]
VAL -- "invalid" --> REJ["Outcome: invalid output,
logged with provider"]
style S fill:#EBF8FF,stroke:#0073BB,stroke-width:2px,color:#004C74
style VAL fill:#F3E8FF,stroke:#7C3AED,stroke-width:2px,color:#4C1D95
style USE fill:#F0FFF4,stroke:#1D8102,stroke-width:2px,color:#145202
style REJ fill:#FEF2F2,stroke:#D13212,stroke-width:2px,color:#991B1BSafety settings as named policies. Configuration names a policy, such as “standard” or “clinical documentation”. The Azure adapter maps it to the content filter configuration that must be attached to the deployment, and checks at start-up that it is. The Vertex adapter maps it to safetySettings sent on every request. The policy name goes into the log next to every block.
Retry inside the adapter. The Azure adapter honours retry-after-ms with jitter. The Vertex adapter uses exponential backoff. Neither retries a block, a refusal or an invalid request. Fail-over to the other provider happens above the adapters, only for calls whose schema and policy the other adapter can honour.

Testing the design
Because this article ran nothing, the design comes with the tests that would have to pass before relying on it. Most can run without either cloud.
- Compiler tests. For each schema in use, the compiled form for each provider, and the report of what was lost, checked in as a snapshot. A change to the application schema that makes a provider drop a constraint shows up in review.
- Outcome mapping tests. Recorded or hand-written responses for each row of the outcome table, per provider, run through the adapter. These need no network.
- Contract tests against each cloud. A small scheduled suite that sends a prompt each provider should block, a schema near Vertex’s complexity limit, and a burst that should be throttled, and checks the adapter reports the right outcome. This is the only part that needs credentials, and the only part that notices when a provider changes behaviour.
- Counters per outcome and provider. In production, a rising count of “blocked on output” on one provider after a model version change is the signal that matters, and it is invisible if blocks are folded into errors.
Cost
No costs were measured. Two structural points follow from the documentation. On Vertex the response schema counts toward input tokens, so a large schema is paid for on every call. On Azure, failed requests count toward the per-minute limit, so a retry policy that ignores retry-after-ms reduces the capacity it is trying to wait for. The abstraction itself adds a validation pass per response, which is small next to a model call.
Production considerations
- Log the provider, the deployment or model, the policy name and the outcome on every call. Without them, a difference in behaviour between clouds cannot be told from a difference in input.
- Treat “unscreened” as a decision. For regulated content, holding an Azure response whose filter did not run may be the right default.
- Do not fail over on blocks. Sending a prompt that one provider blocked to the other provider until something answers is a policy bypass, whatever the intent.
- Pin model versions and rerun contract tests on change. Schema support, thresholds and defaults are properties of models as much as of services.
- Keep data residency out of fail-over logic. Which regions and endpoints a workload may use is fixed by policy, and the router only chooses among those.
Decision framework
| Your situation | Approach | Why |
|---|---|---|
| One provider in production, the other a possibility | Single adapter, outcome type and schema validation now; second adapter later | The outcome type and validator pay off even with one provider |
| Both providers serve the same feature | Full design: compiler, outcome mapping, named policies, per-adapter retry | Every leak in this article reaches users |
| Fail-over for availability | Only between adapters whose capability profiles both satisfy the call | A fail-over to a weaker guarantee is a silent downgrade |
| Structured output with strict constraints | Validate every response against the full schema, on both providers | Neither dialect enforces everything |
| Clinical or other sensitive text | Named safety policies, every block counted and reviewable | Filters tuned for general content will block legitimate text |
| Considering the OpenAI-compatible Vertex endpoint | Use it as the Vertex adapter’s transport if convenient | It unifies the wire format, not the behaviour |
Key takeaways
- Azure OpenAI and Vertex AI agree on the shape of a chat request and differ on schema support, safety reporting, throttling and where settings live.
- Structured output optionality is inverted between them, and each enforces constraints the other does not. Validate every response against the full schema.
- A blocked prompt is an HTTP 400 on Azure and a successful response with a field set on Vertex. Model outcomes as a closed type, not as exceptions.
- A 429 means a deployment limit on Azure and shared contention on Vertex. Retry inside each adapter, by that provider’s rules.
- The abstraction is worth having when it names the differences. One that hides them moves the failures into production.
If you run the same workload on both clouds and have found a difference this article does not cover, leave a comment.
This is a design article and nothing in it was run. The statements about each service were taken from the provider documentation pages listed below: Azure OpenAI structured outputs, content filtering, and quota and rate limits; Vertex AI structured output, safety and content filters, OpenAI compatibility, and standard pay-as-you-go capacity. Where a page did not cover a point, the tables say so instead of filling the gap. No request was sent to Azure OpenAI or Vertex AI, no sample was built, and no latency, cost or block rate was measured. Provider behaviour changes with model versions and service updates; check each point against the current documentation and your own deployments before relying on it.
Glossary
- Abstraction leakβ A difference between implementations that callers of a shared interface still have to know about.
- Adapterβ The component that translates the shared interface to one provider’s API.
- Capability profileβ A declaration of what an adapter can guarantee, checked before a call is sent.
- Deploymentβ In Azure OpenAI, a named instance of a model with its own quota and content filter.
- Structured outputβ A response constrained to a supplied JSON schema.
- Content filterβ Azure’s classification and blocking of prompts and responses by harm category and severity.
- Safety settingsβ Vertex AI’s per-request thresholds for blocking by harm category.
- Refusalβ A model’s explicit decision not to answer, reported separately from the content.
- TPM and RPMβ Tokens per minute and requests per minute, the units of Azure OpenAI quota.
- Provisioned throughputβ Capacity reserved in advance, offered by both providers, as opposed to shared pay-as-you-go capacity.
References & Further Reading
- → Azure OpenAI: Structured outputsβ Supported schema subset, limits, required fields and parallel function calls
- → Azure OpenAI: Content filteringβ Categories, severity levels, how blocks and filter errors are reported, and modified filtering
- → Azure OpenAI: Quota and rate limitsβ Tokens and requests per minute, evaluation windows, 429 responses and rate-limit headers
- → Vertex AI: Structured outputβ responseSchema, supported fields, property ordering and complexity errors
- → Vertex AI: Safety and content filtersβ Harm categories, thresholds, scoring methods and blocked-response reasons
- → Vertex AI: OpenAI compatibilityβ Calling Gemini through the OpenAI libraries, endpoint form and access tokens
- → Vertex AI: Standard pay-as-you-go capacityβ Shared capacity, what a 429 means and the documented remedies
Discover more from C4: Container, Code, Cloud & Context
Subscribe to get the latest posts sent to your email.