Almost nothing reads clinical data straight from the FHIR server that holds it. A care-coordination app keeps its own task list, a reporting database keeps flattened copies, a search index keeps whatever makes search fast. Each of those is a second copy of part of the record, and each is wrong from the moment the record changes until something tells it.
FHIR has a standard answer, the Subscription resource. Azure Health Data Services has its own: change events published through Event Grid. Either one gets a notification to your code. Neither makes the downstream copy correct, because notifications arrive more than once, out of order, late, and sometimes not at all, and the changes they announce include corrections and deletions that a naive consumer turns into wrong data that looks current.
This article builds the consumer: an Azure Function on .NET 10 that keeps a small projection of blood pressure readings in step with the FHIR service. The whole design rests on one rule, that the version stored for a resource only ever moves forward, enforced in a single SQL statement. The rest of the article is about why that rule is enough for duplicates and reordering, and what it does not cover.
The processing rule was tested in C# and in Python against the same scripted scenario. The Function App was then built on .NET 10 and run in Docker with a PostgreSQL store and a real FHIR server, and a replay script drove it with events in the Azure Health Data Services schema. Those events were constructed by the script, not published by Event Grid, and nothing was deployed to Azure. The details are under How this was checked.

Subscriptions or events
| FHIR Subscription | Azure Health Data Services events | |
|---|---|---|
| What it is | A resource in the FHIR specification. In R4 the client posts a search query as the criteria and the server notifies a channel when a matching resource changes. R5 replaces criteria with server-defined topics, and an implementation guide backports that model to R4 | A feature of the Azure service. Changes are published to an Event Grid system topic, and you attach subscriptions to it |
| What triggers it | Whatever the criteria or topic selects | A resource created, updated or soft deleted |
| What arrives | R4 can send the resource or an empty ping. The topic-based model sends a notification bundle | Identifiers only: resource type, id and version |
| Filtering | FHIR search expressions | Event Grid filters on event type and data fields |
| Delivery | Defined by the server; the specification leaves guarantees open | At least once, no ordering guarantee, retried for up to 24 hours |
| Portability | Standard, but server support varies widely | Azure only |
On Azure the choice is made for you: events are the mechanism the FHIR service documents. Elsewhere, Subscriptions are the portable option. The consumer in this article works with either, because it treats a notification as a hint that something changed and reads the truth from the server. Only the parsing of the envelope differs.
What an event tells you
This is the sample from Microsoft’s documentation, in the Event Grid schema.
{
"id": "e4c7f556-d72c-e7f7-1069-1e82ac76ab41",
"topic": "/subscriptions/{subscription-id}/resourceGroups/{resource-group-name}/providers/Microsoft.HealthcareApis/workspaces/{workspace-name}",
"subject": "{fhir-account}.fhir.azurehealthcareapis.com/Patient/e0a1f743-1a70-451f-830e-e96477163902",
"data": {
"resourceType": "Patient",
"resourceFhirAccount": "{fhir-account}.fhir.azurehealthcareapis.com",
"resourceFhirId": "e0a1f743-1a70-451f-830e-e96477163902",
"resourceVersionId": 1
},
"eventType": "Microsoft.HealthcareApis.FhirResourceCreated",
"dataVersion": "1",
"metadataVersion": "1",
"eventTime": "2021-09-08T01:14:04.5613214Z"
}
Three things follow from its shape. There is no clinical content in it, which keeps protected data out of Event Grid, its logs and its dead-letter store, provided resource ids are not themselves meaningful. The consumer therefore needs read access to the FHIR service, and that is an identity and a role assignment to plan. And the version number is there: resourceVersionId is what makes it possible to tell a new change from one already seen.
Events are published only while the feature is on. Changes made before it was enabled, or while it was off, produce nothing. Microsoft’s figures are delivery within a second on average and a ceiling of 5,000 events per second per workspace, with a warning that a bulk import can outrun event processing and leave a backlog measured in days.
How delivery goes wrong
Event Grid is explicit about what it promises. Each event is delivered at least once. Order is not guaranteed. An endpoint has 30 seconds to answer, and anything other than a 200 to 204 is a failed delivery.
| What happens | Why | What Event Grid does |
|---|---|---|
| The same event arrives twice | The endpoint answered too slowly or with an error after doing the work | Retries on a schedule: 10 seconds, 30 seconds, 1 minute, 5 minutes, 10 minutes, 30 minutes, 1 hour, then hours apart |
| Version 3 arrives before version 2 | No ordering guarantee, and a retry of an older event can land after a newer one | Nothing. Ordering is the consumer’s problem |
| An update arrives after the delete | The same | Nothing |
| An event never arrives | 30 attempts or 24 hours passed, whichever came first, or the endpoint answered 400 or 413, which are not retried | Drops the event, unless a dead-letter container is configured. Dead-lettering is off by default |
| Delivery slows for everything | Repeated failures mark the endpoint unhealthy | Delays new deliveries as well as retries, in some cases for hours |
The last two rows are the ones that surprise people. A consumer that fails loudly on a bad event can stall every good event behind it, and a consumer that is down for a day loses events for good unless dead-lettering was switched on in advance.
One rule: the stored version only moves forward
A consumer can try to defend against each of those cases separately, with a table of processed event ids for duplicates, sequence buffers for ordering and special handling for deletes. It can also make one decision that covers all of them. Every row in the projection records the version of the resource it was built from, and a write is accepted only if its version is higher than the one stored.
%%{init: {'theme':'base','themeVariables':{'primaryColor':'#FFF3E0','primaryTextColor':'#232F3E','primaryBorderColor':'#FF9900','lineColor':'#232F3E','clusterBkg':'#F8FAFC','clusterBorder':'#CBD5E1','edgeLabelBackground':'#FFFFFF','nodeTextColor':'#232F3E'}}}%%
flowchart TD
A[Event: kind, resource id, version N] --> B{Is the stored version
N or later?}
B -- "Yes" --> S1[Skip: duplicate or late arrival]
B -- "No" --> C{Is it a delete?}
C -- "Yes" --> D[Write tombstone at version N
only if still newer]
C -- "No" --> E[Read the current resource
from the FHIR server]
E --> F{Does it still exist?}
F -- "No" --> S2[Skip: its delete event will follow]
F -- "Yes" --> G[Write the row at the server's version
only if still newer]
G --> H{entered-in-error?}
H -- "Yes" --> R[Row kept as retracted, values removed]
H -- "No" --> OK[Row holds the current values]
classDef ok fill:#F0FFF4,stroke:#1D8102,stroke-width:2px,color:#145202
classDef skip fill:#EBF8FF,stroke:#0073BB,stroke-width:2px,color:#004C74
classDef warn fill:#FEF2F2,stroke:#D13212,stroke-width:2px,color:#991B1B
class OK,D ok
class S1,S2 skip
class R warnTwo details carry the weight. First, the handler does not project the version the event names. It reads the resource as it is now and stores that, with the version the server reports. An event is treated as a signal that something changed, so a late event for version 1 of a resource that is already at version 2 produces a correct row at version 2, and the event for version 2 then has nothing left to do. Second, the comparison is made by the database, in the same statement as the write.
namespace FhirEvents.Core;
public enum Outcome
{
/// <summary>The projection now holds a newer version of the resource.</summary>
Applied,
/// <summary>The projection now records the resource as deleted.</summary>
Tombstoned,
/// <summary>The projection already held this version or a later one. Duplicates and late arrivals end here.</summary>
SkippedNotNewer,
/// <summary>The resource no longer exists on the server. Its delete event will record that.</summary>
SkippedGone,
/// <summary>A resource type this projection does not hold.</summary>
Ignored,
}
/// <summary>
/// Applies one change event to the projection. Safe to call any number of times, in any order, for the same
/// resource: the stored version only ever moves forward.
/// </summary>
public sealed class FhirChangeHandler(IFhirReader fhir, IProjectionStore store)
{
public const string ProjectedType = "Observation";
public async Task<Outcome> HandleAsync(FhirChangeEvent change, CancellationToken cancellationToken)
{
if (change.ResourceType != ProjectedType)
{
return Outcome.Ignored;
}
// A cheap read that saves a call to the FHIR server for duplicates and late arrivals.
// Correctness does not rest on it: the conditional write below decides.
ProjectionRow? stored = await store.GetAsync(change.ResourceType, change.ResourceId, cancellationToken);
if (stored?.Version >= change.Version)
{
return Outcome.SkippedNotNewer;
}
if (change.Kind == ChangeKind.Deleted)
{
ProjectionRow tombstone = ProjectionRow.Tombstone(change.ResourceType, change.ResourceId, change.Version);
return await store.UpsertIfNewerAsync(tombstone, cancellationToken) ? Outcome.Tombstoned : Outcome.SkippedNotNewer;
}
// The event says something changed. What is projected is the server's current state, which may
// already be later than the version the event names.
FhirReadResult current = await fhir.ReadCurrentAsync(change.ResourceType, change.ResourceId, cancellationToken);
if (current.Resource is null)
{
return Outcome.SkippedGone;
}
ProjectionRow row = ProjectionRow.FromObservation(current.Resource, current.Version);
return await store.UpsertIfNewerAsync(row, cancellationToken) ? Outcome.Applied : Outcome.SkippedNotNewer;
}
}
The early read of the stored version is an optimisation: it saves a call to the FHIR server for the common cases. It is not what makes the handler correct, because two instances can both pass that check for the same resource at the same moment. The store settles it.
using FhirEvents.Core;
using Npgsql;
namespace FhirEvents.Functions;
public sealed class PostgresProjectionStore(NpgsqlDataSource database) : IProjectionStore
{
public async Task EnsureSchemaAsync(CancellationToken cancellationToken)
{
await using NpgsqlCommand command = database.CreateCommand("""
CREATE TABLE IF NOT EXISTS bp_projection (
resource_type text NOT NULL,
resource_id text NOT NULL,
version bigint NOT NULL,
deleted boolean NOT NULL,
retracted boolean NOT NULL,
patient_id text,
effective text,
systolic numeric,
diastolic numeric,
processed_at timestamptz NOT NULL DEFAULT now(),
PRIMARY KEY (resource_type, resource_id)
)
""");
await command.ExecuteNonQueryAsync(cancellationToken);
}
public async Task<ProjectionRow?> GetAsync(string resourceType, string resourceId, CancellationToken cancellationToken)
{
await using NpgsqlCommand command = database.CreateCommand("""
SELECT version, deleted, retracted, patient_id, effective, systolic, diastolic
FROM bp_projection WHERE resource_type = $1 AND resource_id = $2
""");
command.Parameters.AddWithValue(resourceType);
command.Parameters.AddWithValue(resourceId);
await using NpgsqlDataReader reader = await command.ExecuteReaderAsync(cancellationToken);
if (!await reader.ReadAsync(cancellationToken))
{
return null;
}
return new ProjectionRow(resourceType, resourceId, reader.GetInt64(0), reader.GetBoolean(1), reader.GetBoolean(2),
reader.IsDBNull(3) ? null : reader.GetString(3),
reader.IsDBNull(4) ? null : reader.GetString(4),
reader.IsDBNull(5) ? null : reader.GetDecimal(5),
reader.IsDBNull(6) ? null : reader.GetDecimal(6));
}
/// <summary>
/// One statement decides and writes. The WHERE clause on the conflict branch is what makes a duplicate
/// or a late delivery a no-op, even when two instances process the same resource at the same moment.
/// </summary>
public async Task<bool> UpsertIfNewerAsync(ProjectionRow row, CancellationToken cancellationToken)
{
await using NpgsqlCommand command = database.CreateCommand("""
INSERT INTO bp_projection
(resource_type, resource_id, version, deleted, retracted, patient_id, effective, systolic, diastolic)
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)
ON CONFLICT (resource_type, resource_id) DO UPDATE SET
version = EXCLUDED.version, deleted = EXCLUDED.deleted, retracted = EXCLUDED.retracted,
patient_id = EXCLUDED.patient_id, effective = EXCLUDED.effective,
systolic = EXCLUDED.systolic, diastolic = EXCLUDED.diastolic, processed_at = now()
WHERE bp_projection.version < EXCLUDED.version
""");
command.Parameters.AddWithValue(row.ResourceType);
command.Parameters.AddWithValue(row.ResourceId);
command.Parameters.AddWithValue(row.Version);
command.Parameters.AddWithValue(row.Deleted);
command.Parameters.AddWithValue(row.Retracted);
command.Parameters.AddWithValue((object?)row.PatientId ?? DBNull.Value);
command.Parameters.AddWithValue((object?)row.Effective ?? DBNull.Value);
command.Parameters.AddWithValue((object?)row.Systolic ?? DBNull.Value);
command.Parameters.AddWithValue((object?)row.Diastolic ?? DBNull.Value);
return await command.ExecuteNonQueryAsync(cancellationToken) == 1;
}
}
The WHERE clause on the conflict branch is the whole concurrency design. PostgreSQL takes a row lock for the update, evaluates the condition against the committed row and reports zero rows affected when the stored version is already as high. No distributed lock, no table of processed event ids, and no assumption about how many Function instances are running.
A table of processed event ids handles duplicates and nothing else: it has to be kept and pruned, and it says nothing about order. The resource version handles duplicates, reordering and late arrivals with one integer per row that the projection needs anyway.
Corrections and deletions
A blood pressure recorded against the wrong patient is not deleted in a well-run record. Its status is set to entered-in-error, and that arrives as an ordinary update event. A projection that only copies values would keep serving the wrong reading with a clean conscience. The handler keeps the row, marks it retracted and removes the values, so that whatever was derived downstream, an alert, a report line, a task, can be found and undone.
A soft delete produces a FhirResourceDeleted event, and the handler writes a tombstone at that version. The tombstone matters as much as the delete: without it, a late update for an earlier version would bring the reading back. With it, the late update fails the version check like any other stale event.
The documented delete event is raised for a soft delete. I found no event listed for a hard delete or a purge of history, which are the operations used to honour an erasure request. A projection fed only by events will keep that patient’s data indefinitely. Erasure therefore needs its own path to every downstream copy: an explicit instruction to each consumer, recorded and confirmed, as part of the erasure procedure itself. The projection is personal data in its own right and belongs in the record of processing with a retention period.
The Function
%%{init: {'theme':'base','themeVariables':{'primaryColor':'#FFF3E0','primaryTextColor':'#232F3E','primaryBorderColor':'#FF9900','lineColor':'#232F3E','signalColor':'#0073BB','signalTextColor':'#0F172A','actorBkg':'#FFF3E0','actorBorder':'#FF9900','actorTextColor':'#232F3E','noteBkgColor':'#FFF8E7','noteBorderColor':'#FF9900','noteTextColor':'#232F3E'}}}%%
sequenceDiagram
participant EG as Event Grid
participant FN as Function
participant FS as FHIR service
participant DB as Projection store
EG->>FN: Event: Observation bp-1, version 3
FN->>DB: Stored version for bp-1?
DB-->>FN: 1
FN->>FS: GET Observation/bp-1
FS-->>FN: Resource at version 3
FN->>DB: Upsert where stored version is lower
DB-->>FN: 1 row written
FN-->>EG: Success
EG->>FN: Event: Observation bp-1, version 2 (late)
FN->>DB: Stored version for bp-1?
DB-->>FN: 3
FN-->>EG: Success, nothing to doThe Function App has two ways in. The Event Grid trigger is the one an Event Grid subscription points at. The HTTP endpoint does the same work for deliveries that come through a gateway, and answers both subscription validation handshakes, the Event Grid one that echoes a validation code and the CloudEvents one that answers an OPTIONS request. A third function reads a row back, which is how the test client sees what the events produced.
using System.Net;
using System.Text.Json.Nodes;
using FhirEvents.Core;
using Microsoft.Azure.Functions.Worker;
using Microsoft.Azure.Functions.Worker.Http;
using Microsoft.Extensions.Logging;
namespace FhirEvents.Functions;
public sealed class EventFunctions(FhirChangeHandler handler, IProjectionStore store, ILogger<EventFunctions> logger)
{
/// <summary>
/// The Event Grid subscription's handler. An exception fails the invocation, and Event Grid then delivers
/// the event again under its retry policy. Malformed events are logged and dropped: retrying cannot fix them.
/// </summary>
[Function("ProjectFromEventGrid")]
public async Task FromEventGrid([EventGridTrigger] string eventJson, CancellationToken cancellationToken) =>
await ApplyAsync(eventJson, cancellationToken);
/// <summary>
/// The same processing behind an HTTP endpoint, for deliveries that arrive through a gateway instead of
/// directly from Event Grid. It answers both subscription validation handshakes.
/// </summary>
[Function("ProjectFromWebhook")]
public async Task<HttpResponseData> FromWebhook(
[HttpTrigger(AuthorizationLevel.Function, "post", "options", Route = "events")] HttpRequestData request,
CancellationToken cancellationToken)
{
// CloudEvents handshake: an OPTIONS request naming the origin that wants to deliver.
if (request.Method.Equals("OPTIONS", StringComparison.OrdinalIgnoreCase))
{
HttpResponseData allowed = request.CreateResponse(HttpStatusCode.OK);
if (request.Headers.TryGetValues("WebHook-Request-Origin", out IEnumerable<string>? origins))
{
allowed.Headers.Add("WebHook-Allowed-Origin", origins.First());
}
return allowed;
}
string body = await new StreamReader(request.Body).ReadToEndAsync(cancellationToken);
// Event Grid schema handshake: echo the validation code.
if (TryGetValidationCode(body) is { } code)
{
return await JsonAsync(request, new JsonObject { ["validationResponse"] = code });
}
return await JsonAsync(request, await ApplyAsync(body, cancellationToken));
}
/// <summary>Reads one row of the projection. Used by the test client to see what the events produced.</summary>
[Function("GetProjection")]
public async Task<HttpResponseData> GetProjection(
[HttpTrigger(AuthorizationLevel.Function, "get", Route = "projection/{id}")] HttpRequestData request,
string id, CancellationToken cancellationToken)
{
ProjectionRow? row = await store.GetAsync(FhirChangeHandler.ProjectedType, id, cancellationToken);
if (row is null)
{
return request.CreateResponse(HttpStatusCode.NotFound);
}
return await JsonAsync(request, new JsonObject
{
["resourceId"] = row.ResourceId,
["version"] = row.Version,
["deleted"] = row.Deleted,
["retracted"] = row.Retracted,
["patientId"] = row.PatientId,
["effective"] = row.Effective,
["systolic"] = row.Systolic,
["diastolic"] = row.Diastolic,
});
}
private async Task<JsonObject> ApplyAsync(string delivery, CancellationToken cancellationToken)
{
(IReadOnlyList<FhirChangeEvent> events, IReadOnlyList<string> rejected) = FhirChangeEvent.ParseDelivery(delivery);
foreach (string reason in rejected)
{
logger.LogWarning("Dropped an event that cannot be processed: {Reason}", reason);
}
JsonArray results = [];
foreach (FhirChangeEvent change in events)
{
Outcome outcome = await handler.HandleAsync(change, cancellationToken);
// Resource type, id and version are identifiers, not clinical content. Nothing else is logged.
logger.LogInformation("Event {EventId} {Kind} {ResourceType}/{ResourceId} v{Version}: {Outcome}",
change.EventId, change.Kind, change.ResourceType, change.ResourceId, change.Version, outcome);
results.Add(new JsonObject { ["eventId"] = change.EventId, ["outcome"] = outcome.ToString() });
}
return new JsonObject { ["results"] = results, ["rejected"] = new JsonArray([.. rejected.Select(r => (JsonNode)r)]) };
}
private static string? TryGetValidationCode(string body)
{
try
{
return JsonNode.Parse(body) is JsonArray { Count: 1 } events
&& (string?)events[0]?["eventType"] == "Microsoft.EventGrid.SubscriptionValidationEvent"
? (string?)events[0]?["data"]?["validationCode"]
: null;
}
catch (System.Text.Json.JsonException)
{
return null;
}
}
private static async Task<HttpResponseData> JsonAsync(HttpRequestData request, JsonObject body)
{
HttpResponseData response = request.CreateResponse(HttpStatusCode.OK);
response.Headers.Add("Content-Type", "application/json");
await response.WriteStringAsync(body.ToJsonString());
return response;
}
}
Error handling is deliberately split in two. A malformed event is logged and dropped, because delivering it again cannot fix it and failing on it would mark the endpoint unhealthy for every other event. A failure to reach the FHIR service or the database is allowed to throw: the invocation fails, Event Grid delivers again later, and because the handler is idempotent the retry is safe however much of the first attempt completed. The log lines carry the event id, resource type, resource id and version. They carry no clinical content.
The same rule in Python
Not every consumer is a Function. Where Event Grid delivers to a queue and a worker pulls from it, the rule is the same and so is its shape. This version was run against the same scripted scenario as the C# handler and produced identical outcomes for all thirteen deliveries.
"""The same projection rule as the C# handler, for a consumer that pulls events from a queue.
The stored version only moves forward. A delivery whose version is not newer than what is stored is a
duplicate or a late arrival and changes nothing. A create or update projects the server's current state,
not the version the event names. A delete writes a tombstone.
"""
from __future__ import annotations
from dataclasses import dataclass
from typing import Callable, Optional
PROJECTED_TYPE = "Observation"
@dataclass(frozen=True)
class Row:
resource_id: str
version: int
deleted: bool = False
retracted: bool = False
systolic: Optional[float] = None
diastolic: Optional[float] = None
class InMemoryStore:
"""Stands in for a table with a conditional upsert. Not safe across processes; a real consumer uses the database."""
def __init__(self) -> None:
self.rows: dict[str, Row] = {}
def get(self, resource_id: str) -> Optional[Row]:
return self.rows.get(resource_id)
def upsert_if_newer(self, row: Row) -> bool:
current = self.rows.get(row.resource_id)
if current is not None and current.version >= row.version:
return False
self.rows[row.resource_id] = row
return True
def row_from_observation(observation: dict, version: int) -> Row:
retracted = observation.get("status") == "entered-in-error"
def component(code: str) -> Optional[float]:
for part in observation.get("component", []):
if any(c.get("code") == code for c in part.get("code", {}).get("coding", [])):
return part.get("valueQuantity", {}).get("value")
return None
return Row(observation["id"], version, retracted=retracted,
systolic=None if retracted else component("8480-6"),
diastolic=None if retracted else component("8462-4"))
def handle(event: dict, store: InMemoryStore, read_current: Callable[[str], Optional[tuple[dict, int]]]) -> str:
"""event: {"kind": "Created" | "Updated" | "Deleted", "resourceType", "resourceId", "version"}.
read_current returns (resource, version) or None when the resource is gone. It may raise: let it, so the
message goes back to the queue and is retried.
"""
if event.get("resourceType", PROJECTED_TYPE) != PROJECTED_TYPE:
return "Ignored"
stored = store.get(event["resourceId"])
if stored is not None and stored.version >= event["version"]:
return "SkippedNotNewer"
if event["kind"] == "Deleted":
tombstone = Row(event["resourceId"], event["version"], deleted=True)
return "Tombstoned" if store.upsert_if_newer(tombstone) else "SkippedNotNewer"
current = read_current(event["resourceId"])
if current is None:
return "SkippedGone"
resource, version = current
return "Applied" if store.upsert_if_newer(row_from_observation(resource, version)) else "SkippedNotNewer"
What the replay showed
The replay script changes a real resource on the FHIR server at each step and then delivers events about it to the running Function App: duplicated, out of order, late, and for resources that are already gone.
| Delivered | Outcome | Projection afterwards |
|---|---|---|
| Created, version 1 | Applied | Version 1, 142/88 |
| The same event again | Skipped: not newer | Unchanged |
| Updated version 3, before version 2 | Applied | Version 3, 138/86 |
| Updated version 2, late | Skipped: not newer | Still version 3 |
| Updated version 4, status entered-in-error | Applied | Retracted, values removed |
| Deleted, version 5 | Tombstoned | Deleted at version 5 |
| The version 4 correction again, after the delete | Skipped: not newer | Still deleted |
| Created version 1 of a resource already at version 2 | Applied | Version 2, the current state |
| Created version 1 of a resource already deleted | Skipped: gone | No row, until its delete event wrote the tombstone |
| One delivery holding a Patient event, a malformed event and a duplicate | Ignored, dropped, skipped | Unchanged |
| A delivery through the Event Grid trigger’s own endpoint | Accepted | Row written |
| A delivery with no key | HTTP 401 | Unchanged |
All 22 checks in the script passed. A further run from a browser sent 120 deliveries for one resource at once, twenty copies of each of six versions in random order. One was applied, 119 were skipped, and the row finished at version 6 with the values of version 6. That is the conditional write doing its job under concurrency, on one Function host; it says nothing yet about several hosts, though the statement does not depend on how many there are.

The replay described here was repeated on 12 November 2026 with HAPI FHIR 8.6.0 as the FHIR server and PostgreSQL 16 as the store. The table and the concurrency figures are from that run.
Rule: the C# core passes 10 self-test checks on the .NET 10 SDK, including 120 concurrent deliveries against an in-memory store with the same conditional contract, and a FHIR failure that must propagate. The Python rule ran the same scenario with identical outcomes. Live: the Function App was built on the .NET 10 SDK image and run on the Azure Functions .NET 10 isolated image in Docker, with PostgreSQL and a real FHIR server. The replay script passed 22 of 22 checks, one of them through the Event Grid trigger’s endpoint, and the concurrent run from a browser gave the figures above. Not checked: Event Grid itself, since the events were built by the script; Azure Health Data Services; managed identity; dead-lettering and retries driven by Event Grid; and several Function instances at once. The listings are the files that ran, fetched from the repository commit.
When events are not enough: reconciliation
The version rule makes the projection correct for every event that arrives. It can do nothing about one that does not. Events are lost when the endpoint is unreachable for longer than the retry window and no dead-letter container was configured, when the feature was switched off for a while, and when data was loaded in a way that outran or bypassed event delivery. A hard delete, as noted above, may never have had an event at all.
So a projection that has to be right needs a second, slower path that does not depend on events: a scheduled sweep that asks the FHIR service what changed since the last run, using _lastUpdated and history, and feeds the answers through the same handler. Because the handler is idempotent, the sweep and the event stream can overlap freely. Run it daily, and after any outage. The sample does not include the sweep.
Running it in production
- Identity. The Function reads the FHIR service with its managed identity and the
FHIR Data Readerrole. Event Grid should deliver with a managed identity too, so that the Function accepts events only from the system topic. The sample uses function keys locally, which is the weakest acceptable option. - Dead-lettering. Configure a dead-letter container on the subscription before the first event flows, and alert on anything landing in it. Without it, an event that exhausts its retries is gone.
- Lag. The event carries the time the change committed and the row records when it was processed. The difference is the number to watch, and after a bulk load it can be large for a long time.
- Scale-out. Functions add instances under load, and two instances will process events for the same resource at once. The conditional write is what makes that safe. Size the database connection pool for the instance count.
- Throttling. Every effective change costs one read of the FHIR service. A burst of events is a burst of reads; a failed read throws, Event Grid backs off, and the handler picks up where it left off.
- Batching. Event Grid batching is all-or-nothing per delivery. With an idempotent handler a failed batch can simply be delivered again.
- Cost. Three meters run: Event Grid operations, Function executions, and FHIR reads. Skipped deliveries cost an execution and a database lookup, and no FHIR read.
Choosing the mechanism
- Use events with a version-guarded consumer when a downstream copy should follow the record within seconds and can tolerate being briefly behind.
- Add reconciliation whenever the copy is used for anything that must be complete, such as reporting or population lists.
- Use a scheduled export, not events, when the consumer wants a daily snapshot and has no use for near real time.
- Do not use events as an audit trail. They announce that a version exists, not who changed what, and they can be missed.
Key takeaways
- An Azure FHIR event names a resource and a version and carries no content. The consumer reads the resource, so it needs an identity with read access.
- Event Grid delivers at least once, in no guaranteed order, for at most 24 hours, and drops what it cannot deliver unless dead-lettering is configured.
- Store the source version with every projected row and accept a write only when the version is higher. One conditional statement covers duplicates, reordering, late arrivals and concurrent instances.
- Keep tombstones for deletes and retraction markers for corrections. Removing the row loses the protection and the audit trail together.
- Events are a hint that something changed. Pair them with a periodic reconciliation, and give erasure requests a path of their own.
If you run a consumer like this against Azure Health Data Services, I would like to hear in the comments how event lag behaved during your largest load.
Glossary
- Event Gridβ Azure’s publish and subscribe service. The FHIR service publishes change events to a system topic in it.
- System topicβ The Event Grid topic that represents an Azure resource, here the Health Data Services workspace.
- At-least-once deliveryβ Every event is delivered one or more times. Duplicates are expected.
- Idempotentβ Safe to apply more than once: the result after the second time is the same as after the first.
- Projectionβ A downstream copy of part of the record, shaped for one consumer.
- Tombstoneβ A row that records a resource was deleted, and at which version.
- entered-in-errorβ The FHIR status that withdraws a resource recorded by mistake without deleting it.
- Soft deleteβ A delete that keeps earlier versions in history. A hard delete removes them.
- Dead-letter containerβ Storage where Event Grid puts events it could not deliver, when configured.
- FHIR Subscriptionβ The FHIR resource through which a client asks a server to notify it of changes.
- Reconciliationβ A periodic comparison of the projection with the source that does not depend on events.
References & Further Reading
- → Sample code for this articleβ The Function App, the core handler, the Python rule and the replay script
- → What are events? Azure Health Data Servicesβ Which operations raise events and what handlers are supported
- → Events message structuresβ The event fields and the sample used above
- → Events FAQβ Bundles, duplicates, throughput and lag during bulk loads
- → Event Grid message delivery and retryβ The retry schedule, limits, status codes and dead-lettering
- → Azure Event Grid trigger for Azure Functionsβ The trigger used by the Function
- → FHIR R4 Subscriptionβ The standard notification resource
- → Subscriptions R5 Backportβ Topic-based subscriptions for R4 servers
- → PostgreSQL INSERTβ ON CONFLICT DO UPDATE and its WHERE clause
- → Migrating from Azure API for FHIR to Azure Health Data Servicesβ Events are one of the capabilities the newer service adds
Discover more from C4: Container, Code, Cloud & Context
Subscribe to get the latest posts sent to your email.