On this page
Industrial and commercial IoT projects rarely fail on connectivity. They fail on the data: readings that arrive twice, out of order or hours late after a gateway lost its uplink; alarms that flap because a value hovers at the limit; dashboards that disagree with the historian because one counted duplicates and the other did not. Real-time decisions are only as good as the handling of those cases.
This article sets out which processing belongs at the edge, which in a cloud stream and which in a data lake, and the four techniques that make telemetry trustworthy: report-by-exception, deduplication, event-time windows with watermarks, and alarms with hysteresis. The sample code implements each in plain Python so the behaviour, and the settings that control it, can be read and tested.
Three tiers, three jobs

The edge gateway sits next to the equipment. It translates industrial protocols (Modbus, OPC UA) into MQTT, applies safety interlocks and local rules that cannot wait for a round trip, filters what is worth sending, and buffers readings when the uplink drops. AWS IoT Greengrass and Azure IoT Edge are the managed runtimes for this tier.
The stream tier receives telemetry from every site through an MQTT broker such as AWS IoT Core or Azure IoT Hub and processes it within seconds: deduplicating, aggregating into windows, raising alarms and updating operational dashboards. Apache Flink (offered on AWS as Amazon Managed Service for Apache Flink) and Azure Stream Analytics are typical engines.
The lake keeps every raw reading, cheaply, for audits, reprocessing and model training. It is where trends over months are found and where anomaly models are trained before being pushed back to the stream or the edge.

Report by exception at the edge
A sensor sampled every second produces 86,400 readings a day, most of them identical to the last. A deadband filter forwards a reading only when it changes by more than a threshold, plus a heartbeat at a fixed interval so the cloud can tell a steady value from a dead device. The threshold should come from the sensor’s accuracy and the process: forwarding changes smaller than the sensor’s noise sends noise.
Duplicates, late data and windows
MQTT at QoS 1, which AWS IoT Core and most brokers use for important telemetry, delivers at least once: a message whose acknowledgement is lost is sent again. Gateways that buffer while offline replay their backlog when the link returns, so readings arrive late and out of order. Two rules follow. Give each message an ID at the source and deduplicate on it. Timestamp readings at the device, and aggregate on that event time rather than on arrival time.
%%{init: {'theme': 'base', 'themeVariables': {'signalColor': '#0073BB', 'signalTextColor': '#0F172A', 'actorBkg': '#FFF3E0', 'actorBorder': '#FF9900', 'actorTextColor': '#232F3E', 'noteBkgColor': '#FFF8E7', 'noteBorderColor': '#FF9900', 'noteTextColor': '#232F3E', 'lineColor': '#232F3E'}}}%%
sequenceDiagram
participant D as Sensor
participant G as Edge gateway
participant B as MQTT broker
participant S as Stream processor
D->>G: Reading at 10:00:05
Note over G,B: Uplink down for 4 minutes
G->>G: Buffer readings to disk
G->>B: Replay backlog, QoS 1
B->>S: Reading at 10:00:05, arrives 10:04:10
B->>S: Same reading again (lost PUBACK)
S->>S: Drop duplicate by message ID
Note over S: Window 10:00-10:01 still open<br/>if within allowed latenessStream engines handle lateness with a watermark: the engine’s estimate of how far event time has progressed, usually the latest event time seen minus an allowed lateness. A window is finalised when the watermark passes its end. Readings that arrive later than that are either dropped, routed to a side output, or used to correct the result, but they must not silently change a number someone already acted on. The allowed lateness is a business decision: how long results may wait for stragglers versus how often a site goes offline.
The sample implements deduplication, event-time tumbling windows with a watermark (counting readings that are too late instead of merging them), the deadband filter and a hysteresis alarm:
"""Small, dependency-free building blocks for IoT telemetry processing.
- Deadband: report-by-exception filter for the edge; only forwards a reading
when it moves more than a threshold or a heartbeat interval has passed.
- Deduplicator: drops redeliveries (MQTT QoS 1 and most queues are
at-least-once) by message ID within a bounded memory.
- TumblingWindows: event-time windows with a watermark, so readings that
arrive late (devices buffer while offline) land in the right window, and
readings later than the allowed lateness are counted instead of silently
corrupting a closed window.
- HysteresisAlarm: raises after N consecutive breaches and clears only below
a lower threshold, so a value hovering at the limit does not flap.
These are the patterns stream engines (Apache Flink, Kinesis, Azure Stream
Analytics) provide; seeing them in plain Python makes their settings easier
to reason about.
"""
from __future__ import annotations
from collections import OrderedDict, defaultdict
from dataclasses import dataclass
@dataclass(frozen=True)
class Reading:
device: str
ts: float # event time, seconds since epoch, set by the device
value: float
msg_id: str = ""
class Deadband:
def __init__(self, threshold: float, heartbeat_s: float):
if threshold < 0 or heartbeat_s <= 0:
raise ValueError("threshold must be >= 0 and heartbeat_s > 0")
self.threshold, self.heartbeat = threshold, heartbeat_s
self._last: dict[str, Reading] = {}
def accept(self, r: Reading) -> bool:
last = self._last.get(r.device)
if last is None or abs(r.value - last.value) > self.threshold or r.ts - last.ts >= self.heartbeat:
self._last[r.device] = r
return True
return False
class Deduplicator:
def __init__(self, capacity: int = 100_000):
self.capacity = capacity
self._seen: OrderedDict[str, None] = OrderedDict()
def first_time(self, msg_id: str) -> bool:
if not msg_id:
return True # cannot dedupe without an ID
if msg_id in self._seen:
self._seen.move_to_end(msg_id)
return False
self._seen[msg_id] = None
if len(self._seen) > self.capacity:
self._seen.popitem(last=False)
return True
@dataclass
class WindowResult:
device: str
start: float
end: float
count: int
mean: float
maximum: float
class TumblingWindows:
"""Event-time tumbling windows per device with a watermark.
The watermark is the highest event time seen minus `allowed_lateness_s`.
A window is emitted once the watermark passes its end; readings for an
already-emitted window are counted in `too_late`.
"""
def __init__(self, size_s: float, allowed_lateness_s: float):
if size_s <= 0 or allowed_lateness_s < 0:
raise ValueError("size_s must be > 0 and allowed_lateness_s >= 0")
self.size, self.lateness = size_s, allowed_lateness_s
self._open: dict[tuple[str, float], list[float]] = defaultdict(list)
self._max_ts = float("-inf")
self._emitted_until: dict[str, float] = defaultdict(lambda: float("-inf"))
self.too_late = 0
def _start(self, ts: float) -> float:
return ts - (ts % self.size)
@property
def watermark(self) -> float:
return self._max_ts - self.lateness
def add(self, r: Reading) -> list[WindowResult]:
start = self._start(r.ts)
if start + self.size <= self._emitted_until[r.device]:
self.too_late += 1
return []
self._open[(r.device, start)].append(r.value)
self._max_ts = max(self._max_ts, r.ts)
return self._close_ready()
def _close_ready(self, force: bool = False) -> list[WindowResult]:
out = []
for (device, start) in sorted(self._open, key=lambda k: k[1]):
end = start + self.size
if force or end <= self.watermark:
vals = self._open.pop((device, start))
out.append(WindowResult(device, start, end, len(vals), sum(vals) / len(vals), max(vals)))
self._emitted_until[device] = max(self._emitted_until[device], end)
return out
def flush(self) -> list[WindowResult]:
return self._close_ready(force=True)
class HysteresisAlarm:
def __init__(self, raise_above: float, clear_below: float, consecutive: int = 3):
if clear_below >= raise_above:
raise ValueError("clear_below must be lower than raise_above")
self.hi, self.lo, self.n = raise_above, clear_below, consecutive
self._breaches: dict[str, int] = defaultdict(int)
self.active: set[str] = set()
def update(self, device: str, value: float) -> str | None:
"""Return 'RAISED', 'CLEARED' or None."""
if device in self.active:
if value < self.lo:
self.active.discard(device)
self._breaches[device] = 0
return "CLEARED"
return None
self._breaches[device] = self._breaches[device] + 1 if value > self.hi else 0
if self._breaches[device] >= self.n:
self.active.add(device)
return "RAISED"
return NoneAlarms that do not flap
An alarm that fires whenever a value crosses 80 and clears whenever it drops below 80 will fire dozens of times while a temperature hovers at 80. Two settings fix that: require several consecutive breaches before raising, and clear only below a lower threshold (here 70). The gap between the two thresholds should exceed the signal’s normal noise. Alarm floods are a safety problem in their own right, because operators stop reading them.
Interlocks and emergency stops belong in the equipment’s own control system or the edge, where they work without a network. Cloud analytics can warn, schedule maintenance and optimise; it must not be the only thing that stops a machine.
Store raw readings in the lake before any filtering beyond the edge deadband. Window sizes, lateness and alarm thresholds will change, and reprocessing history with new settings is only possible if the original readings were kept.
Give every device its own credential (an X.509 certificate per device, not a shared key), scope its MQTT permissions to its own topics, and have a way to revoke it. Telemetry can be personal data when it is tied to a person, such as a vehicle, a wearable or a smart meter, so classify it and apply retention rules in the lake as well as the stream.
streamproc.py was tested with Python 3.12 and pytest==8.2.0: eight tests cover deadband forwarding and heartbeats, deduplication with bounded memory, late readings within allowed lateness, too-late readings counted rather than merged, per-device windows, hysteresis raise and clear, and input validation. It has no dependencies and was not run against a broker or a stream engine.
Key takeaways
- Edge for millisecond and offline decisions, stream for fleet-wide decisions in seconds, lake for history and models.
- Filter by exception at the edge with a threshold above sensor noise and a heartbeat.
- QoS 1 means duplicates: give messages IDs at the source and deduplicate.
- Aggregate on device event time with a watermark, and decide explicitly what happens to data that is too late.
- Use consecutive breaches and hysteresis for alarms, keep safety functions out of the cloud, and keep the raw data.
How do you handle late and duplicate telemetry in your pipelines? Share your watermark and alarm settings in the comments.
The complete sample for this article is on GitHub: nithinmohantk/blog-samples-working-repo/iot-stream-windows. Clone it with git clone https://github.com/nithinmohantk/blog-samples-working-repo.git and open the iot-stream-windows folder. Its README says how to run it and what has been verified.
Glossary
- Edge gateway
- A device near the equipment that collects, filters and forwards data and can act locally.
- Deadband
- A filter that forwards a value only when it changes by more than a set amount.
- QoS 1
- MQTT at-least-once delivery: messages are resent until acknowledged, so duplicates are possible.
- Event time
- The time a reading was taken at the device, as opposed to when it arrived.
- Watermark
- A stream engine’s estimate of event-time progress, used to decide when a window is complete.
- Tumbling window
- Fixed, non-overlapping time intervals used to aggregate a stream.
- Hysteresis
- Using separate raise and clear thresholds so a signal near the limit does not toggle an alarm.
References & Further Reading
- → Does AWS IoT Core support QoS 2?— AWS answer: QoS 0 and 1 supported; idempotent handling with QoS 1
- → Sample code for this article— Deadband, deduplication, event-time windows and hysteresis alarm
Discover more from C4: Container, Code, Cloud & Context
Subscribe to get the latest posts sent to your email.