IoT Analytics in Real Time: What to Do at the Edge, in the Stream and in the Lake

On this page

Industrial and commercial IoT projects rarely fail on connectivity. They fail on the data: readings that arrive twice, out of order or hours late after a gateway lost its uplink; alarms that flap because a value hovers at the limit; dashboards that disagree with the historian because one counted duplicates and the other did not. Real-time decisions are only as good as the handling of those cases.

This article sets out which processing belongs at the edge, which in a cloud stream and which in a data lake, and the four techniques that make telemetry trustworthy: report-by-exception, deduplication, event-time windows with watermarks, and alarms with hysteresis. The sample code implements each in plain Python so the behaviour, and the settings that control it, can be read and tested.

Three tiers, three jobs

IoT architecture: devices feed an edge gateway that translates protocols, filters and buffers; an MQTT uplink to cloud ingestion; stream processing feeding alerts, a time-series store and a raw data lake
Decisions that need milliseconds, or must work without the uplink, stay at the site.

The edge gateway sits next to the equipment. It translates industrial protocols (Modbus, OPC UA) into MQTT, applies safety interlocks and local rules that cannot wait for a round trip, filters what is worth sending, and buffers readings when the uplink drops. AWS IoT Greengrass and Azure IoT Edge are the managed runtimes for this tier.

The stream tier receives telemetry from every site through an MQTT broker such as AWS IoT Core or Azure IoT Hub and processes it within seconds: deduplicating, aggregating into windows, raising alarms and updating operational dashboards. Apache Flink (offered on AWS as Amazon Managed Service for Apache Flink) and Azure Stream Analytics are typical engines.

The lake keeps every raw reading, cheaply, for audits, reprocessing and model training. It is where trends over months are found and where anomaly models are trained before being pushed back to the stream or the edge.

Edge gateway, stream processing and data lake compared on latency, behaviour without the uplink, data seen, good uses and caveats
Put each decision in the cheapest tier that meets its latency and connectivity needs.

Report by exception at the edge

A sensor sampled every second produces 86,400 readings a day, most of them identical to the last. A deadband filter forwards a reading only when it changes by more than a threshold, plus a heartbeat at a fixed interval so the cloud can tell a steady value from a dead device. The threshold should come from the sensor’s accuracy and the process: forwarding changes smaller than the sensor’s noise sends noise.

Duplicates, late data and windows

MQTT at QoS 1, which AWS IoT Core and most brokers use for important telemetry, delivers at least once: a message whose acknowledgement is lost is sent again. Gateways that buffer while offline replay their backlog when the link returns, so readings arrive late and out of order. Two rules follow. Give each message an ID at the source and deduplicate on it. Timestamp readings at the device, and aggregate on that event time rather than on arrival time.

%%{init: {'theme': 'base', 'themeVariables': {'signalColor': '#0073BB', 'signalTextColor': '#0F172A', 'actorBkg': '#FFF3E0', 'actorBorder': '#FF9900', 'actorTextColor': '#232F3E', 'noteBkgColor': '#FFF8E7', 'noteBorderColor': '#FF9900', 'noteTextColor': '#232F3E', 'lineColor': '#232F3E'}}}%%
sequenceDiagram
    participant D as Sensor
    participant G as Edge gateway
    participant B as MQTT broker
    participant S as Stream processor
    D->>G: Reading at 10:00:05
    Note over G,B: Uplink down for 4 minutes
    G->>G: Buffer readings to disk
    G->>B: Replay backlog, QoS 1
    B->>S: Reading at 10:00:05, arrives 10:04:10
    B->>S: Same reading again (lost PUBACK)
    S->>S: Drop duplicate by message ID
    Note over S: Window 10:00-10:01 still open<br/>if within allowed lateness

Stream engines handle lateness with a watermark: the engine’s estimate of how far event time has progressed, usually the latest event time seen minus an allowed lateness. A window is finalised when the watermark passes its end. Readings that arrive later than that are either dropped, routed to a side output, or used to correct the result, but they must not silently change a number someone already acted on. The allowed lateness is a business decision: how long results may wait for stragglers versus how often a site goes offline.

The sample implements deduplication, event-time tumbling windows with a watermark (counting readings that are too late instead of merging them), the deadband filter and a hysteresis alarm:

Python141 lines
"""Small, dependency-free building blocks for IoT telemetry processing.

- Deadband: report-by-exception filter for the edge; only forwards a reading
  when it moves more than a threshold or a heartbeat interval has passed.
- Deduplicator: drops redeliveries (MQTT QoS 1 and most queues are
  at-least-once) by message ID within a bounded memory.
- TumblingWindows: event-time windows with a watermark, so readings that
  arrive late (devices buffer while offline) land in the right window, and
  readings later than the allowed lateness are counted instead of silently
  corrupting a closed window.
- HysteresisAlarm: raises after N consecutive breaches and clears only below
  a lower threshold, so a value hovering at the limit does not flap.

These are the patterns stream engines (Apache Flink, Kinesis, Azure Stream
Analytics) provide; seeing them in plain Python makes their settings easier
to reason about.
"""
from __future__ import annotations

from collections import OrderedDict, defaultdict
from dataclasses import dataclass


@dataclass(frozen=True)
class Reading:
    device: str
    ts: float          # event time, seconds since epoch, set by the device
    value: float
    msg_id: str = ""


class Deadband:
    def __init__(self, threshold: float, heartbeat_s: float):
        if threshold < 0 or heartbeat_s <= 0:
            raise ValueError("threshold must be >= 0 and heartbeat_s > 0")
        self.threshold, self.heartbeat = threshold, heartbeat_s
        self._last: dict[str, Reading] = {}

    def accept(self, r: Reading) -> bool:
        last = self._last.get(r.device)
        if last is None or abs(r.value - last.value) > self.threshold or r.ts - last.ts >= self.heartbeat:
            self._last[r.device] = r
            return True
        return False


class Deduplicator:
    def __init__(self, capacity: int = 100_000):
        self.capacity = capacity
        self._seen: OrderedDict[str, None] = OrderedDict()

    def first_time(self, msg_id: str) -> bool:
        if not msg_id:
            return True                     # cannot dedupe without an ID
        if msg_id in self._seen:
            self._seen.move_to_end(msg_id)
            return False
        self._seen[msg_id] = None
        if len(self._seen) > self.capacity:
            self._seen.popitem(last=False)
        return True


@dataclass
class WindowResult:
    device: str
    start: float
    end: float
    count: int
    mean: float
    maximum: float


class TumblingWindows:
    """Event-time tumbling windows per device with a watermark.

    The watermark is the highest event time seen minus `allowed_lateness_s`.
    A window is emitted once the watermark passes its end; readings for an
    already-emitted window are counted in `too_late`.
    """

    def __init__(self, size_s: float, allowed_lateness_s: float):
        if size_s <= 0 or allowed_lateness_s < 0:
            raise ValueError("size_s must be > 0 and allowed_lateness_s >= 0")
        self.size, self.lateness = size_s, allowed_lateness_s
        self._open: dict[tuple[str, float], list[float]] = defaultdict(list)
        self._max_ts = float("-inf")
        self._emitted_until: dict[str, float] = defaultdict(lambda: float("-inf"))
        self.too_late = 0

    def _start(self, ts: float) -> float:
        return ts - (ts % self.size)

    @property
    def watermark(self) -> float:
        return self._max_ts - self.lateness

    def add(self, r: Reading) -> list[WindowResult]:
        start = self._start(r.ts)
        if start + self.size <= self._emitted_until[r.device]:
            self.too_late += 1
            return []
        self._open[(r.device, start)].append(r.value)
        self._max_ts = max(self._max_ts, r.ts)
        return self._close_ready()

    def _close_ready(self, force: bool = False) -> list[WindowResult]:
        out = []
        for (device, start) in sorted(self._open, key=lambda k: k[1]):
            end = start + self.size
            if force or end <= self.watermark:
                vals = self._open.pop((device, start))
                out.append(WindowResult(device, start, end, len(vals), sum(vals) / len(vals), max(vals)))
                self._emitted_until[device] = max(self._emitted_until[device], end)
        return out

    def flush(self) -> list[WindowResult]:
        return self._close_ready(force=True)


class HysteresisAlarm:
    def __init__(self, raise_above: float, clear_below: float, consecutive: int = 3):
        if clear_below >= raise_above:
            raise ValueError("clear_below must be lower than raise_above")
        self.hi, self.lo, self.n = raise_above, clear_below, consecutive
        self._breaches: dict[str, int] = defaultdict(int)
        self.active: set[str] = set()

    def update(self, device: str, value: float) -> str | None:
        """Return 'RAISED', 'CLEARED' or None."""
        if device in self.active:
            if value < self.lo:
                self.active.discard(device)
                self._breaches[device] = 0
                return "CLEARED"
            return None
        self._breaches[device] = self._breaches[device] + 1 if value > self.hi else 0
        if self._breaches[device] >= self.n:
            self.active.add(device)
            return "RAISED"
        return None

Alarms that do not flap

An alarm that fires whenever a value crosses 80 and clears whenever it drops below 80 will fire dozens of times while a temperature hovers at 80. Two settings fix that: require several consecutive breaches before raising, and clear only below a lower threshold (here 70). The gap between the two thresholds should exceed the signal’s normal noise. Alarm floods are a safety problem in their own right, because operators stop reading them.

⚠️
DO NOT PUT SAFETY IN THE CLOUD

Interlocks and emergency stops belong in the equipment’s own control system or the edge, where they work without a network. Cloud analytics can warn, schedule maintenance and optimise; it must not be the only thing that stops a machine.

💡
KEEP THE RAW DATA

Store raw readings in the lake before any filtering beyond the edge deadband. Window sizes, lateness and alarm thresholds will change, and reprocessing history with new settings is only possible if the original readings were kept.

⚖️
DEVICE IDENTITY AND DATA OWNERSHIP

Give every device its own credential (an X.509 certificate per device, not a shared key), scope its MQTT permissions to its own topics, and have a way to revoke it. Telemetry can be personal data when it is tied to a person, such as a vehicle, a wearable or a smart meter, so classify it and apply retention rules in the lake as well as the stream.

📝
HOW THIS WAS CHECKED

streamproc.py was tested with Python 3.12 and pytest==8.2.0: eight tests cover deadband forwarding and heartbeats, deduplication with bounded memory, late readings within allowed lateness, too-late readings counted rather than merged, per-device windows, hysteresis raise and clear, and input validation. It has no dependencies and was not run against a broker or a stream engine.

Key takeaways

  • Edge for millisecond and offline decisions, stream for fleet-wide decisions in seconds, lake for history and models.
  • Filter by exception at the edge with a threshold above sensor noise and a heartbeat.
  • QoS 1 means duplicates: give messages IDs at the source and deduplicate.
  • Aggregate on device event time with a watermark, and decide explicitly what happens to data that is too late.
  • Use consecutive breaches and hysteresis for alarms, keep safety functions out of the cloud, and keep the raw data.

How do you handle late and duplicate telemetry in your pipelines? Share your watermark and alarm settings in the comments.

Glossary

Edge gateway
A device near the equipment that collects, filters and forwards data and can act locally.
Deadband
A filter that forwards a value only when it changes by more than a set amount.
QoS 1
MQTT at-least-once delivery: messages are resent until acknowledged, so duplicates are possible.
Event time
The time a reading was taken at the device, as opposed to when it arrived.
Watermark
A stream engine’s estimate of event-time progress, used to decide when a window is complete.
Tumbling window
Fixed, non-overlapping time intervals used to aggregate a stream.
Hysteresis
Using separate raise and clear thresholds so a signal near the limit does not toggle an alarm.

References & Further Reading


Discover more from C4: Container, Code, Cloud & Context

Subscribe to get the latest posts sent to your email.

Written by

Nithin Mohan TK

Hello Namaste! 👋🏻 I'm an accomplished Enterprise Solution Architect & Engineering Leader with 20+ years of progressive experience architecting, building, and scaling transformative enterprise solutions. ⚜️ Solution Architect / AI & Cloud 📌 Galway, Ireland

Leave a comment

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.