Telemetry and data pipelines

Key takeaways

Your devices are sending thousands of messages per minute, temperature readings, GPS coordinates, error codes, battery levels. That raw stream is not useful until it is ingested, processed, stored, and made queryable. This lesson explains how telemetry data pipelines work, the architecture that transforms a firehose of device messages into reliable, actionable data.

What you'll learn

What a telemetry pipeline is and its four stages

A telemetry pipeline is the end-to-end data architecture that handles device data from the moment it arrives in the cloud until it is available for dashboards, alerts, and analytics. A well-designed pipeline has four stages: ingestion, processing, storage, and serving. Understanding these stages gives you a mental map of where your data is and what is happening to it at each point.

How the ingestion stage handles incoming data

Ingestion is the front door of your pipeline, where device messages arrive, typically from an MQTT broker, a CoAP gateway, or an HTTP endpoint. Its one job is to accept incoming data reliably and buffer it for processing. It must handle variable load, since devices do not report at perfectly uniform intervals, and survive traffic spikes, such as a thousand devices reconnecting at once after an outage. Message queues and event streaming platforms are the standard tools here; they decouple the rate data arrives from the rate it is processed, acting as a shock absorber between your devices and your processing logic.

What happens during stream processing

Stream processing operates on data as it arrives, in real time or near-real time, turning raw data into meaningful information. Common operations include:

• Parsing and validation: extracting fields from the raw payload and confirming the data is plausible (is the temperature within a sensible range?)
• Enrichment: adding context the device did not send, such as device name, location metadata, fleet assignment, or owner information
• Transformation: converting units, normalizing formats, and computing derived values like speed from sequential GPS coordinates
• Alerting and rules: triggering notifications when data crosses thresholds, such as battery below 10%, temperature above 85°C, or a device offline for more than 30 minutes

Some pipelines also include an edge processing component, where basic filtering and aggregation happen on the device or a local gateway before data reaches the cloud.

How storage choices fit time-series data

Processed telemetry needs to be stored, and the choice matters enormously because IoT telemetry is time-series data: every reading has a timestamp, and most queries involve time ranges. Time-series databases are optimized for this pattern, offering fast writes, efficient compression of sequential data, and built-in aggregation functions (average, min, max over time windows). For long-term archival and complex analytics, many architectures add a data lake alongside the time-series database: the database handles recent, high-frequency queries, while the data lake stores historical data cost-effectively for batch analysis and machine learning.

What the serving and API stage provides

The serving stage makes processed, stored data available to applications. It includes REST APIs for programmatic access, WebSocket connections for real-time dashboards, and query interfaces for ad-hoc analysis. A good serving layer supports both real-time queries ("what is the current temperature of device X?") and historical queries ("what was the average temperature across all devices last month?").

How stream processing and batch processing differ

Most IoT pipelines use both, because they are complementary rather than competing. Stream processing handles data as it arrives, with latencies measured in seconds; use it for real-time alerts, live dashboards, and immediate operational decisions. Batch processing runs on accumulated data at scheduled intervals, hourly, daily, or weekly; use it for trend analysis, reporting, machine learning model training, and cost optimization. A common pattern streams data for immediate alerts while simultaneously feeding it into a data lake for nightly batch analytics.

Why cellular IoT creates data quality challenges

Cellular IoT introduces specific data quality issues. Devices may retransmit messages if they do not receive an acknowledgment (especially with MQTT QoS 1), network interruptions may cause gaps, and poor clock synchronization may produce inaccurate timestamps. A robust pipeline therefore includes deduplication logic (recognizing and discarding duplicate messages), gap detection (identifying missing data points), and timestamp reconciliation.

Lesson 5.3 Self-assessment

Frequently asked questions