Design a Connected Vehicle Telemetry Ingestion Pipeline
You are designing the telemetry ingestion platform for an automotive company with millions of connected vehicles worldwide. Each vehicle continuously emits location, battery status, speed, and hardware diagnostic events over cellular connections.
The system must ingest high-velocity data points reliably, evaluate safety and diagnostic rules in real time (e.g., collision alerts, severe battery overheating), and persist data for both fleet monitoring queries and downstream machine-learning models.
Your task is to design a high-throughput ingestion architecture that handles high write traffic, network instability from moving vehicles, and provides low-latency streaming alerts without overloading long-term storage.
- Ingest 5-second telemetry payloads from up to 10 million active vehicles.
- Evaluate stream processing rules to trigger real-time driver/fleet alerts within 1 second.
- Store historical telemetry to support point-in-time vehicle status lookups and time-series range queries.
- Support graceful client side batching and backpressure when vehicles reconnect after losing cellular signal.
- Handle sustained write throughput of 2,000,000 events per second.
- Achieve p99 ingestion endpoint latency under 50ms (acknowledging payload receipt).
- Ensure 99.99% availability for the write path.
- Provide durable long-term storage capable of retaining 90 days of high-resolution telemetry and 3 years of aggregated telemetry.
10 million connected vehicles transmitting 500-byte payloads every 5 seconds yields 2,000,000 write QPS and roughly 1 GB/sec (86.4 TB/day) of uncompressed raw telemetry ingest.
- End-to-end architecture diagram detailing edge ingestion, streaming queue, stream processors, and storage tiers.
- Data model and partition key design for message queues and time-series persistence.
- Flow description of the real-time alerting bypass path versus the batch persistence path.
- Capacity estimation and storage lifecycle management strategy (hot, warm, cold).
Strong designs place a stateless ingestion tier behind an API gateway/load balancer that validates payloads and immediately writes to a distributed log (e.g., Apache Kafka, Pulsar) before acknowledging receipt. Evaluates how candidate handles variable cellular connectivity and burst traffic without degrading write latency.
Candidate articulates a clear partition key strategy (e.g., partitioning message streams by Vehicle ID or hash buckets) to ensure per-vehicle event ordering for stream processing, while demonstrating how to prevent partition hotspots when localized vehicle clusters surge.
Demonstrates why standard relational databases fail at this scale. Proposes LSM-tree based time-series/columnar storage (e.g., ClickHouse, Cassandra, or Timestream) for recent operational telemetry paired with object storage (e.g., S3/Parquet) for long-term historical analysis.
Shows a dedicated stream-processing path (e.g., Flink/Spark Streaming) consuming directly from the distributed log to evaluate threshold alerts in-memory, avoiding the disk write latency of primary database persistence for time-sensitive notifications.
Every functional requirement in the brief is visibly served by something on the board, and the non-functional targets are addressed rather than ignored.
Components are labelled, data flows are drawn as connections between them, and the direction of each flow is unambiguous.
Follow-up: How would your partition key strategy and streaming pipeline handle an sudden localized surge where 100,000 vehicles in a single metropolitan area experience extreme weather, triggering massive diagnostic alerts simultaneously without causing hot partitions in your message broker?
The architecture must handle high sustained write throughput with zero data loss for safety events. Network connectivity from vehicles is spotty, meaning telemetry bursts can occur after reconnections. The write path must be decoupled from analytical reads and storage persistence.
- Views
- 9