Design an Event-Driven Multi-Warehouse Order Fulfillment Choreography Platform
You are designing the core fulfillment backend for a global e-commerce enterprise. When a customer places an order, the system must coordinate multiple independent services—Inventory Reservation, Payment Authorization, Fraud Assessment, Warehouse Allocation, and Shipping Dispatch—without relying on a monolithic orchestrator or synchronous RPC chains. Instead, the architecture must leverage event-driven choreography where each domain service listens for domain events, updates its local state, and emits downstream events.\n\nThe challenge lies in managing complex distributed state transitions under high concurrency (e.g., flash sales), handling out-of-order or duplicate event deliveries, enforcing eventual consistency across heterogeneous services, and handling graceful compensation (e.g., releasing inventory if payment fails) without cascading failures or deadlocks.
- Coordinate multi-step order processing (Fraud Check, Inventory Hold, Payment Capture, Order Allocation, Shipping Label Generation) via event choreography.
- Support asynchronous compensation workflows (e.g., stock release, payment refund) upon step failure or processing timeouts.
- Enforce strict per-order event processing sequence while maintaining horizontal scalability across independent domain consumers.
- Provide deterministic event deduplication and idempotent event handling across all domain consumer services.
- End-to-end processing latency < 5 seconds for 99% of orders from creation to dispatch readiness.
- 99.99% system availability with zero data loss for completed or in-flight financial and inventory events.
- High throughput under flash-sale spikes without cross-SKU head-of-line blocking.
- Eventual consistency guaranteed across domain data stores within 10 seconds under peak load.
Peak volume of 50,000 orders/sec during global flash sales. Up to 10,000 orders/sec targeted at a single hot SKU. Daily volume of 100 million domain events generated across 5 distinct microservice domains.
- High-level architecture showing event bus topologies, message queues, state stores, and domain consumer boundaries.
- Detailed choreography sequence / state machine mapping events, local domain state updates, and compensating triggers.
- Partitioning and sharding strategy for event streams to handle hot SKUs and prevent queue head-of-line blocking.
- Failure recovery and idempotency mechanism diagram showing out-of-order message handling and transactional outbox/inbox patterns.
Evaluates how effectively the candidate models decentralized state transitions through domain events (e.g., OrderPlaced, StockReserved, PaymentAuthorized, AllocationFailed) without a centralized orchestrator, including explicitly defined compensating event flows and failure boundaries.
Assesses the strategy for partitioning message streams (e.g., ordering by Order ID vs SKU ID), avoiding head-of-line blocking during flash sales, and handling consumer group scaling for high-concurrency event streams.
Checks whether the design includes a robust transactional inbox/outbox pattern, deduplication stores, sequence vector checks, or state guardrails to safely process duplicate, delayed, or out-of-order events.
Evaluates the mechanism for detecting stuck in-flight transactions (e.g., payment pending timeout), triggering asynchronous compensation events, and reconciling mismatched states across domain databases.
Assesses the handling of non-retryable processing errors, dead-letter queue (DLQ) topology, replay strategies, and backpressure isolation between independent domain event streams.
Every functional requirement in the brief is visibly served by something on the board, and the non-functional targets are addressed rather than ignored.
Components are labelled, data flows are drawn as connections between them, and the direction of each flow is unambiguous.
Follow-up: Suppose a network partition isolates the Payment Service for 15 minutes while millions of events accumulate in the event log. How does your choreography architecture prevent queue backpressure from stalling upstream inventory holds, and how do you handle state reconciliation once the partition heals without emitting out-of-order compensating events?
The system must operate entirely on event-driven messaging pipelines without synchronous cross-service HTTP/gRPC blocking calls during the core fulfillment path. The system must support high-volume flash sales on isolated SKUs without queue partitioning hotspots blocking unrelated orders. Total end-to-end fulfillment processing time from order placement to shipping manifest generation must complete within 5 seconds at the 99th percentile.
- Views
- 7