Design an Automated Recurring Subscription Billing Engine
Design the core billing execution subsystem for a global SaaS subscription platform. On any given day, millions of subscription invoices become due. Your system must trigger billing, execute charges against third-party payment gateways (e.g., Stripe, Adyen), handle transient network failures with smart retries, and guarantee that no customer is charged twice for the same billing cycle even in the face of worker crashes, network partitions, or duplicate webhook deliveries.
- Schedule and trigger recurring billing events when a subscription cycle completes.
- Execute charges via external payment gateways using strict idempotency controls.
- Ingest and process payment result webhooks asynchronously from payment gateways.
- Maintain accurate state transitions (Pending, Executing, Succeeded, Failed, Retrying) for every invoice and payment attempt.
- Zero duplicate charges under any failure scenario (strict idempotency).
- High availability (99.99%) for webhook ingestion and durable charge state tracking.
- Bounded payment execution latency (p95 latency under 3s, excluding third-party gateway response delays).
- Strong consistency for individual invoice and transaction settlement records.
50 million active subscriptions; 1.6 million invoices processed daily, peaking at 5,000 charge executions per second during UTC midnight billing windows. Gateway response latency averages 1.5s with up to a 2% timeout rate.
- High-level system architecture showing the scheduler, charge execution workers, idempotency store, and payment gateway integration.
- Data schema and explicit state machine diagram for Invoices, Payment Attempts, and Idempotency Keys.
- Detailed failure handling workflow for HTTP timeouts, worker crashes mid-flight, and duplicate webhook ingestion.
Evaluates deterministic idempotency key generation (e.g., hash of invoice_id, renewal_cycle, attempt_number), transactional key reservation before gateway invocation, and key propagation to third-party payment gateways.
Checks for clear, unambiguous state transitions (e.g., Initiated -> Gateway_Pending -> Succeeded/Failed/Ambiguous) and distributed locking or database row-locking mechanisms that prevent concurrent workers from processing the same invoice.
Assesses how the design resolves ambiguous payment states caused by network timeouts, including active gateway inquiry APIs, out-of-order webhook handling via persistent idempotency keys, and asynchronous reconciliation jobs.
Evaluates how time-bucketed subscription renewals are partitioned and distributed across execution workers to handle midnight UTC load spikes without hotspots or skipped invoices.
Every functional requirement in the brief is visibly served by something on the board, and the non-functional targets are addressed rather than ignored.
Components are labelled, data flows are drawn as connections between them, and the direction of each flow is unambiguous.
Follow-up: If an external payment gateway times out during a charge request, the payment status remains unknown. How do you design your retry, inquiry, and reconciliation mechanisms to prevent duplicate charges while ensuring valid charges are eventually settled?
Must handle 50 million active subscriptions globally. Peak billing throughput occurs at midnight UTC across monthly billing boundaries, reaching up to 5,000 charge executions per second. Payments involve external third-party payment gateways with variable latency (500ms to 5s) and potential timeout responses where payment outcome is ambiguous.
- Views
- 21