Payout orchestration for a digital lender

The lender disbursed loans to customers' bank accounts through two payment providers. Payouts were fired straight from the API request, retried by a cron job, and reconciled against provider reports in a spreadsheet each morning.

1. The API writes the payout and an outbox event in one Postgres transaction. If the request dies, nothing is half-done.
2. A relay publishes outbox rows to Kafka, keyed by payout ID so one payout is never handled by two workers at once.
3. Go workers dispatch to providers under per-provider rate limits and fall back to the other provider on sustained errors.
4. Webhooks and status polling update the ledger. The reconciler compares ledger and provider reports hourly and flags any mismatch.

Two weeks after launch, Provider B started returning timeouts for payouts that had actually succeeded. Our workers marked them retryable, and the optimistic lock stopped a second dispatch, but 212 payouts sat in limbo for six hours until the reconciler caught up.

The fix was to treat a timeout as unknown, not failed, and query the provider's status endpoint before any retry. If I did it again, I'd model "unknown" as a first-class state from day one and alert on how long payouts stay in it.