idempotency

A billion-row backfill is a deployable, throttled, resumable job with checkpoints and verification gates. Ad-hoc scripts against prod are how it becomes an incident.

Wall-clock schedules meet daylight-saving transitions: spring forward deletes an hour of cron targets, fall back repeats one. Schedule in UTC, add idempotency keys, rehearse in staging.

Cron is the most-deployed scheduler on earth and the worst-behaved at scale. The scored content is everything crontab cannot do: exactly-once firing, thundering herds, and what happens when the scheduler itself dies.

Timeouts guarantee uncertainty, not failure: the first request may have succeeded after the client gave up. The idempotency key pattern is the industry answer; the scored details are storage, expiry and concurrency.

The difficult decisions are preserving offline intent and enforcing rules that span devices. Interviewers want conflict classes named and handled deliberately, not a CRDT framework proposed on instinct.

Trace a retry through the broker, database and external provider. The guarantee changes at each boundary, and a recorded intent does not prove a payment or email completed.

A disputed invoice needs a reproducible calculation. Retain raw usage events and versioned pricing so each line can be explained and corrected.

Cron fires at wall-clock times; a scheduler reasons about data intervals and blocked frontiers. The scored details are per-partition idempotency and catch-up semantics.