Race-free job dispatch with Postgres row-level locking

Context

Orbit dispatches jobs to many concurrent workers. The core hazard is a race: two workers read the same "pending" job and both start running it, violating at-least-once semantics by turning it into "at-least-twice."

Decision and tradeoffs

I already had PostgreSQL as the source of truth for job state. I could have added a dedicated broker (RabbitMQ, Redis streams), but that introduces a second system to keep consistent with the database. Instead I made the claim itself atomic in Postgres using SELECT ... FOR UPDATE SKIP LOCKED.

Tradeoff: this couples dispatch to the database's throughput, but for long-running jobs (seconds to minutes) the claim rate is low and the simplicity of a single source of truth wins.

How it works

SKIP LOCKED lets each worker grab a different unlocked row without blocking on rows another worker already holds:

UPDATE jobs
SET status = 'running', worker_id = $1, leased_until = now() + interval '30s'
WHERE id = (
  SELECT id FROM jobs
  WHERE status = 'pending'
  ORDER BY priority DESC, created_at
  FOR UPDATE SKIP LOCKED
  LIMIT 1
)
RETURNING id;

The whole claim is one transaction, so a job moves to running exactly once. A separate reaper expires leased_until to recover jobs from crashed workers.

Outcome

No duplicate claims under concurrency, at-least-once delivery, and no extra infrastructure — plus a job_attempts table that records every lease, exit code, and log reference for full auditability.