Workers and Recovery
Graph node executors routed tokens. Durable workers lease commands.
WorkflowWorkerSupervisor combines bounded recovery, timer expiry, activity work, and spawned-workflow reconciliation. Start one with a globally unique workerId on every replica.
Roles
- API / decider — start runs, complete tasks, deliver signals, drive replay.
- Activity worker — lease commands, run handlers, write completions.
- Timer / recovery — fire timers, expire tasks, reclaim leases, resume interrupted decisions.
- Spawn reconciler — create child runs once and complete parent spawn commands.
Fencing
- Only the current
claimGenerationowner may complete a command. - Heartbeats must be shorter than the lease.
FOR UPDATE SKIP LOCKEDclaims work without blocking other workers.- A late completion after reassignment is rejected.
Recovery Inbox
Creating a run or recording a new fact sets decision_pending. A successful decision clears it. Workflows waiting for an activity, signal, timer, user task, or child run stay blocked without consuming recovery capacity.
Operator Actions
Retry, redrive, and cancel are audited. They record actor, reason, and timestamp. The inspector exposes the same actions.