- Fault tolerance — If the process crashes mid-run, the latest checkpoint lets you resume without re-executing already-completed work.
- Human-in-the-loop — When a run pauses for approval, the checkpoint preserves the full conversation and pending state so the run can resume hours or days later.
- Auditability — The checkpoint chain provides a step-by-step record of every phase the run passed through, useful for debugging and compliance.
Checkpoint model
Field reference
Resume behavior
1
Load latest checkpoint
The runner calls
memory.get_state(thread_id, checkpoint_latest_key(run_id)) to
fetch the most recent checkpoint for the given run. If no checkpoint exists, a
AgentCheckpointCorruptionError is raised.2
Validate shape
The checkpoint record must be a dict with the required fields (
run_id,
thread_id, phase, payload). Missing or malformed fields cause an
AgentCheckpointCorruptionError. The runner also normalizes legacy checkpoint
formats through _normalize_checkpoint_record().3
Check for terminal state
If the checkpoint’s
phase is run_terminal and the payload contains a
terminal_result, the run is already complete. The runner returns a pre-resolved
handle with the deserialized AgentResult — no re-execution occurs.4
Load runtime snapshot
For non-terminal checkpoints, the runner loads the full runtime snapshot which
contains the conversation messages, counters, usage aggregates, and any pending
LLM response. This snapshot is used to reconstruct the execution context.
5
Resume execution
The runner calls
run_handle() with the restored snapshot. Execution continues
from the step where the run was interrupted. If a pending_llm_response exists
in the snapshot, the runner skips the LLM call and proceeds directly to tool
execution for that response.Resume code example
What gets stored in the payload
Thepayload field carries different data depending on the checkpoint phase. The
most important payload is the runtime snapshot persisted at step_started and
post_llm phases, which contains everything needed for full resume:
Phase-specific payloads
Beyond the runtime snapshot, individual phase checkpoints carry lighter payloads:Async write-behind behavior
By default, checkpoint writes are asynchronous (RunnerConfig.checkpoint_async_writes=True):
- Writes are queued and flushed by a background writer.
- Repeated
runtime_statewrites may be coalesced (checkpoint_coalesce_runtime_state=True). - Terminal states perform a bounded flush (
checkpoint_flush_timeout_s) before returning.
Effect replay and idempotency
When a run resumes and re-enters a tool batch, the runner checks for previously persisted effect results before re-executing tools. Each tool call’s result is stored with aninput_hash (derived from tool name and arguments) and an output_hash. On
resume, if a matching effect result exists for a tool call ID with a matching input hash,
the stored result is replayed instead of re-executing the tool. This guarantees
idempotent resume for tools with side effects.
The replayed_effect_count field in the runtime snapshot tracks how many tool calls
were satisfied from replay rather than fresh execution.
Common failure scenarios
Missing checkpoint — Callingrunner.resume() with a run_id that has no
checkpoint raises AgentCheckpointCorruptionError. This can happen if the memory store
was cleared or the run never persisted its first checkpoint (crashed before
run_started).
Corrupted payload — If the checkpoint record exists but is not a valid dict or
is missing required keys, AgentCheckpointCorruptionError is raised. The runner does
not attempt partial recovery from corrupted checkpoints.
Pending LLM response corruption — If a checkpoint has pending_llm_response set
but the serialized response cannot be deserialized, the runner raises
AgentCheckpointCorruptionError rather than making a duplicate LLM call.
Stale session tokens — Provider session tokens stored in checkpoints may expire
between the original run and the resume attempt. The runner passes the stored
session_token and checkpoint_token to the provider, but the provider may reject
them. In that case, the LLM call fails and follows the normal retry/fallback chain.