Skip to main content
Checkpoints are the persistence mechanism that allows AFK agent runs to survive process restarts, crashes, and intentional pauses. At key boundaries during execution (step start, pre-LLM call, post-tool batch, run terminal), the runner writes a checkpoint record to the memory store. Each checkpoint captures enough state to reconstruct the execution context and resume from where the run left off. Checkpoints matter for three reasons:
  1. Fault tolerance — If the process crashes mid-run, the latest checkpoint lets you resume without re-executing already-completed work.
  2. Human-in-the-loop — When a run pauses for approval, the checkpoint preserves the full conversation and pending state so the run can resume hours or days later.
  3. Auditability — The checkpoint chain provides a step-by-step record of every phase the run passed through, useful for debugging and compliance.

Checkpoint model

Field reference

Resume behavior

1

Load latest checkpoint

The runner calls memory.get_state(thread_id, checkpoint_latest_key(run_id)) to fetch the most recent checkpoint for the given run. If no checkpoint exists, a AgentCheckpointCorruptionError is raised.
2

Validate shape

The checkpoint record must be a dict with the required fields (run_id, thread_id, phase, payload). Missing or malformed fields cause an AgentCheckpointCorruptionError. The runner also normalizes legacy checkpoint formats through _normalize_checkpoint_record().
3

Check for terminal state

If the checkpoint’s phase is run_terminal and the payload contains a terminal_result, the run is already complete. The runner returns a pre-resolved handle with the deserialized AgentResult — no re-execution occurs.
4

Load runtime snapshot

For non-terminal checkpoints, the runner loads the full runtime snapshot which contains the conversation messages, counters, usage aggregates, and any pending LLM response. This snapshot is used to reconstruct the execution context.
5

Resume execution

The runner calls run_handle() with the restored snapshot. Execution continues from the step where the run was interrupted. If a pending_llm_response exists in the snapshot, the runner skips the LLM call and proceeds directly to tool execution for that response.

Resume code example

What gets stored in the payload

The payload field carries different data depending on the checkpoint phase. The most important payload is the runtime snapshot persisted at step_started and post_llm phases, which contains everything needed for full resume:

Phase-specific payloads

Beyond the runtime snapshot, individual phase checkpoints carry lighter payloads:

Async write-behind behavior

By default, checkpoint writes are asynchronous (RunnerConfig.checkpoint_async_writes=True):
  • Writes are queued and flushed by a background writer.
  • Repeated runtime_state writes may be coalesced (checkpoint_coalesce_runtime_state=True).
  • Terminal states perform a bounded flush (checkpoint_flush_timeout_s) before returning.
This improves loop throughput while preserving terminal durability.

Effect replay and idempotency

When a run resumes and re-enters a tool batch, the runner checks for previously persisted effect results before re-executing tools. Each tool call’s result is stored with an input_hash (derived from tool name and arguments) and an output_hash. On resume, if a matching effect result exists for a tool call ID with a matching input hash, the stored result is replayed instead of re-executing the tool. This guarantees idempotent resume for tools with side effects. The replayed_effect_count field in the runtime snapshot tracks how many tool calls were satisfied from replay rather than fresh execution.

Common failure scenarios

Missing checkpoint — Calling runner.resume() with a run_id that has no checkpoint raises AgentCheckpointCorruptionError. This can happen if the memory store was cleared or the run never persisted its first checkpoint (crashed before run_started). Corrupted payload — If the checkpoint record exists but is not a valid dict or is missing required keys, AgentCheckpointCorruptionError is raised. The runner does not attempt partial recovery from corrupted checkpoints. Pending LLM response corruption — If a checkpoint has pending_llm_response set but the serialized response cannot be deserialized, the runner raises AgentCheckpointCorruptionError rather than making a duplicate LLM call. Stale session tokens — Provider session tokens stored in checkpoints may expire between the original run and the resume attempt. The runner passes the stored session_token and checkpoint_token to the provider, but the provider may reject them. In that case, the LLM call fails and follows the normal retry/fallback chain.