Control & Session

The LLM runtime includes built-in policies for handling transient failures, managing costs, and protecting against provider outages. Configure them via the builder profile or individual policy settings.

Policy pipeline

Every LLM request passes through this policy chain:

Built-in profiles

Use profile() to apply a curated set of policies:

from afk.llms import LLMBuilder

# Development: no retry, no caching, fast failures
dev = LLMBuilder().provider("openai").model("gpt-5.2-mini").profile("development").build()

# Production: retry, circuit breaker, rate limiting, caching
prod = LLMBuilder().provider("openai").model("gpt-5.2-mini").profile("production").build()

Profile	Retry	Cache	Rate Limit	Circuit Breaker	Timeout
`development`	None	None	None	None	30s
`production`	3 attempts, exponential backoff	In-memory, 5min TTL	60 req/min	5 failures → 30s open	60s
`batch`	5 attempts, longer backoff	None	20 req/min	10 failures → 60s open	120s

Individual policies

Configure each policy independently:

Retry
Circuit breaker
Rate limiting
Caching
Timeout

Retry transient LLM failures with exponential backoff.

client = (
    LLMBuilder()
    .provider("openai")
    .model("gpt-5.2-mini")
    .retry(max_attempts=3, backoff_base=1.0, backoff_max=30.0)
    .build()
)

Parameter	Default	Description
`max_attempts`	3	Total attempts (1 initial + 2 retries)
`backoff_base`	1.0	Initial delay in seconds
`backoff_max`	30.0	Maximum delay between retries
`retryable_errors`	`[429, 500, 502, 503]`	HTTP status codes to retry

Stop calling a failing provider to prevent cascading failures.

client = (
    LLMBuilder()
    .provider("openai")
    .model("gpt-5.2-mini")
    .circuit_breaker(failure_threshold=5, recovery_timeout=30.0)
    .build()
)

Prevent exceeding provider rate limits.

client = (
    LLMBuilder()
    .provider("openai")
    .model("gpt-5.2-mini")
    .rate_limit(requests_per_minute=60, tokens_per_minute=100_000)
    .build()
)

Cache identical requests to reduce cost and latency.

client = (
    LLMBuilder()
    .provider("openai")
    .model("gpt-5.2-mini")
    .cache(ttl_seconds=300, max_entries=1000)
    .build()
)

Cache keys are derived from the request content (messages, model, temperature, tools). Identical requests return cached responses.

Hard timeout on LLM requests.

client = (
    LLMBuilder()
    .provider("openai")
    .model("gpt-5.2-mini")
    .timeout(connect_s=5.0, read_s=60.0)
    .build()
)

Fallback chains

Configure a chain of models to try when the primary model fails:

from afk.agents import Agent, FailSafeConfig

agent = Agent(
    name="resilient",
    model="gpt-5.2",
    fail_safe=FailSafeConfig(
        fallback_model_chain=["gpt-5.2-mini", "gpt-5.2-nano"],
    ),
)
# If gpt-5.2 fails → try gpt-5.2-mini → try gpt-5.2-nano

Tuning cheat sheet

Goal	Setting
Reduce costs	Lower temperature, use cheaper model, enable caching
Reduce latency	Enable caching, use faster model, set tight timeout
Handle outages	Enable retry + circuit breaker, add fallback chain
High throughput	Set rate limits high, use batch profile, increase concurrency
Consistent output	Set `temperature=0.0`, enable structured output

Welcome

Learn

Core Concepts

Scaling Up

Communication

LLM Layer

Reliability

Control & Session

Policy pipeline

Built-in profiles

Individual policies

Fallback chains

Tuning cheat sheet

Next steps

Agent Integration

Observability

Welcome

Learn

Core Concepts

Scaling Up

Communication

LLM Layer

Reliability

​Policy pipeline

​Built-in profiles

​Individual policies

​Fallback chains

​Tuning cheat sheet

​Next steps

Agent Integration

Observability

Policy pipeline

Built-in profiles

Individual policies

Fallback chains

Tuning cheat sheet

Next steps