> ## Documentation Index
> Fetch the complete documentation index at: https://afk.arpan.sh/llms.txt
> Use this file to discover all available pages before exploring further.

# LLM Layer

> Provider-portable LLM runtime with retry, caching, circuit breaking, and middleware.

The LLM layer normalizes communication with language models across all supported providers. Your agent code uses provider-agnostic contracts (`LLMRequest` / `LLMResponse`) while built-in adapters handle the provider-specific details.

## The LLMBuilder

Create LLM clients with the builder pattern:

```python theme={null}
from afk.llms import LLMBuilder

client = (
    LLMBuilder()
    .provider("openai")
    .model("gpt-5.5")
    .build()
)
```

<Steps>
  <Step title="Choose a provider">
    ```python theme={null}
    builder = LLMBuilder().provider("openai")
    # Also: "anthropic", "litellm", or a custom adapter
    ```
  </Step>

  <Step title="Set the model">
    ```python theme={null}
    builder = builder.model("gpt-5.5")
    ```
  </Step>

  <Step title="Add policies (optional)">
    ```python theme={null}
    builder = builder.profile("production")
    # retry, timeout, rate limit, circuit breaker
    ```
  </Step>

  <Step title="Build">
    ```python theme={null}
    client = builder.build()
    ```
  </Step>
</Steps>

## Middleware

The LLM layer supports middleware for intercepting and transforming requests and responses. Use middleware for logging, tracing, caching, and custom request/response handling.

### Built-in middleware

```python theme={null}
from afk.llms import LLMBuilder
from afk.llms.middleware import MiddlewareStack
from afk.llms.middleware.timeout import (
    TimeoutMiddleware,
    EmbedTimeoutMiddleware,
    StreamTimeoutMiddleware,
    TimeoutConfig,
)

config = TimeoutConfig(
    default_timeout_s=30.0,
    chat_timeout_s=60.0,
    embed_timeout_s=15.0,
    stream_timeout_s=45.0,
)

stack = MiddlewareStack(
    chat=[TimeoutMiddleware(config)],
    embed=[EmbedTimeoutMiddleware(config)],
    stream=[StreamTimeoutMiddleware(config)],
)

client = (
    LLMBuilder()
    .provider("openai")
    .model("gpt-5.5")
    .with_middlewares(stack)
    .build()
)
```

### Custom middleware

```python theme={null}
from afk.llms import LLMBuilder, LLMRequest, LLMResponse
from afk.llms.middleware import MiddlewareStack

async def tracing_middleware(call_next, req: LLMRequest) -> LLMResponse:
    """Add tracing metadata to requests."""
    req.metadata = req.metadata or {}
    req.metadata["trace_id"] = generate_trace_id()
    req.metadata["span_name"] = "llm.chat"
    return await call_next(req)

client = (
    LLMBuilder()
    .provider("openai")
    .model("gpt-5.5")
    .with_middlewares(MiddlewareStack(
        chat=[tracing_middleware],
        embed=[],
        stream=[],
    ))
    .build()
)
```

### Middleware protocols

| Protocol              | Operation          | Signature                                                       |
| --------------------- | ------------------ | --------------------------------------------------------------- |
| `LLMChatMiddleware`   | Non-streaming chat | `async (call_next, req: LLMRequest) -> LLMResponse`             |
| `LLMEmbedMiddleware`  | Embeddings         | `async (call_next, req: EmbeddingRequest) -> EmbeddingResponse` |
| `LLMStreamMiddleware` | Streaming chat     | `(call_next, req: LLMRequest) -> AsyncIterator[LLMStreamEvent]` |

## Supported providers

<CardGroup cols={3}>
  <Card title="OpenAI" icon="circle-o">
    GPT-4.1, GPT-4.1-mini, GPT-4.1-nano, o-series
  </Card>

  <Card title="Anthropic" icon="circle-a">
    Claude Opus and Sonnet families
  </Card>

  <Card title="LiteLLM" icon="circle-l">
    100+ providers via the LiteLLM proxy
  </Card>
</CardGroup>

All providers expose the same `LLMClient` interface. Your agent code never touches provider-specific types.

## Example model choices

Use these as starting points, then verify model availability, pricing, and context limits with your provider account.

| Scenario                   | Starting point                           |
| -------------------------- | ---------------------------------------- |
| General purpose            | OpenAI `gpt-5.5`                         |
| Complex reasoning          | OpenAI `gpt-5.5` or Anthropic `opus-4.8` |
| Cost-sensitive             | OpenAI `gpt-5.5`                         |
| Non-OpenAI/Anthropic model | LiteLLM adapter                          |
| Custom or self-hosted      | Custom adapter                           |

## How agents use the LLM layer

You rarely build `LLMClient` directly. Agents resolve their model automatically:

```python theme={null}
# Option 1: Model name (auto-resolved)
agent = Agent(name="demo", model="gpt-5.5", instructions="Answer directly.")

# Option 2: Pre-built client (full control)
client = LLMBuilder().provider("openai").model("gpt-5.5").profile("production").build()
agent = Agent(name="demo", model=client, instructions="Answer directly.")
```

## Next steps

<CardGroup cols={2}>
  <Card title="Contracts" icon="file-contract" href="/llms/contracts">
    LLMRequest / LLMResponse — what flows across the boundary.
  </Card>

  <Card title="Adapters" icon="plug" href="/llms/adapters">
    Built-in providers and custom adapter registration.
  </Card>
</CardGroup>