# Setting Limits – Inspect

## Overview

In open-ended model conversations (for example, an agent evaluation with tool usage) it’s possible that a model will get “stuck” attempting to perform a task with no realistic prospect of completing it. Further, sometimes models will call commands in a sandbox that take an extremely long time (or worst case, hang indefinitely).

For this type of evaluation it’s normally a good idea to set limits on some combination of total time, total messages, turns, tokens used, and/or cost. This article covers:

1.  [Sample Limits](#sample-limits) — limits applied to individual samples within a task.
2.  [Scoped Limits](#scoped-limits) — limits applied to arbitrary blocks of code.
3.  [Agent Limits](#agent-limits) — limits applied to agent execution.

## Sample Limits

Sample limits don’t result in errors, but rather an early exit from execution (samples that encounter limits are still scored, albeit nearly always as “incorrect”).

### Time Limit

Here we set a `time_limit` of 15 minutes (15 x 60 seconds) for each sample within a task:

``` python
@task
def intercode_ctf():
    return Task(
        dataset=read_dataset(),
        solver=[
            system_message("system.txt"),
            use_tools([bash(timeout=3 * 60)]),
            generate(),
        ],
        time_limit=15 * 60,
        scorer=includes(),
        sandbox="docker",
    )
```

Note that we also set a timeout of 3 minutes for the [bash()](./reference/inspect_ai.tool.html.md#bash) command. This isn’t required but is often a good idea so that a single wayward bash command doesn’t consume the entire `time_limit`.

We can also specify a time limit at the CLI or when calling [eval()](./reference/inspect_ai.html.md#eval):

``` bash
inspect eval ctf.py --time-limit 900
```

Appropriate timeouts will vary depending on the nature of your task so please view the above as examples only rather than recommend values.

### Working Limit

The `working_limit` differs from the `time_limit` in that it measures only the time spent working (as opposed to retrying in response to rate limits or waiting on other shared resources). Working time is computed based on total clock time minus time spent on (a) unsuccessful model generations (e.g. rate limited requests); and (b) waiting on shared resources (e.g. Docker containers or subprocess execution).

> **NOTE:**
>
> In order to distinguish successful generate requests from rate limited and retried requests, Inspect installs hooks into the HTTP client of various model packages. This is not possible for some models (`azureai`) and in these cases the `working_time` will include any internal retries that the model client performs.

Here we set an `working_limit` of 10 minutes (10 x 60 seconds) for each sample within a task:

``` python
@task
def intercode_ctf():
    return Task(
        dataset=read_dataset(),
        solver=[
            system_message("system.txt"),
            use_tools([bash(timeout=3 * 60)]),
            generate(),
        ],
        working_limit=10 * 60,
        scorer=includes(),
        sandbox="docker",
    )
```

### Message Limit

Message limits enforce a limit on the number of messages in any conversation (e.g. a [TaskState](./reference/inspect_ai.solver.html.md#taskstate), [AgentState](./reference/inspect_ai.agent.html.md#agentstate), or any input to [generate()](./reference/inspect_ai.solver.html.md#generate)).

Message limits are checked:

- Whenever you call [generate()](./reference/inspect_ai.solver.html.md#generate) on any model. A [LimitExceededError](./reference/inspect_ai.util.html.md#limitexceedederror) will be raised if the number of messages passed in `input` parameter to [generate()](./reference/inspect_ai.solver.html.md#generate) is equal to or exceeds the limit. This is to avoid proceeding to another (wasteful) generate call if we’re already at the limit.

- Whenever `TaskState.messages` or `AgentState.messages` is mutated, but a [LimitExceededError](./reference/inspect_ai.util.html.md#limitexceedederror) is only raised if the count exceeds the limit.

Here we set a `message_limit` of 30 for each sample within a task:

``` python
@task
def intercode_ctf():
    return Task(
        dataset=read_dataset(),
        solver=[
            system_message("system.txt"),
            use_tools([bash(timeout=120)]),
            generate(),
        ],
        message_limit=30,
        scorer=includes(),
        sandbox="docker",
    )
```

This sets a limit of 30 total messages in a conversation before the model is forced to give up. At that point, whatever `output` happens to be in the [TaskState](./reference/inspect_ai.solver.html.md#taskstate) will be scored (presumably leading to a score of incorrect).

### Token Limit

Token usage (using `total_tokens` of [ModelUsage](./reference/inspect_ai.model.html.md#modelusage)) is automatically recorded for all models. Token limits are checked whenever [generate()](./reference/inspect_ai.solver.html.md#generate) is called. By default token limits meter all tokens; use a limit with type `"output"` to meter only output tokens (which include reasoning tokens), or an arithmetic formula to meter a weighted mix (see below).

Here we set a `token_limit` of 500K for each sample within a task:

``` python
@task
def intercode_ctf():
    return Task(
        dataset=read_dataset(),
        solver=[
            system_message("system.txt"),
            use_tools([bash(timeout=120)]),
            generate(),
        ],
        token_limit=(1024*500),
        scorer=includes(),
        sandbox="docker",
    )
```

The `token_limit` also accepts strings with magnitude suffixes (e.g. `"500k"` or `"1m"`) and can be scoped to only output tokens by using a [TokenLimit](./reference/inspect_ai.util.html.md#tokenlimit) value or an `"output:"` prefix:

``` python
from inspect_ai.util import TokenLimit

Task(
    ...,
    token_limit=TokenLimit(tokens=1024*500, type="output")
)

# equivalent string form (also usable from the CLI, e.g.
# `inspect eval ctf.py --token-limit output:500k`)
Task(
    ...,
    token_limit="output:500k"
)
```

Output token limits meter only the tokens generated by the model (including reasoning tokens), so they are unaffected by growth in conversation input tokens.

The `type` can also be an arithmetic formula over the variables `input` and `output`, letting you meter a weighted mix of token types (e.g. to weight cheaper input tokens down):

``` python
# meter 10% of input tokens plus all output tokens
Task(..., token_limit=TokenLimit(tokens=1024*500, type="(input * 0.1) + output"))

# equivalent string / CLI form:
#   inspect eval ctf.py --token-limit "(input*0.1)+output:500k"
Task(..., token_limit="(input*0.1)+output:500k")
```

Formulas support `+`, `-`, `*`, `/`, parentheses, and unary minus. In a formula `input` is the true prompt size (including cached tokens) and `output` includes reasoning tokens; the computed value is floored to an integer.

> **IMPORTANT: Important**
>
> It’s important to note that the `token_limit` is for all tokens used within the execution of a sample. If you want to limit the number of tokens that can be yielded from a single call to the model you should use the `max_tokens` generation option.

### Turn Limit

A “turn” is a single model generation (one call to the model that produces an assistant message). Turn limits are distinct from message limits, which count *all* messages in the conversation (user, assistant, and tool messages). One turn often results in several messages (e.g. an assistant message plus the tool messages from its tool calls).

A turn is recorded once per top-level [generate()](./reference/inspect_ai.solver.html.md#generate) call (after retries and fallbacks have resolved, and including cache hits). Generations made via `model.compact()` do not count as turns. Turn limits are checked whenever a turn is recorded.

Here we set a `turn_limit` of 300 for each sample within a task:

``` python
@task
def intercode_ctf():
    return Task(
        dataset=read_dataset(),
        solver=[
            system_message("system.txt"),
            use_tools([bash(timeout=120)]),
            generate(),
        ],
        turn_limit=300,
        scorer=includes(),
        sandbox="docker",
    )
```

This limits the agent to 300 model generations before it is forced to give up. As with other sample limits, whatever `output` happens to be in the [TaskState](./reference/inspect_ai.solver.html.md#taskstate) at that point will be scored.

### Cost Limit

Cost is computed from token usage and model cost data (see [Model Cost](#model-cost)). Cost limits are checked whenever [generate()](./reference/inspect_ai.solver.html.md#generate) is called.

Here we set a `cost_limit` of \$2.00 for each sample within a task:

``` python
@task
def intercode_ctf():
    return Task(
        dataset=read_dataset(),
        solver=[
            system_message("system.txt"),
            use_tools([bash(timeout=120)]),
            generate(),
        ],
        cost_limit=2.00,
        scorer=includes(),
        sandbox="docker",
    )
```

> **IMPORTANT: Important**
>
> The `cost_limit` requires model cost data to be configured via [set_model_cost()](./reference/inspect_ai.model.html.md#set_model_cost) or `--model-cost-config`. An error will be raised if a cost limit is set without cost data for all models used in the evaluation.

#### Model Cost

Cost tracking requires cost data for each model present in the eval or eval set. There are two ways to set cost data:

**Python API:**

``` python
from inspect_ai.model import set_model_cost, ModelCost

set_model_cost("openai/gpt-4o", ModelCost(
    input=2.50, output=10.00,
    input_cache_write=0, input_cache_read=1.25,
))
```

**CLI (YAML or JSON file):**

Each model needs a price set for `input`, `output`, `input_cache_write`, and `input_cache_read`. Prices should be given in dollars per million tokens. Set unused fields to `0`.

Below is an example cost config file given in YAML:

``` yaml
openai/gpt-4o:
    input: 2.50
    output: 10.00
    input_cache_write: 0
    input_cache_read: 1.25
anthropic/claude-sonnet-4-5-20250514:
    input: 3.00
    output: 15.00
    input_cache_write: 3.75
    input_cache_read: 0.30
```

(As of Feb 9 2026, all major model providers count reasoning tokens as output tokens, so no separate price needs to be provided for reasoning tokens. If your use case requires separate calculation of reasoning token prices, contact us.)

When model cost data is configured, costs will be tracked for the sample as a whole, as well as any events within the sample that have a ModelUsage field.

Additionally, configuring model cost data allows setting sample cost limits:

``` bash
inspect eval ctf.py --model-cost-config pricing.yaml --cost-limit 2.00
```

### Custom Limit

When limits are exceeded, a [LimitExceededError](./reference/inspect_ai.util.html.md#limitexceedederror) is raised and caught by the main Inspect sample execution logic. If you want to create custom limit types, you can enforce them by raising a [LimitExceededError](./reference/inspect_ai.util.html.md#limitexceedederror) as follows:

``` python
from inspect_ai.util import LimitExceededError

raise LimitExceededError(
    "custom",
    value=value,
    limit=limit,
    message=f"A custom limit was exceeded: {value}"
)
```

### Query Usage

We can determine how much of a sample limit has been used, what the limit is, and how much of the resource is remaining:

``` python
sample_time_limit = sample_limits().time
print(f"{sample_time_limit.remaining:.0f} seconds remaining")
```

Note that [sample_limits()](./reference/inspect_ai.util.html.md#sample_limits) only retrieves the sample-level limits, not [scoped limits](#scoped-limits) or [agent limits](#agent-limits).

Sample limit usage is also recorded in the log: each [EvalSample](./reference/inspect_ai.log.html.md#evalsample) (and its summary) carries `turn_count` (number of top-level model generations) along with its token limit configuration and metered usage (`token_limit`, `token_limit_type`, and `token_limit_usage` — `None` when no token limit was configured). `turn_count` and `token_limit_usage` are default [samples_df()](./dataframe.html.md) columns (add e.g. `SampleColumn("token_limit", path="token_limit")` for the others), and all of them are shown live by [`inspect ctl sample list`](./control-channel.html.md#sample-status).

## Scoped Limits

You can also apply limits at arbitrary scopes, independent of the sample or agent-scoped limits. For instance, applied to a specific block of code. For example:

``` python
with token_limit(1024*500):
    ...
```

A [LimitExceededError](./reference/inspect_ai.util.html.md#limitexceedederror) will be raised if the limit is exceeded. The `source` field on [LimitExceededError](./reference/inspect_ai.util.html.md#limitexceedederror) will be set to the [Limit](./reference/inspect_ai.util.html.md#limit) instance that was exceeded.

When catching [LimitExceededError](./reference/inspect_ai.util.html.md#limitexceedederror), ensure that your `try` block encompasses the usage of the limit context manager as some [LimitExceededError](./reference/inspect_ai.util.html.md#limitexceedederror) exceptions are raised at the scope of closing the context manager:

``` python
try:
    with token_limit(1024*500):
        ...
except LimitExceededError:
    ...
```

The [apply_limits()](./reference/inspect_ai.util.html.md#apply_limits) function accepts a list of [Limit](./reference/inspect_ai.util.html.md#limit) instances. If any of the limits passed in are exceeded, the `limit_error` property on the `LimitScope` yielded when opening the context manager will be set to the exception. By default, all [LimitExceededError](./reference/inspect_ai.util.html.md#limitexceedederror) exceptions are propagated. However, if `catch_errors` is true, errors which are as a direct result of exceeding one of the limits passed to it will be caught. It will always allow [LimitExceededError](./reference/inspect_ai.util.html.md#limitexceedederror) exceptions triggered by other limits (e.g. Sample scoped limits) to propagate up the call stack.

``` python
with apply_limits(
    [token_limit(1000), message_limit(10)], catch_errors=True
) as limit_scope:
    ...
if limit_scope.limit_error:
    print(f"One of our limits was hit: {limit_scope.limit_error}")
```

### Checking Usage

You can query how much of a limited resource has been used so far via the `usage` property of a scoped limit. For example:

``` python
with token_limit(10_000) as limit:
    await generate()
    print(f"Used {limit.usage:,} of 10,000 tokens")
```

If you’re passing the limit instance to [apply_limits()](./reference/inspect_ai.util.html.md#apply_limits) or an agent and want to query the usage, you should keep a reference to it:

``` python
limit = token_limit(10_000)
with apply_limits([limit]):
    await generate()
    print(f"Used {limit.usage:,} of 10,000 tokens")
```

### Time Limit

To limit the wall clock time to 15 minutes within a block of code:

``` python
with time_limit(15 * 60):
    ...
```

Internally, this uses [`anyio`’s cancellation scopes](https://anyio.readthedocs.io/en/stable/cancellation.html). The block will be cancelled at the first yield point (e.g. `await` statement).

### Working Limit

The `working_limit` differs from the `time_limit` in that it measures only the time spent working (as opposed to retrying in response to rate limits or waiting on other shared resources). Working time is computed based on total clock time minus time spent on (a) unsuccessful model generations (e.g. rate limited requests); and (b) waiting on shared resources (e.g. Docker containers or subprocess execution).

> **NOTE:**
>
> In order to distinguish successful generate requests from rate limited and retried requests, Inspect installs hooks into the HTTP client of various model packages. This is not possible for some models (`azureai`) and in these cases the `working_time` will include any internal retries that the model client performs.

To limit the working time to 10 minutes:

``` python
with working_limit(10 * 60):
    ...
```

Unlike time limits, this is not driven by `anyio`. It is checked periodically such as from [generate()](./reference/inspect_ai.solver.html.md#generate) and after each [Solver](./reference/inspect_ai.solver.html.md#solver) runs.

### Message Limit

Message limits enforce a limit on the number of messages in any conversation (e.g. a [TaskState](./reference/inspect_ai.solver.html.md#taskstate), [AgentState](./reference/inspect_ai.agent.html.md#agentstate), or any input to [generate()](./reference/inspect_ai.solver.html.md#generate)).

Message limits are checked:

- Whenever you call [generate()](./reference/inspect_ai.solver.html.md#generate) on any model. A [LimitExceededError](./reference/inspect_ai.util.html.md#limitexceedederror) will be raised if the number of messages passed in `input` parameter to [generate()](./reference/inspect_ai.solver.html.md#generate) is equal to or exceeds the limit. This is to avoid proceeding to another (wasteful) generate call if we’re already at the limit.

- Whenever `TaskState.messages` or `AgentState.messages` is mutated, but a [LimitExceededError](./reference/inspect_ai.util.html.md#limitexceedederror) is only raised if the count exceeds the limit.

Scoped message limits behave differently to scoped token limits in that only the innermost active [message_limit()](./reference/inspect_ai.util.html.md#message_limit) is checked.

To limit the conversation length within a block of code:

``` python
@agent
def myagent() -> Agent:
    async def execute(state: AgentState):

        with message_limit(50):
            # A LimitExceededError will be raised when the limit is exceeded
            ...
            with message_limit(None):
                # The limit of 50 is temporarily removed in this block of code
                ...
```

> **IMPORTANT: Important**
>
> It’s important to note that [message_limit()](./reference/inspect_ai.util.html.md#message_limit) limits the total number of messages in the conversation, not just “new” messages appended by an agent.

### Token Limit

Token usage (using `total_tokens` of [ModelUsage](./reference/inspect_ai.model.html.md#modelusage)) is automatically recorded for all models. Token limits are checked whenever [generate()](./reference/inspect_ai.solver.html.md#generate) is called. By default token limits meter all tokens; use a limit with type `"output"` to meter only output tokens (which include reasoning tokens), or an arithmetic formula to meter a weighted mix (see below).

To limit the total number of tokens which can be used in a block of code:

``` python
@agent
def myagent(tokens: int = (1024*500)) -> Agent:
    async def execute(state: AgentState):

        with token_limit(tokens):
            # a LimitExceededError will be raised if the limit is exceeded
            ...
```

The limits can be stacked. Tokens used while a context manager is open count towards all open token limits.

``` python
@agent
def myagent() -> Solver:
    async def execute(state: AgentState):

        with token_limit(1024*500):
            ...
            with token_limit(1024*200):
                # Tokens used here count towards both active limits
                ...
```

To meter only output tokens (which include reasoning tokens) rather than all tokens, pass `type="output"`:

``` python
with token_limit(1024*500, type="output"):
    # a LimitExceededError will be raised if more than
    # 500K output tokens are used
    ...
```

`type` can also be an arithmetic formula over the variables `input` and `output` (supporting `+`, `-`, `*`, `/`, parentheses, and unary minus), to meter a weighted mix of token types:

``` python
with token_limit(1024*500, type="(input * 0.1) + output"):
    # meters 10% of input tokens plus all output tokens
    ...
```

In a formula, `input` is the true prompt size (including cached tokens) and `output` includes reasoning tokens; the computed value is floored to an integer. Limits of different types can be stacked—each limit meters its own basis.

> **IMPORTANT: Important**
>
> It’s important to note that [token_limit()](./reference/inspect_ai.util.html.md#token_limit) is for all tokens used *while the context manager is open*. If you want to limit the number of tokens that can be yielded from a single call to the model you should use the `max_tokens` generation option.

#### Suspending Token Limits

To run a block of code that should not count against any active token limits, use [suspend_token_limit()](./reference/inspect_ai.util.html.md#suspend_token_limit):

``` python
with token_limit(10_000):
    await generate()  # counts against the 10k budget
    with suspend_token_limit():
        # tokens used here are not metered against the 10k limit,
        # and any inner `token_limit()` is also suspended
        await expensive_summary()
    await generate()  # counts again
```

Unlike `with token_limit(None):`, which only suppresses the innermost limit’s check, [suspend_token_limit()](./reference/inspect_ai.util.html.md#suspend_token_limit) fully disables both recording and checking across all active token limits for the duration of the block.

### Turn Limit

A “turn” is a single model generation (one call to the model that produces an assistant message). Turn limits are distinct from message limits, which count *all* messages in the conversation (user, assistant, and tool messages). One turn often results in several messages (e.g. an assistant message plus the tool messages from its tool calls).

A turn is recorded once per top-level [generate()](./reference/inspect_ai.solver.html.md#generate) call (after retries and fallbacks have resolved, and including cache hits). Generations made via `model.compact()` do not count as turns. Turn limits are checked whenever a turn is recorded.

To limit the total number of turns (model generations) which can be used in a block of code:

``` python
@agent
def myagent(turns: int = 300) -> Agent:
    async def execute(state: AgentState):

        with turn_limit(turns):
            # a LimitExceededError will be raised if the limit is exceeded
            ...
```

Like token limits, turn limits can be stacked — a generation counts towards all open turn limits. To run generations that should not count against any active turn limit, use `suspend_turn_limit()`:

``` python
with turn_limit(10):
    await generate()  # counts against the 10 turn budget
    with suspend_turn_limit():
        # generations here are not metered against the 10 turn limit
        await auxiliary_generate()
    await generate()  # counts again
```

### Cost Limit

Cost is computed from token usage and model cost data (see [Model Cost](#model-cost)). Cost limits are checked whenever [generate()](./reference/inspect_ai.solver.html.md#generate) is called.

To limit the total cost within a block of code:

``` python
@agent
def myagent(budget: float = 2.00) -> Agent:
    async def execute(state: AgentState):

        with cost_limit(budget):
            # a LimitExceededError will be raised if the limit is exceeded
            ...
```

Cost limits work similarly to token limits, with stacking and tracking of costs used while the context manager is open.

> **IMPORTANT: Important**
>
> Using [cost_limit()](./reference/inspect_ai.util.html.md#cost_limit) requires model cost data to be configured via [set_model_cost()](./reference/inspect_ai.model.html.md#set_model_cost) or `--model-cost-config`. See [Model Cost](#model-cost) for details.

## Agent Limits

To run an agent with one or more limits, pass the limit object in the `limits` argument to a function like [handoff()](./reference/inspect_ai.agent.html.md#handoff), [as_tool()](./reference/inspect_ai.agent.html.md#as_tool), [as_solver()](./reference/inspect_ai.agent.html.md#as_solver) or [run()](./reference/inspect_ai.agent.html.md#run) (see [Using Agents](./agents.html.md#using-agents) for details on the various ways to run agents).

Here we limit an agent we are including as a solver to 500K tokens:

``` python
eval(
    task="research_bench", 
    solver=as_solver(web_surfer(), limits=[token_limit(1024*500)])
)
```

Here we limit an agent [handoff()](./reference/inspect_ai.agent.html.md#handoff) to 500K tokens:

``` python
eval(
    task="research_bench", 
    solver=[
        use_tools(
            addition(),
            handoff(web_surfer(), limits=[token_limit(1024*500)]),
        ),
        generate()
    ]
)
```

### Limit Exceeded

Note that when limits are exceeded during an agent’s execution, the way this is handled differs depending on how the agent was executed:

- For agents used via [as_solver()](./reference/inspect_ai.agent.html.md#as_solver), if a limit is exceeded then the sample will terminate (this is exactly how sample-level limits work).

- For agents that are [run()](./reference/inspect_ai.agent.html.md#run) directly with limits, their limit exceptions will be caught and returned in a tuple. Limits other than the ones passed to [run()](./reference/inspect_ai.agent.html.md#run) will propagate up the stack.

  ``` python
  from inspect_ai.agent import run

  state, limit_error = await run(
      agent=web_surfer(), 
      input="What were the 3 most popular movies of 2020?",
      limits=[token_limit(1024*500)])
  )
  if limit_error:
      ...
  ```

- For tool based agents ([handoff()](./reference/inspect_ai.agent.html.md#handoff) and [as_tool()](./reference/inspect_ai.agent.html.md#as_tool)), if a limit is exceeded then a message to that effect is returned to the model but the *sample continues running*.