Fallbacks

Overview

Claude 5 models include safety classifiers that can decline a request. A decline need not be an error: the API returns a normal response with a refusal stop reason (surfaced by Inspect as stop_reason="content_filter"), and the same request can usually still be served by another Claude model.

The fallback_models generation option enables this automatically. When the requested model’s classifiers decline, the request is retried on one or more fallback models (tried in order) within the same request, using Anthropic’s server-side fallback. Inspect records which model served each generation.

NoteAPI Compatibility

Fallbacks apply only to the first-party Anthropic Claude API (they are ignored, with a warning, on Bedrock, Vertex, and Azure and with batch mode) and only to Claude 5 and later requested models (except Claude Haiku 5.5, which has no permitted targets, so fallback_models is ignored with a warning). Each fallback target must be one of the requested model’s permitted targets, published as allowed_fallback_models on the model’s Models API entry (currently claude-opus-4-8 is the only permitted target for claude-fable-5).

Basic Usage

Specify one or more fallback models (up to three, tried in order) with the --fallback-models CLI option:

inspect eval ctf.py --model anthropic/claude-fable-5 \
   --fallback-models claude-opus-4-8

Or from Python (like all generate config, fallback_models can be specified at the eval, task, or model level):

from inspect_ai import eval

eval(
    "ctf.py",
    model="anthropic/claude-fable-5",
    fallback_models=["claude-opus-4-8"],
)

Only a safety classifier decline triggers the fallback. If every model in the chain declines, the final refusal is returned (stop_reason="content_filter").

Refusals

Without fallbacks configured (or when every fallback also declines), the refusal is surfaced on the model output: stop_reason is "content_filter" and stop_details carries the structured refusal detail:

Field	Description
type	"refusal" for classifier declines.
category	Policy area that triggered the classifier (e.g. "cyber", "bio", "reasoning_extraction"). May be None when the refusal maps to no named category.
explanation	Human readable description (display it, don’t parse it).
categories	All triggering categories (list of StopCategory).

When a fallback serves a request, the API does not report the declining attempt’s refusal category (the declined attempt is unbilled and only its token counts appear in the response diagnostics). Refusal categories are available only on responses that actually ended in a refusal.

Failing on Refusals

By default a refusal is just another model output: agents like react() retry it if retry_refusals is set and otherwise stop, and the sample proceeds to scoring (typically scoring zero, which is indistinguishable from a genuine failure of the task). If you would rather be told loudly that a refusal occurred, set the fail_on_refusal generation option. When enabled, generate() raises a ModelRefusalError after a generation ends with stop_reason="content_filter", which fails the sample with an error whose message starts with Model refusal and includes the model, role and refusal category:

inspect eval ctf.py --model anthropic/claude-fable-5 --fail-on-refusal

Like other generation config, fail_on_refusal can be set eval-wide (as above, or eval(..., fail_on_refusal=True)), on a Task (config=GenerateConfig(fail_on_refusal=True)), on a model (--model-spec or get_model(..., config=...)), or on an individual generate() call. Eval-wide and task settings also apply to any other model used by the eval, including model roles such as graders and ad hoc get_model() calls in solvers or scorers, but a model’s own setting wins, so you can fail on refusals from the agent while letting a grader refuse normally:

inspect eval ctf.py --fail-on-refusal \
  --model-role grader="{model: openai/gpt-5, fail_on_refusal: false}"

Some things to be aware of:

• With the default fail_on_error, the first sample error fails the whole eval, so --fail-on-refusal on its own stops the eval at the first refusal. To record failed samples and keep running, combine it with --no-fail-on-error (or a --fail-on-error threshold, or --continue-on-fail). See Handling Errors for details.
• Refusal retries still apply: retry_refusals=N on react(), deepagent() or the agent bridge means the sample fails only after N+1 consecutive refusals in one step (deepagent() defaults to 3 retries, react() to none).
• retry_on_error and score_on_error treat a refusal like any other sample error, and eval_set() retries a task whose log status is error as usual.
• The refusal counter and --log-refusals still run, and ModelRefusalError (exported from inspect_ai.model) carries the refused output, so the category is available to code that catches it.
• Because the CLI flag is unset by default, --fail-on-refusal never disables an option enabled by a task or model config; pass --no-fail-on-refusal to do that.
• For scaffolds bridged in-process with agent_bridge(), the error surfaces as an exception from the scaffold’s own SDK call, so a scaffold that catches broad exceptions around its API calls will swallow it. Scaffolds running in a sandbox via sandbox_agent_bridge() fail the sample regardless of how they handle the error response they receive.
• A refusal during an MCP sampling request (a generation an MCP server asks Inspect to perform on its behalf) is returned to the server as an MCP error rather than failing the sample, the same as an exceeded limit there today.

Fallback Logging

When a fallback serves a generation, Inspect records it on the model output, in the message content, and in a per-sample rollup.

On the model output, ModelOutput.model reports the model that actually produced the response, and ModelOutput.fallback records the handoff as a ModelFallback:

Field	Description
model	Model that was originally requested.
fallback_model	Model that served the request.
count	Number of requests served by the fallback. Usually 1 on a single output. An Anthropic turn continued after pause_turn sends several requests, so its output counts each one that fell back. The sample rollup sums the counts.
metadata	Provider diagnostics. For Anthropic, the handoffs chain and the per-attempt usage.iterations billing record.

A turn that sticky routing sends straight to the fallback model (no model declined it) is recorded the same way, with an empty handoffs chain.

The assistant message also carries a content marker at the point of the handoff, which is what allows Inspect to replay fallen-back conversations on subsequent turns.

At the sample level, EvalSample.model_fallbacks (and sample summaries) aggregate the fallbacks that occurred during the sample (across solvers, subagents, and scorers) as a list of ModelFallback entries keyed by requested and serving model. The rollup is also available in dataframes: samples_df() includes a fallbacks column with the total count, and the full detail is available via a custom column:

from inspect_ai.analysis import SampleColumn, SampleSummary, samples_df

df = samples_df(
    "logs",
    columns=SampleSummary + [
        SampleColumn("model_fallbacks", path="model_fallbacks")
    ],
)
df[df.fallbacks > 0]

Costs

Cost estimation (including the cost_limit option) prices each attempt at the rates of the model that ran it, as Anthropic bills it. An attempt that declined partway through its response is billed, and so is included. An attempt that declined before any output is billed only for some refusal categories (bio, frontier_llm and reasoning_extraction), which a fallback response does not report, so it is left out. For those categories the estimated cost, and the spend a cost_limit sees, is lower than the billed amount. The per-attempt billing record is in ModelFallback.metadata["iterations"] on each ModelEvent’s output.

Fallback models need cost data too (see Model Cost). If a fallback model has none, the request is priced at the requested model’s rates and a warning is shown.

Viewer

The viewer surfaces fallbacks in several places:

• The samples list includes a Fallbacks column (shown when any sample has fallbacks) with the total count per sample. The has_fallbacks and fallbacks variables are available for filtering (e.g. has_fallbacks or fallbacks > 2).

• The sample header annotates the model, e.g. anthropic/claude-fable-5 (fallback → claude-opus-4-8).

• In the transcript, fallen-back model calls carry a fallback → <model> badge in their title bar, and a marker appears in the assistant message content at the point of the handoff.

The task display shown while an eval is running (and the inspect acp session view) annotate the model the same way.

Fallback Scanning

Scanners from Inspect Scout can locate fallbacks in a set of logs. The scanner below emits a result for each generation served by a fallback model, with the handoff as the value, an explanation noting the refusal followed by the served message content, and a reference to the originating event:

from inspect_ai.event import ModelEvent
from inspect_scout import Reference, Result, Scanner, scanner


@scanner(events=["model"])
def model_fallbacks() -> Scanner[ModelEvent]:
    """Find generations served by a fallback model."""

    async def scan(event: ModelEvent) -> Result:
        fallback = event.output.fallback
        if fallback is None:
            return Result(value=None)
        return Result(
            value=f"{fallback.model} → {fallback.fallback_model}",
            explanation=f"{fallback.model} refused this request: "
            + event.output.message.text,
            references=[Reference(type="event", id=event.uuid or "")],
        )

    return scan