# Inspect

> Open-source framework for large language model evaluations

## Basics

- [Welcome](https://inspect.aisi.org.uk/index.html.md): Welcome and overview of Inspect AI.
- [Tutorial](https://inspect.aisi.org.uk/tutorial.html.md): Step-by-step walkthrough of Inspect, from a first eval through agents, analysis, and the broader feature set.
- [Options](https://inspect.aisi.org.uk/options.html.md): Covers the various options available for evaluations as well as how to manage model credentials.
- [Log Viewer](https://inspect.aisi.org.uk/log-viewer.html.md): How to use Inspect View to develop and debug evaluations, including how to provide additional log metadata and integrate it with Python logging.
- [VS Code](https://inspect.aisi.org.uk/vscode.html.md): Using the Inspect VS Code Extension to run, tune, debug, and visualise evaluations.

## Components

- [Tasks](https://inspect.aisi.org.uk/tasks.html.md): Tasks bring together datasets, solvers, and scorers to define an evaluation. Strategies for creating flexible, re-usable tasks and for configuring and overriding them at runtime.
- [Datasets](https://inspect.aisi.org.uk/datasets.html.md): Datasets provide samples to evaluation tasks. How to adapt various data sources for use with Inspect, including multi-modal data.
- [Solvers](https://inspect.aisi.org.uk/solvers.html.md): Solvers encompass prompt engineering and other elicitation strategies. Using built-in solvers and creating your own.
- [Scorers](https://inspect.aisi.org.uk/scorers.html.md): Scorers evaluate the work of solvers and aggregate scores into metrics. Overview of the built-in scorers and pointers to custom scorers, metrics, and the scoring workflow.

## Models

- [Using Models](https://inspect.aisi.org.uk/models.html.md): Models provide a uniform API for evaluating a variety of large language models and using models within evaluations.
- [Providers](https://inspect.aisi.org.uk/providers.html.md): Usage details and available options for the various supported model providers.
- [Caching](https://inspect.aisi.org.uk/caching.html.md): Caching enables you to cache model output to reduce API calls, saving both time and expense.
- [Concurrency](https://inspect.aisi.org.uk/models-concurrency.html.md): Tuning model API connection limits, adaptive concurrency, and rate-limit handling.
- [Compaction](https://inspect.aisi.org.uk/compaction.html.md): Compacting message histories for long-running agents that exceed the context window.
- [Fallbacks](https://inspect.aisi.org.uk/fallbacks.html.md): Retrying classifier-refused requests on a fallback model, and how fallbacks are recorded in logs, dataframes, and the viewer.
- [Multimodal](https://inspect.aisi.org.uk/multimodal.html.md): How to use images, audio, and video in evaluations.
- [Reasoning](https://inspect.aisi.org.uk/reasoning.html.md): Additional options and data available for reasoning models.
- [Structured Output](https://inspect.aisi.org.uk/structured.html.md): How to constrain model output to a particular JSON schema.
- [Batch Mode](https://inspect.aisi.org.uk/models-batch.html.md): Using batch processing APIs for model inference.

## Scoring

- [Standard Scorers](https://inspect.aisi.org.uk/standard-scorers.html.md): The built-in scorers that ship with Inspect (text matching, multiple choice, model grading, math, and perplexity), the metrics they report, and where to go next.
- [Custom Scorers](https://inspect.aisi.org.uk/custom-scorers.html.md): Write your own scorers using the Score, Value, and Target types, call models from scorers, and access the sandbox.
- [Model Grading](https://inspect.aisi.org.uk/model-graded.html.md): Use another model to grade outputs with model_graded_qa() and model_graded_fact(). Customise templates, instructions, grader models, multi-model voting, and chat history.
- [Scoring Metrics](https://inspect.aisi.org.uk/metrics.html.md): Built-in scoring metrics, metric grouping, clustered standard errors, custom metrics, and reducing epochs.
- [Multiple Scorers](https://inspect.aisi.org.uk/multiple-scorers.html.md): Use several scorers together, emit multiple scores from one scorer, attach complex metrics, reduce multiple scores, and access the sandbox.
- [Scoring Workflow](https://inspect.aisi.org.uk/scoring-workflow.html.md): Defer scoring with --no-score, re-score logs with the inspect score command and the score() function, control append vs. overwrite, and edit scores.
- [Perplexity](https://inspect.aisi.org.uk/perplexity.html.md): Perplexity-based scorers (perplexity() and target_perplexity()) for evaluating how well a model predicts text using prompt log probabilities.

## Agents

- [Using Agents](https://inspect.aisi.org.uk/agents.html.md): Agents combine planning, memory, and tool usage to pursue complex, longer horizon tasks.
- [ReAct Agent](https://inspect.aisi.org.uk/react-agent.html.md): Using and customizing the built-in ReAct agent.
- [Deep Agent](https://inspect.aisi.org.uk/deepagent.html.md): Using and customizing the built-in deep agent with subagent delegation, memory, and planning.
- [Checkpointing](https://inspect.aisi.org.uk/checkpointing.html.md)
- [Intervention](https://inspect.aisi.org.uk/intervention.html.md)
- [Multi Agent](https://inspect.aisi.org.uk/multi-agent.html.md): Composing agents together in multi-agent architectures.
- [Custom Agents](https://inspect.aisi.org.uk/agent-custom.html.md): APIs for creating custom agents.
- [Agent Bridge](https://inspect.aisi.org.uk/agent-bridge.html.md): Integrating agents from third-party frameworks like OpenAI Agents SDK, LangChain, and Pydantic AI.
- [Human Agent](https://inspect.aisi.org.uk/human-agent.html.md): The human_cli() agent for human baselining on computing tasks.

## Tools

- [Tool Basics](https://inspect.aisi.org.uk/tools.html.md): Tools extend the capabilities of models by registering Python functions for them to call. How to create custom tools.
- [Standard Tools](https://inspect.aisi.org.uk/tools-standard.html.md): Inspect's built-in tools for code execution, text editing, computer use, web search, and web browsing.
- [MCP Tools](https://inspect.aisi.org.uk/tools-mcp.html.md): How to integrate tools from Model Context Protocol providers.
- [Custom Tools](https://inspect.aisi.org.uk/tools-custom.html.md): Advanced custom tool features including sandboxing, error handling, and dynamic tool definitions.
- [Sandboxing](https://inspect.aisi.org.uk/sandboxing.html.md): Isolating code generated by models and setting up complex computing environments for tasks.
- [Tool Approval](https://inspect.aisi.org.uk/approval.html.md): Fine-grained policies for approving tool calls made by models.

## Running

- [Eval Sets](https://inspect.aisi.org.uk/eval-sets.html.md): Describing, running, and analysing larger sets of evaluation tasks.
- [Parallelism](https://inspect.aisi.org.uk/parallelism.html.md): Running multiple tasks and models in parallel, sandbox concurrency, and writing parallel custom code.
- [Handling Errors](https://inspect.aisi.org.uk/handling-errors.html.md): Techniques for dealing with runtime errors and recovering from crashes during evaluation.
- [Setting Limits](https://inspect.aisi.org.uk/setting-limits.html.md): Setting time, message, token, and cost limits on evaluation tasks, samples, and agent execution.
- [Control Channel](https://inspect.aisi.org.uk/control-channel.html.md): Launch evals from scripts and agents (inspect eval --json launch handoff, and --detach to run an eval in the background, detached from the terminal), then observe and direct them from another process with the inspect ctl commands - live task and sample status, error and retry detail, transcript events, task and sample cancellation, launch-config retuning, log flushing, and keep-alive.
- [Early Stopping](https://inspect.aisi.org.uk/early-stopping.html.md): The early stopping API for ending tasks early based on previously scored samples.
- [Task Sources](https://inspect.aisi.org.uk/task-source.html.md): Generate an eval's tasks dynamically from code — a seed plus result-driven follow-ups — while the eval runs.
- [Tracing](https://inspect.aisi.org.uk/tracing.html.md): Advanced execution tracing tools for diagnosing runtime issues.

## Analysis

- [Log Files](https://inspect.aisi.org.uk/eval-logs.html.md): Getting the most out of evaluation logs for developing, debugging, and analyzing evaluations.
- [Dataframes](https://inspect.aisi.org.uk/dataframe.html.md): APIs for extracting dataframes of evals, samples, messages, and events from log files.
- [Scanners](https://inspect.aisi.org.uk/scanners.html.md): Scanners review transcripts to find issues like misconfigured environments, refusals, and evaluation awareness.
- [Inspect Viz](https://inspect.aisi.org.uk/inspect-viz.html.md): Inspect Viz is a companion package for creating high quality, interactive visualisations from Inspect evaluation logs.
- [Task Views](https://inspect.aisi.org.uk/task-views.html.md): Use ViewerConfig to customize how a task's samples, scores, and scanner results render in the Inspect log viewer.

## Extensions

- [Model APIs](https://inspect.aisi.org.uk/extensions-model-api.html.md): Extending Inspect with new Model APIs (model hosting services, local inference engines, etc.).
- [Components](https://inspect.aisi.org.uk/extensions-components.html.md): Distributing Inspect components (tasks, solvers, scorers, and tools) within Python packages so they can be referenced by name.
- [Sandboxes](https://inspect.aisi.org.uk/extensions-sandboxes.html.md): Extending Inspect with new Sandboxes (local or cloud container runtimes for tool execution).
- [Approvers](https://inspect.aisi.org.uk/extensions-approvers.html.md): Extending Inspect with new Approvers (approve, modify, or reject tool calls).
- [Hooks](https://inspect.aisi.org.uk/extensions-hooks.html.md): Extending Inspect with Hooks (run code during Inspect lifecycle events for logging and monitoring frameworks).
- [Filesystems](https://inspect.aisi.org.uk/extensions-filesystems.html.md): Extending Inspect with new Filesystems (storage for datasets, prompts, and evaluation logs).

## Reference

- [inspect_ai.agent](https://inspect.aisi.org.uk/reference/inspect_ai.agent.html.md): Agent scaffolds and protocol.
- [inspect_ai.analysis](https://inspect.aisi.org.uk/reference/inspect_ai.analysis.html.md): Data frames for analysis.
- [inspect_ai.approval](https://inspect.aisi.org.uk/reference/inspect_ai.approval.html.md): Approvers and approval policies.
- [inspect_ai.dataset](https://inspect.aisi.org.uk/reference/inspect_ai.dataset.html.md): Reading samples from datasets.
- [inspect_ai.event](https://inspect.aisi.org.uk/reference/inspect_ai.event.html.md): Transcript events.
- [inspect_ai.hooks](https://inspect.aisi.org.uk/reference/inspect_ai.hooks.html.md): Hook into Inspect lifecycle events.
- [inspect_ai.log](https://inspect.aisi.org.uk/reference/inspect_ai.log.html.md): List, read, write, and analyse logs.
- [inspect_ai.model](https://inspect.aisi.org.uk/reference/inspect_ai.model.html.md): Model interface and providers.
- [inspect_ai](https://inspect.aisi.org.uk/reference/inspect_ai.html.md): Tasks, evaluation, and scoring.
- [inspect_ai.scorer](https://inspect.aisi.org.uk/reference/inspect_ai.scorer.html.md): Task scoring and metrics.
- [inspect_ai.solver](https://inspect.aisi.org.uk/reference/inspect_ai.solver.html.md): Prompting and elicitation.
- [inspect_ai.tool](https://inspect.aisi.org.uk/reference/inspect_ai.tool.html.md): Built-in and custom tool functions.
- [inspect_ai.util](https://inspect.aisi.org.uk/reference/inspect_ai.util.html.md): Miscellaneous utility functions.
- [inspect_ai.viewer](https://inspect.aisi.org.uk/reference/inspect_ai.viewer.html.md): Customize Inspect view behavior.