Tool Basics

Overview

Many models now have the ability to interact with client-side Python functions in order to expand their capabilities. This enables you to equip models with your own set of custom tools so they can perform a wider variety of tasks.

Inspect natively supports registering Python functions as tools and providing these tools to models that support them. Inspect also includes several standard tools for code execution, text editing, computer use, web search, and web browsing.

Tools and Agents

One application of tools is to run them within an agent scaffold that pursues an objective over multiple interactions with a model. The scaffold uses the model to help make decisions about which tools to use and when, and orchestrates calls to the model to use the tools. This is covered in more depth in the Agents section.

Standard Tools

Inspect has several standard tools built-in, including:

  • Bash and Python for executing arbitrary shell and Python code.

  • Bash Session for creating a stateful bash shell that retains its state across calls from the model.

  • Text Editor which enables viewing, creating and editing text files.

  • Web Browser, which provides the model with a headless Chromium web browser that supports navigation, history, and mouse/keyboard interactions.

  • Computer, which provides the model with a desktop computer (viewed through screenshots) that supports mouse and keyboard interaction.

  • Web Search, which uses the Google Search API to execute and summarise web searches.

  • Think, which provides models the ability to include an additional thinking step as part of getting to its final answer.

If you are only interested in using the standard tools, check out their respective documentation links above. To learn more about creating your own tools read on below.

Defining Tools

Here we’ll describe the basics of defining and using a custom tool. See the Custom Tools article for details on more advanced custom tool features including sandboxing, error handling, and dynamic tool definitions.

Here’s a simple tool that adds two numbers. The @tool decorator is used to register it with the system:

from inspect_ai.tool import tool

@tool
def add():
    async def execute(x: int, y: int):
        """
        Add two numbers.

        Args:
            x: First number to add.
            y: Second number to add.

        Returns:
            The sum of the two numbers.
        """
        return x + y

    return execute

Annotations

Note that we provide type annotations for both arguments:

async def execute(x: int, y: int)

Further, we provide descriptions for each parameter in the documention comment:

Args:
    x: First number to add.
    y: Second number to add.

Type annotations and descriptions are required for tool declarations so that the model can be informed which types to pass back to the tool function and what the purpose of each parameter is.

Note that you while you are required to provide default descriptions for tools and their parameters within doc comments, you can also make these dynamically customisable by users of your tool (see the section on Tool Descriptions for details on how to do this).

Using Tools

We can use the addition() tool in an evaluation by passing it to the use_tools() Solver:

from inspect_ai import Task, task
from inspect_ai.dataset ipmort Sample
from inspect_ai.solver import generate, use_tools
from inspect_ai.scorer import match

@task
def addition_problem():
    return Task(
        dataset=[Sample(input="What is 1 + 1?", target=["2"])],
        solver=[
            use_tools(add()), 
            generate()
        ],
        scorer=match(numeric=True),
    )

Note that this tool doesn’t make network requests or do heavy computation, so is fine to run as inline Python code. If your tool does do more elaborate things, you’ll want to make sure it plays well with Inspect’s concurrency scheme. For network requests, this amounts to using async HTTP calls with httpx. For heavier computation, tools should use subprocesses as described in the next section.

Note that when using tools with models, the models do not call the Python function directly. Rather, the model generates a structured request which includes function parameters, and then Inspect calls the function and returns the result to the model.

Learning More

  • Custom Tools provides details on more advanced custom tool features including sandboxing, error handling, and dynamic tool definitions.

  • Standard Tools describes Inspect’s built-in tools for code execution, text editing computer use, web search, and web browsing.