Filesystems

Filesystems with fsspec

Datasets, prompt templates, and evaluation logs can be stored using either the local filesystem or a remote filesystem. Inspect uses the fsspec package to read and write files, which provides support for a wide variety of filesystems, including:

• Amazon S3
• Google Cloud Storage
• Azure Blob Storage
• Azure Data Lake Storage
• DVC

Support for Amazon S3 is built in to Inspect via the s3fs package. Other filesystems may require installation of additional packages. See the list of built in filesystems and other known implementations for all supported storage back ends.

See Custom Filesystems below for details on implementing your own fsspec compatible filesystem as a storage back-end.

Filesystem Functions

The following Inspect API functions use fsspec:

• resource() for reading prompt templates and other supporting files.

• csv_dataset() and json_dataset() for reading datasets (note that files referenced within samples can also use fsspec filesystem references).

• list_eval_logs() , read_eval_log(), write_eval_log(), and retryable_eval_logs().

For example, to use S3 you would prefix your paths with s3://:

# read a prompt template from s3
prompt_template("s3://inspect-prompts/ctf.txt")

# read a dataset from S3
csv_dataset("s3://inspect-datasets/ctf-12.csv")

# read eval logs from S3
list_eval_logs("s3://my-s3-inspect-log-bucket")

Custom Filesystems

See the fsspec developer documentation for details on implementing a custom filesystem. Note that if your implementation is only for use with Inspect, you need to implement only the subset of the fsspec API used by Inspect. The properties and methods used by Inspect include:

• sep
• open()
• makedirs()
• info()
• created()
• exists()
• ls()
• walk()
• unstrip_protocol()
• invalidate_cache()

As with Model APIs and Sandbox Environments, fsspec filesystems should be registered using a setuptools entry point. For example, if your package is named evaltools and you have implemented a myfs:// filesystem using the MyFs class exported from the root of the package, you would register it like this in pyproject.toml:

Setuptools

[project.entry-points."fsspec.specs"]
myfs = "evaltools:MyFs"

uv

[project.entry-points."fsspec.specs"]
myfs = "evaltools:MyFs"

Poetry

[tool.poetry.plugins."fsspec.specs"]
myfs = "evaltools:MyFs"

Once this package is installed, you’ll be able to use myfs:// with Inspect without any further registration.