inspect_ai.event

Core Events

ModelEvent

Call to a language model.

class ModelEvent(BaseEvent)

Attributes

event Literal['model']: Event type.
model str: Model name.
role str | None: Model role.
input list[ChatMessage]: Model input (list of messages).
tools list[ToolInfo]: Tools available to the model.
tool_choice ToolChoice: Directive to the model which tools to prefer.
config GenerateConfig: Generate config used for call to model.
output ModelOutput: Output from model.
retries int | None: Retries for the model API request.
error str | None: Error which occurred during model call.
cache Literal['read', 'write'] | None: Was this a cache read or write.
call ModelCall | None: Raw call made to model API.
completed datetime | None: Time that model call completed (see timestamp for started)
working_time float | None: working time for model call that succeeded (i.e. was not retried).

ToolEvent

Call to a tool.

Source

class ToolEvent(BaseEvent)

Attributes

event Literal['tool']: Event type.
type Literal['function']: Type of tool call (currently only ‘function’)
id str: Unique identifier for tool call.
function str: Function called.
arguments dict[str, JsonValue]: Arguments to function.
view ToolCallContent | None: Custom view of tool call input.
result ToolResult: Function return value.
truncated tuple[int, int] | None: Bytes truncated (from,to) if truncation occurred
error ToolCallError | None: Error that occurred during tool call.
completed datetime | None: Time that tool call completed (see timestamp for started)
working_time float | None: Working time for tool call (i.e. time not spent waiting on semaphores).
agent str | None: Name of agent if the tool call was an agent handoff.
failed bool | None: Did the tool call fail with a hard error?.
message_id str | None: Id of ChatMessageTool associated with this event.
cancelled bool: Was the task cancelled?

ApprovalEvent

Tool approval.

Source

class ApprovalEvent(BaseEvent)

Attributes

event Literal['approval']: Event type
message str: Message generated by model along with tool call.
call ToolCall: Tool call being approved.
view ToolCallView | None: View presented for approval.
approver str: Aprover name.
decision Literal['approve', 'modify', 'reject', 'escalate', 'terminate']: Decision of approver.
modified ToolCall | None: Modified tool call for decision ‘modify’.
explanation str | None: Explanation for decision.

SandboxEvent

Sandbox execution or I/O

Source

class SandboxEvent(BaseEvent)

Attributes

event Literal['sandbox']: Event type
action Literal['exec', 'read_file', 'write_file']: Sandbox action
cmd str | None: Command (for exec)
options dict[str, JsonValue] | None: Options (for exec)
file str | None: File (for read_file and write_file)
input str | None: Input (for cmd and write_file). Truncated to 100 lines.
result int | None: Result (for exec)
output str | None: Output (for exec and read_file). Truncated to 100 lines.
completed datetime | None: Time that sandbox action completed (see timestamp for started)

InfoEvent

Event with custom info/data.

Source

class InfoEvent(BaseEvent)

Attributes

event Literal['info']: Event type.
source str | None: Optional source for info event.
data JsonValue: Data provided with event.

LoggerEvent

Log message recorded with Python logger.

Source

class LoggerEvent(BaseEvent)

Attributes

event Literal['logger']: Event type.
message LoggingMessage: Logging message

ErrorEvent

Event with sample error.

Source

class ErrorEvent(BaseEvent)

Attributes

event Literal['error']: Event type.
error EvalError: Sample error

SpanBeginEvent

Mark the beginning of a transcript span.

Source

class SpanBeginEvent(BaseEvent)

Attributes

event Literal['span_begin']: Event type.
id str: Unique identifier for span.
parent_id str | None: Identifier for parent span.
type str | None: Optional ‘type’ field for span.
name str: Span name.

SpanEndEvent

Mark the end of a transcript span.

Source

class SpanEndEvent(BaseEvent)

Attributes

event Literal['span_end']: Event type.
id str: Unique identifier for span.

Event Tree

event_tree

Build a tree representation of a sequence of events.

Organize events heirarchially into event spans.

Source

def event_tree(events: Sequence[Event]) -> EventTree

events Sequence[Event]: Sequence of Event.

event_sequence

Flatten a span forest back into a properly ordered seqeunce.

Source

def event_sequence(tree: EventTree) -> Iterable[Event]

tree EventTree: Event tree

EventTree

Tree of events (has invividual events and event spans).

Source

EventTree: TypeAlias = list[EventNode]

EventNode

Node in an event tree.

Source

EventNode: TypeAlias = Union["SpanNode", Event]

SpanNode

Event tree node representing a span of events.

Source

@dataclass
class SpanNode

Attributes

id str: Span id.
parent_id str | None: Parent span id.
type str | None: Optional ‘type’ field for span.
name str: Span name.
begin SpanBeginEvent: Span begin event.
end SpanEndEvent | None: Span end event (if any).
children list[EventNode]: Children in the span.

Eval Events

SampleInitEvent

Beginning of processing a Sample.

Source

class SampleInitEvent(BaseEvent)

Attributes

event Literal['sample_init']: Event type.
sample Sample: Sample.
state JsonValue: Initial state.

SampleLimitEvent

The sample was unable to finish processing due to a limit

Source

class SampleLimitEvent(BaseEvent)

Attributes

event Literal['sample_limit']: Event type.
type Literal['message', 'time', 'working', 'token', 'operator', 'custom']: Type of limit that halted processing
message str: A message associated with this limit
limit float | None: The limit value (if any)

StateEvent

Change to the current TaskState

Source

class StateEvent(BaseEvent)

Attributes

event Literal['state']: Event type.
changes list[JsonChange]: List of changes to the TaskState

StoreEvent

Change to data within the current Store.

Source

class StoreEvent(BaseEvent)

Attributes

event Literal['store']: Event type.
changes list[JsonChange]: List of changes to the Store.

InputEvent

Input screen interaction.

Source

class InputEvent(BaseEvent)

Attributes

event Literal['input']: Event type.
input str: Input interaction (plain text).
input_ansi str: Input interaction (ANSI).

ScoreEvent

Event with score.

Can be the final score for a Sample, or can be an intermediate score resulting from a call to score.

Source

class ScoreEvent(BaseEvent)

Attributes

event Literal['score']: Event type.
score Score: Score value.
target str | list[str] | None: “Sample target.
intermediate bool: Was this an intermediate scoring?

ScoreEditEvent

Event recorded when a score is edited.

Source

class ScoreEditEvent(BaseEvent)

Attributes

event Literal['score_edit']: Event type.
score_name str: Name of the score being edited.
edit ScoreEdit: The edit being applied to the score.

Types

LoggingLevel

Logging level.

Source

LoggingLevel = Literal[
    "debug", "trace", "http", "sandbox", "info", "warning", "error", "critical"
]

LoggingMessage

Message written to Python log.

Source

class LoggingMessage(BaseModel)

Attributes

name str | None: Logger name (e.g. ‘httpx’)
level LoggingLevel: Logging level.
message str: Log message.
created float: Message created time.
filename str: Logged from filename.
module str: Logged from module.
lineno int: Logged from line number.