Runs resource

Access these methods through client.runs.

`client.runs.start`

Start a run from an uploaded agent name or complete contract, in the project whose handle is project.

Signature

async def start(
    agent: str | AgentContractRequest,
    benchmark: str,
    *,
    model: str | None = None,
    concurrency: int = 5,
    priority: int | None = None,
    task_ids: Sequence[str] | None = None,
    slice_str: str | None = None,
    dataset: str | None = None,
    dataset_version: str | None = None,
    label: str | None = None,
    lambda_function: str | None = None,
    provider: str | None = None,
    agent_kwargs: Mapping[str, str] | None = None,
    secrets: Mapping[str, str] | None = None,
    service_headers: Mapping[str, str] | None = None,
    ignore_custom_services: bool = False,
    project: str | None = None,
) -> StartBenchmarkResponse

Parameters

required · str | AgentContractRequest

required · str

optional · str | None · default:

optional · int · default:

optional · int | None · default:

optional · Sequence[str] | None · default:

optional · str | None · default:

optional · Mapping[str, str] | None · default:

optional · bool · default:

optional · str | None · default:

Returns StartBenchmarkResponse For examples, see the Python SDK guide.

`client.runs.fetch`

Fetch the latest state of a run.

Signature

async def fetch(run_id: UUID) -> FetchBenchmarkResponse

Parameters

required · UUID

Returns FetchBenchmarkResponse For examples, see the Python SDK guide. Discover valid run-filter values from this organization’s run history.

Signature

async def filter_options() -> FilterOptionsResponse

Returns FilterOptionsResponse For examples, see the Python SDK guide.

`client.runs.list`

List runs using typed filters and pagination.

Signature

async def list(request: FetchBenchmarksRequest | None = None) -> FetchBenchmarksResponse

Parameters

optional · FetchBenchmarksRequest | None · default:

Returns FetchBenchmarksResponse For examples, see the Python SDK guide.

`client.runs.iter`

Iterate matching runs using cursor pagination, starting at the supplied cursor. Use list() for explicit offset pagination. Filters and page size are preserved.

Signature

async def iter(
    request: FetchBenchmarksRequest | None = None,
) -> AsyncIterator[BenchmarkTableRow]

Parameters

optional · FetchBenchmarksRequest | None · default:

Returns AsyncIterator[BenchmarkTableRow] For examples, see the Python SDK guide.

`client.runs.stream`

Yield the run’s state as it changes until the run completes or the server disconnects. Raises ValkyrieStreamError if the server sends an error event; use events() to see how the stream ended.

Signature

async def stream(run_id: UUID) -> AsyncIterator[FetchBenchmarkResponse]

Parameters

required · UUID

Returns AsyncIterator[FetchBenchmarkResponse] For examples, see the Python SDK guide.

`client.runs.events`

Yield each update, then the complete, error or disconnect event that ends the stream. If the body ends without one of those, iteration stops after the last update.

Signature

async def events(run_id: UUID) -> AsyncIterator[RunStreamEvent]

Parameters

required · UUID

Returns AsyncIterator[RunStreamEvent] For examples, see the Python SDK guide.

`client.runs.results`

Fetch final results or upload them and return S3 links.

Overload 1

async def results(
    run_id: UUID,
    *,
    task_ids: Sequence[str] | None = None,
    upload_to_s3: Literal[False] = False,
) -> FinalViewResponse

Overload 2

async def results(
    run_id: UUID,
    *,
    task_ids: Sequence[str] | None = None,
    upload_to_s3: Literal[True],
) -> S3UploadResultsResponse

Overload 3

async def results(
    run_id: UUID,
    *,
    task_ids: Sequence[str] | None = None,
    upload_to_s3: bool,
) -> RetrieveResultsResponse

Parameters

required · UUID

optional · Sequence[str] | None · default:

optional · bool · default:

Returns The selected overload determines the return type.

Overload	Return type
1	`FinalViewResponse`
2	`S3UploadResultsResponse`
3	RetrieveResultsResponse

For examples, see the Python SDK guide.

`client.runs.preview`

Archive the current S3 results, replace them with a fresh snapshot, and run the optional callback.

Signature

async def preview(
    run_id: UUID,
    *,
    task_ids: Sequence[str] | None = None,
) -> S3UploadResultsResponse

Parameters

required · UUID

optional · Sequence[str] | None · default:

Returns S3UploadResultsResponse For examples, see the Python SDK guide.

`client.runs.metadata`

Fetch the stored launch metadata for a run.

Signature

async def metadata(run_id: UUID) -> FetchBenchmarkMetadataResponse

Parameters

required · UUID

Returns FetchBenchmarkMetadataResponse For examples, see the Python SDK guide.

`client.runs.results_exist`

Check whether the canonical result file already exists in S3.

Signature

async def results_exist(run_id: UUID) -> ResultsExistResponse

Parameters

required · UUID

Returns ResultsExistResponse For examples, see the Python SDK guide.

`client.runs.analyze`

Yield typed progress events while analyzing a finished run, ending with done. A cached result arrives as a single done event. Raises ValkyrieStreamError when the analyzer reports an error.

Signature

async def analyze(
    run_id: UUID,
    *,
    no_cache: bool = False,
    lambda_function: str | None = None,
) -> AsyncIterator[AnalyzeEvent]

Parameters

required · UUID

optional · bool · default:

optional · str | None · default:

Returns AsyncIterator[AnalyzeEvent] For examples, see the Python SDK guide.

`client.runs.stream_outputs`

Yield chunks from the tar archive containing run outputs.

Signature

async def stream_outputs(
    run_id: UUID,
    *,
    task_ids: Sequence[str] | None = None,
) -> AsyncIterator[bytes]

Parameters

required · UUID

optional · Sequence[str] | None · default:

Returns AsyncIterator[bytes] For examples, see the Python SDK guide.

`client.runs.download_outputs`

Download run outputs into a new directory and unpack nested task tarballs. Existing directories are refused. Extraction accepts only regular files and directories. Positive limits bound the downloaded archive, expanded bytes, and total member count.

Signature

async def download_outputs(
    run_id: UUID,
    output_dir: str | Path,
    *,
    task_ids: Sequence[str] | None = None,
    max_archive_bytes: int = 1073741824,
    max_expanded_bytes: int = 5368709120,
    max_entries: int = 100000,
) -> Path

Parameters

required · UUID

required · str | Path

optional · Sequence[str] | None · default:

optional · int · default:

Returns Path For examples, see the Python SDK guide.

`client.runs.update_concurrency`

Change an active run’s concurrency limit. Existing tasks continue running.

Signature

async def update_concurrency(
    run_id: UUID,
    *,
    concurrency: int,
) -> UpdateBenchmarkConcurrencyResponse

Parameters

required · UUID

required · int

Returns UpdateBenchmarkConcurrencyResponse For examples, see the Python SDK guide.

`client.runs.retag`

Put a run in the project whose handle is project, in any run state. Earlier project links are kept.

Signature

async def retag(run_id: UUID, project: str) -> ProjectResponse

Parameters

required · UUID

required · str

Returns ProjectResponse For examples, see the Python SDK guide.

`client.runs.shed`

Lower an active run’s concurrency and force stop the newest tasks above the new limit.

Signature

async def shed(
    run_id: UUID,
    *,
    concurrency: int,
    dry_run: bool = False,
) -> ShedBenchmarkResponse

Parameters

required · UUID

required · int

optional · bool · default:

Returns ShedBenchmarkResponse For examples, see the Python SDK guide.

`client.runs.stop`

Stop a run or selected tasks.

Signature

async def stop(
    run_id: UUID,
    *,
    force: bool = False,
    task_ids: Sequence[str] | None = None,
) -> StopBenchmarkResponse

Parameters

required · UUID

optional · bool · default:

optional · Sequence[str] | None · default:

Returns StopBenchmarkResponse For examples, see the Python SDK guide.

`client.runs.resume`

Resume unfinished work for a run; lambda_function replaces its post-run Lambda.

Signature

async def resume(
    run_id: UUID,
    *,
    concurrency: int | None = None,
    task_ids: Sequence[str] | None = None,
    secrets: Mapping[str, str] | None = None,
    service_headers: Mapping[str, str] | None = None,
    from_scratch: bool = False,
    update_agent: bool = False,
    benchmark_url: str | None = None,
    lambda_function: str | None = None,
) -> RetryOrResumeBenchmarkResponse

Parameters

required · UUID

optional · int | None · default:

optional · Sequence[str] | None · default:

optional · Mapping[str, str] | None · default:

optional · bool · default:

optional · str | None · default:

Returns RetryOrResumeBenchmarkResponse For examples, see the Python SDK guide.

`client.runs.retry`

Retry failed or selected work for a run; lambda_function replaces its post-run Lambda.

Signature

async def retry(
    run_id: UUID,
    *,
    concurrency: int | None = None,
    task_ids: Sequence[str] | None = None,
    secrets: Mapping[str, str] | None = None,
    service_headers: Mapping[str, str] | None = None,
    from_scratch: bool = False,
    update_agent: bool = False,
    benchmark_url: str | None = None,
    lambda_function: str | None = None,
) -> RetryOrResumeBenchmarkResponse

Parameters

required · UUID

optional · int | None · default:

optional · Sequence[str] | None · default:

optional · Mapping[str, str] | None · default:

optional · bool · default:

optional · str | None · default:

Returns RetryOrResumeBenchmarkResponse For examples, see the Python SDK guide.

Get started

Agents

Runs

Benchmarks

Integrations

Python SDK

CLI reference

Python SDK reference

Self-hosting

`client.runs.start`

`client.runs.fetch`

`client.runs.list`

`client.runs.iter`

`client.runs.stream`

`client.runs.events`

`client.runs.results`

`client.runs.preview`

`client.runs.metadata`

`client.runs.results_exist`

`client.runs.analyze`

`client.runs.stream_outputs`

`client.runs.download_outputs`

`client.runs.update_concurrency`

`client.runs.retag`

`client.runs.shed`

`client.runs.stop`

`client.runs.resume`

`client.runs.retry`

​client.runs.start

​client.runs.fetch

​client.runs.filter_options

​client.runs.list

​client.runs.iter

​client.runs.stream

​client.runs.events

​client.runs.results

​client.runs.preview

​client.runs.metadata

​client.runs.results_exist

​client.runs.analyze

​client.runs.stream_outputs

​client.runs.download_outputs

​client.runs.update_concurrency

​client.runs.retag

​client.runs.shed

​client.runs.stop

​client.runs.resume

​client.runs.retry

`client.runs.start`

`client.runs.fetch`

`client.runs.filter_options`

`client.runs.list`

`client.runs.iter`

`client.runs.stream`

`client.runs.events`

`client.runs.results`

`client.runs.preview`

`client.runs.metadata`

`client.runs.results_exist`

`client.runs.analyze`

`client.runs.stream_outputs`

`client.runs.download_outputs`

`client.runs.update_concurrency`

`client.runs.retag`

`client.runs.shed`

`client.runs.stop`

`client.runs.resume`

`client.runs.retry`