Manage runs with Python

client.runs does from Python what valkyrie run does from the Valkyrie CLI. The rules for each action, such as which tasks a retry reruns, are on the CLI guides that each section links to. Every parameter is in the client.runs reference. The examples run inside async with ValkyrieClient() as client:, as in the quickstart.

Start a run

run = await client.runs.start(
    agent="my-agent",
    benchmark="swebench",
    model="anthropic/claude-sonnet-4-6",
    concurrency=10,
    dataset="default",
    label="nightly-swebench",
    agent_kwargs={"temperature": "0.2"},
    secrets={"ANTHROPIC_API_KEY": "anthropic-api-key"},
)
run_id = run.benchmark_id

agent is a name from your library or an AgentContractRequest. The other options match valkyrie run start, including task_ids, slice_str, provider, and priority.

Follow a run

current = await client.runs.fetch(run_id)
print(current.details.status)

async for update in client.runs.stream(run_id):
    print(update.details.status)

stream yields an update each time the run changes, until it finishes.

List runs

list returns one page. iter walks every page and keeps your filters:

from valkyrie.sdk import FetchBenchmarksRequest

nightly = FetchBenchmarksRequest(benchmark_name=["swebench"], label="nightly-swebench")
async for row in client.runs.iter(nightly):
    print(row.id, row.status, row.final_score)

options = await client.runs.filter_options()

filter_options returns the values your organization’s earlier runs used, which the filters accept.

Change, stop, and rerun a run

await client.runs.update_concurrency(run_id, concurrency=20)

await client.runs.shed(run_id, concurrency=10, dry_run=True)

await client.runs.stop(run_id, task_ids=["task-1"])

await client.runs.resume(run_id, concurrency=20)

await client.runs.retry(
    run_id, task_ids=["task-1"], benchmark_url="https://new-benchmark.example.com"
)

await client.runs.retag(run_id, "google-report-xyz")

stop(force=True) also deletes running sandboxes. resume and retry take the same options as the CLI, such as from_scratch, update_agent, secrets, and service_headers. See Manage a run for what each one reruns. retag moves the run to another project in any run state and returns that project; metadata(run_id).project shows the newest.

Get results

results = await client.runs.results(run_id)

subset = await client.runs.results(run_id, task_ids=["task-1", "task-2"])

metadata = await client.runs.metadata(run_id)

Passing task_ids recomputes the score over those tasks only. metadata returns the settings the run started with.

Download outputs

Download and extract the files your agents wrote into a new directory:

await client.runs.download_outputs(run_id, "./outputs", task_ids=["task-1"])

Leave out task_ids for the whole run. The directory must not exist yet. To stream the tar archive yourself instead:

from pathlib import Path

with Path("run-outputs.tar").open("wb") as archive:
    async for chunk in client.runs.stream_outputs(run_id):
        archive.write(chunk)

To list or download individual files, see Download artifacts.

Analyze a run

Send a finished run to its Docent analyzer. A first analysis needs the analyzer’s name from the agent contract:

async for event in client.runs.analyze(run_id, lambda_function="my-agent-analyzer"):
    print(event.event, event.data)

Get started

Agents

Runs

Benchmarks

Integrations

Python SDK

CLI reference

Python SDK reference

Self-hosting

Manage runs with Python

Start a run

Follow a run

List runs

Change, stop, and rerun a run

Get results

Download outputs

Analyze a run

​Start a run

​Follow a run

​List runs

​Change, stop, and rerun a run

​Get results

​Download outputs

​Analyze a run

Start a run

Follow a run

List runs

Change, stop, and rerun a run

Get results

Download outputs

Analyze a run