Publish a dataset

Lay out the task folders

• Put each task in its own folder, named by its task ID, under one parent folder.
• A task folder holds task.toml, instruction.md, and tests/test.sh. It can also hold environment/ and solution/.
• A metric.py in the parent folder computes the dataset’s score from the task rewards.
• A dataset.toml in the parent folder pins the exact tasks and their digests, and its version becomes a tag.

Read secrets in task environments

• In task.toml, use a whole-value ${NAME} to read a secret available to the run. For JSON secrets, this reads the field named NAME.
• Use ${SECRET#FIELD} to select a different JSON field, or ${SECRET#FIELD:-fallback} to use a default when the secret or field is unavailable.
• These references work in agent, verifier, verifier sandbox, and solution environment values, including step verifiers. A field reference requires a JSON object secret.

[verifier.env]
API_KEY = "${ClientConfig#API_KEY}"

Push a folder

valkyrie dataset push my-benchmark/my-dataset ./tasks --tag v1

• The benchmark and the dataset are created when they don’t exist yet.
• Only tasks the dataset doesn’t store yet are uploaded. Pushing an unchanged folder makes no new version.
• Every version gets the latest tag, plus each --tag.

Install from Harbor, GitHub, or Hugging Face

valkyrie dataset install tb/tb2 terminal-bench/terminal-bench-2 --location harbor

Tag versions

valkyrie dataset versions my-benchmark/my-dataset

valkyrie dataset tag my-benchmark/my-dataset@2 stable

valkyrie dataset untag my-benchmark/my-dataset v1

• latest always moves. Another tag on a different version moves only with --force.
• untag removes a tag. latest can’t be removed.

Run an agent on a dataset

valkyrie run start --agent my-agent --dataset my-benchmark/my-dataset@stable

• After @, pass a revision number, a tag, or a digest. Without it, the run uses latest.
• --slice, --task-ids, and --concurrency work as in Start a run.

Inspect tasks

valkyrie dataset task list my-benchmark/my-dataset@2

valkyrie dataset task download my-benchmark/my-dataset@2 hello

Reference

• valkyrie dataset
• client.datasets and client.datasets.versions