Run Tracker locally

A local Tracker keeps run state in a PostgreSQL container and writes artifacts and logs to a folder on your machine. Sandboxes run on Daytona, Modal, or your own Docker daemon, and no AWS account or login is needed. You need Docker, the Valkyrie CLI, and a Daytona API key or Modal token unless you only use Docker sandboxes.

Get the code

git clone https://github.com/vals-ai/Valkyrie.git

cd Valkyrie

make -C services/tracker install

Add your secrets

Create .env at the repository root with your provider settings and the secrets your agents declare:

.env

DAYTONA_API_KEY=API_KEY
DAYTONA_API_URL=https://app.daytona.io/api
DAYTONA_TARGET=us

MODAL_TOKEN_ID=TOKEN_ID
MODAL_TOKEN_SECRET=TOKEN_SECRET

OPENAI_API_KEY=OPENAI_KEY

Replace the following:

API_KEY: your Daytona API key
TOKEN_ID and TOKEN_SECRET: your Modal token’s ID and secret
OPENAI_KEY: an example agent secret, here an OpenAI API key

Each task reads only the selected provider’s settings and the variables its agent contract declares. Valkyrie saves the file’s path with the run, never its values. Agent contracts can’t declare DAYTONA_* or MODAL_TOKEN_* variables.

Start the Tracker

From the repository root:

make local-tracker

This starts PostgreSQL in Docker, applies migrations, and serves the Tracker at http://127.0.0.1:8000. Runs use the code in your checkout, including retries and resumes after you change it. If you restart the Tracker while tasks are waiting to start, retry the run afterwards. The default settings are in local-server.yaml at the repository root:

local-server.yaml

data_root: .local/data
secrets_file: .env

To use other locations, pass your own file with make local-tracker CONFIG=CONFIG_FILE, where CONFIG_FILE is its path. Relative paths in it resolve from its folder.

Choose a benchmark service

Use one of these:

A service Vals AI hosts. Start the Tracker with the hosted services’ domain, and set VALKYRIE_API_KEY in the CLI’s environment to a Valkyrie access key that can use the benchmark:
BENCHMARK_SERVICE_BASE_URL=benchmarks.vals.ai make local-tracker
Your own service, on your machine or at any URL. Run it without authentication and save its URL:
valkyrie benchmark service set my-benchmark http://127.0.0.1:8001

Start a run

export VALKYRIE_URL=http://127.0.0.1:8000

valkyrie agent push ./my-agent

valkyrie project create my-project --name "My Project" --spend-type development --subcategory new_benchmark_generation

valkyrie run start --agent my-agent --benchmark my-benchmark --project my-project \
  --provider daytona --concurrency 1

Without --provider or a default provider, runs use daytona. Every other valkyrie run command works against the local Tracker, which serves artifacts over HTTP, so the CLI doesn’t need access to data_root. Retries, resumes, and forced stops use the provider and secrets file saved on the run.

Docker sandboxes

Docker runs each task in a container on the daemon that the Tracker and your benchmark service share. Select the daemon with DOCKER_HOST, and pass --provider docker or set it as your default with valkyrie provider default docker.

Services Vals AI hosts can’t use Docker sandboxes. Start your own service with CBS_DOCKER_ENABLED=true, built with the create-benchmark-service version that services/tracker/pyproject.toml pins.
Docker can’t change network access between stages, so the benchmark and the agent must allow unrestricted ("*") access in every stage. Allowlists are refused before a container starts.
Containers can’t run Docker themselves.
If the Tracker stops mid-run, the run’s containers keep running. To remove them, run the following, replacing RUN_ID with the run’s ID:
docker rm -f $(docker ps -aq --filter label=io.vals.cbs.managed=true \ --filter label=run-id=RUN_ID)

Reset local state

Run records live in the PostgreSQL Docker volume and run files under data_root. Delete both together, or the remaining records point at missing files:

docker compose -f services/tracker/docker-compose.yml down -v

rm -rf .local

Settings

Setting	Where	Effect
`data_root`	`local-server.yaml`	Folder for artifacts and logs
`secrets_file`	`local-server.yaml`	File the provider settings and agent secrets come from
`secrets_manager_region`	`local-server.yaml`	Reads named secrets from AWS Secrets Manager in this Region instead of `secrets_file`, with the AWS credentials in the Tracker’s environment
`BENCHMARK_SERVICE_BASE_URL`	Tracker environment	Domain of the hosted benchmark services

With secrets_manager_region, map each provider to its secret with valkyrie provider set. Only this source can send a benchmark service’s saved credential.

Limits

Post-run Lambdas, valkyrie run results --s3, result previews, and Docent analysis need S3 storage, so a local Tracker can’t run them.
Some hosted benchmarks start sandboxes from snapshots or images that only Vals AI’s Daytona organization can read, so they fail with your own Daytona account.

Fix common errors

Error	Fix
`Invalid daytona settings: daytona.DAYTONA_TARGET: Field required`	Add the named keys to `secrets_file`.
`Local daytona execution requires a secrets_file`	Set `secrets_file`, or pass `--provider docker`.
`Agent secrets cannot read sandbox provider settings`	Remove `DAYTONA_` and `MODAL_TOKEN_` names from the agent contract.
`Snapshot quota exceeded`	Delete unused snapshots in your Daytona organization.
`Virtualenv not found!`	Run `make -C services/tracker install`.

Get started

Agents

Runs

Benchmarks

Integrations

Python SDK

CLI reference

Python SDK reference

Self-hosting

Run Tracker locally

Get the code

Add your secrets

Start the Tracker

Choose a benchmark service

Start a run

Docker sandboxes

Reset local state

Settings

Limits

Fix common errors

​Get the code

​Add your secrets

​Start the Tracker

​Choose a benchmark service

​Start a run

​Docker sandboxes

​Reset local state

​Settings

​Limits

​Fix common errors

Get the code

Add your secrets

Start the Tracker

Choose a benchmark service

Start a run

Docker sandboxes

Reset local state

Settings

Limits

Fix common errors