Secure a benchmark service

To keep a deployed benchmark service private, have it answer only callers that hold its credential. You set the check in the service, and save the credential in Valkyrie, which sends it with every request to that service. This credential is separate from your Valkyrie login and access keys. Services that Vals AI hosts need none of this: Valkyrie authenticates to them for you.

Require a credential

A service built with create-benchmark-service accepts every request unless it runs with AUTH_REQUIRED=true. With it set, the service rejects requests until you implement the check. Override check_auth in your benchmark class:

import os

from benchmark_service import BenchmarkService


class MyBenchmark(BenchmarkService):
    async def check_auth(self, headers: dict[str, str]) -> bool:
        return headers.get("authorization") == f"Bearer {os.environ['SERVICE_TOKEN']}"

Header names arrive in lowercase. Requests that fail the check get 401 Unauthorized. The create-benchmark-service README shows how to allow some callers only some datasets.

Save the credential with the service

Store the credential as a secret of your organization, exactly as the header carries it, such as Bearer TOKEN, where TOKEN is the service’s token. Then name the secret when you save the service URL:

valkyrie secret set swebench-token

valkyrie benchmark service set swebench https://benchmark.example.com \
  --auth-secret swebench-token

Valkyrie sends the secret’s value as the Authorization header on every request to that service, for your runs and task listings. To use another header, add --auth-header X-Service-Token. Headers that Valkyrie controls, such as Host and X-Api-Key, are refused.

To change the credential, run valkyrie secret set swebench-token again. To stop sending it, save the service again without --auth-secret.
Retries and resumes don’t send the saved credential. Pass it with -H Authorization "Bearer TOKEN" to valkyrie run retry or valkyrie run resume.
Runs with --ignore-custom-services use the hosted service and don’t send the credential.
A local Tracker that reads secrets from a file can’t send a saved credential.

Send request headers

-H NAME VALUE adds a header to the benchmark service requests of one run. Repeat it for more headers:

valkyrie run start --agent my-agent --benchmark swebench --project my-project \
  -H X-Experiment sweep-a \
  -H X-Team research

When a header has the same name as the saved credential’s header, the saved credential wins.

Get started

Agents

Runs

Benchmarks

Integrations

Python SDK

CLI reference

Python SDK reference

Self-hosting

Secure a benchmark service

Require a credential

Save the credential with the service

Send request headers

​Require a credential

​Save the credential with the service

​Send request headers

Require a credential

Save the credential with the service

Send request headers