Skip to main content
In this tutorial you run a small test agent on one SWE-bench task with Vals AI’s hosted Valkyrie, and read the result. The test agent reads the task and changes nothing, so the run scores 0. That’s expected: the goal is to see each step of a run work. You need:
  • Python 3.12, uv, and Git.
  • A Vals AI account. To get one, send an email to contact@vals.ai.
  • A Daytona account, where your tasks’ sandboxes run.

​
Install the CLI

This installs the Valkyrie CLI as valkyrie, with the short alias valk. To check it, run valkyrie --help, which lists the commands.

​
Sign in

The CLI prints a link and a code, and your browser opens the link. Confirm the code and sign in. The output ends with your account and your organization, similar to the following:
If the CLI says that you aren’t in an organization yet, ask Vals AI to add you, and then run valkyrie login again.

​
Connect Daytona

Create a Daytona API key with access to sandboxes and snapshots. Save it in a file named daytona.json:
daytona.json
Replace API_KEY with your Daytona API key. Then store the file as a secret of your organization:
The output is similar to the following:
Point the daytona provider at that secret, using the reference that the output showed:
The CLI confirms with Provider daytona now uses the secret daytona.

​
Create a test agent

An agent is a directory with a contract.yaml that says how to run it. Create one whose agent prints the task’s problem statement and does nothing else:

​
Upload the agent

The CLI confirms with ✓ Agent 'hello-agent' pushed successfully! The agent is now in your organization’s library as hello-agent.

​
Run one task

Valkyrie creates a sandbox in your Daytona account, sets up the first SWE-bench task in it, runs the agent, and grades the result. The output starts with a summary of the benchmark and the agent, followed by a Run ID line. Notice the progress updates that follow: they stream until the run finishes, after a few minutes.

​
Read the result

Replace RUN_ID with the ID from the Run ID line. The CLI confirms with Results saved to 'results-RUN_ID.json'. In that file, final_evaluation.final_score is 0.0, because the test agent didn’t fix the task. You’ve run an agent on a benchmark task, from sandbox to score.

​
Next steps