> ## Documentation Index
> Fetch the complete documentation index at: https://docs.arcprize.org/llms.txt
> Use this file to discover all available pages before exploring further.

# Benchmarking Tooling (BETA)

> Run repeatable agent evaluations.

Currently in beta, the benchmarking harness provides a repeatable way to measure AI performance across model providers.

## ARC-AGI-3 Benchmarking Harness

This developer harness runs model configurations against one or more ARC-AGI-3 environments, records each run, and produces an online scorecard.

## When to use it

* Compare model versions or prompt strategies on the same game set.
* Detect regressions after code or prompt changes.
* Generate official scorecards and replays for sharing.
* Experiment with model configurations and agent architectures.

## Quickstart

### Prerequisites

* **Python**: `3.12+`
* **uv**: package manager. Follow the [uv installation guide](https://docs.astral.sh/uv/getting-started/installation/)
* **ARC-AGI-3 API key**: required to talk to the ARC server. Sign up for a key [here](https://arcprize.org/platform)

### Install

Clone the repository:

```bash theme={null}
git clone https://github.com/arcprize/arc-agi-3-benchmarking.git
cd arc-agi-3-benchmarking
```

From repo root:

```bash theme={null}
uv venv
uv sync
```

Copy the example environment file:

```bash theme={null}
cp .env.example .env
```

### Setting up your environment

Set the ARC API key and your provider keys. You can put them in a `.env` file (see [`.env.example`](https://github.com/arcprize/arc-agi-3-benchmarking/blob/main/.env.example)) or export them in your shell.

The checked-in model configurations currently use these providers:

* [OpenAI](https://developers.openai.com/api/docs/quickstart)
* [Anthropic](https://platform.claude.com/docs/en/manage-claude/authentication)
* [Google Gemini](https://console.cloud.google.com/apis/credentials)
* [xAI](https://docs.x.ai/developers/quickstart)
* [OpenRouter](https://openrouter.ai/api-keys)

Set `ARC_API_KEY` and the provider key required by your chosen model configuration in `.env`. For example:

```bash theme={null}
ARC_API_KEY=your_arc_api_key_here
OPENAI_API_KEY=your_openai_key_here
ANTHROPIC_API_KEY=your_anthropic_key_here
GOOGLE_API_KEY=your_google_key_here
XAI_API_KEY=your_xai_key_here
OPENROUTER_API_KEY=your_openrouter_key_here
```

### Select your game

```bash theme={null}
uv run main.py --list-games
```

### Pick your model

```bash theme={null}
uv run main.py --list-configs
```

### Benchmark

```bash theme={null}
uv run main.py \
  --game=ls20 \
  --config=openai-gpt-5-4-2026-03-05
```

The `--game` option accepts comma-separated game ID prefixes. Omit it to run all available games:

```bash theme={null}
uv run main.py --config=openai-gpt-5-4-2026-03-05
```

Use `--tags=experiment,v1` to add custom scorecard tags.

### Action output

The harness accepts either free-text responses containing an available action name or structured JSON containing exactly one action:

```json theme={null}
{"actions": [{"action_type": "ACTION1"}]}
```

For `ACTION6`, include integer `x` and `y` values from 0 through 63:

```json theme={null}
{"actions": [{"action_type": "ACTION6", "x": 32, "y": 32}]}
```

Invalid structured responses are retried rather than parsed as free text. Before action metadata is sent to ARC, the harness trims large output and reasoning fields to keep the serialized payload within 16 KB. Full model responses remain available in local step recordings.

### Scorecards

When you run a benchmark, a scorecard is saved on the ARC server. If you are logged in, you can view them at [arcprize.org/scorecards](https://arcprize.org/scorecards).

## Learn More

See the [benchmarking repository](https://github.com/arcprize/arc-agi-3-benchmarking#readme) for the latest model configurations and implementation details.