Benchmarking Tooling (BETA)

Currently in beta, the benchmarking harness provides a repeatable way to measure AI performance across model providers.

ARC-AGI-3 Benchmarking Harness

This developer harness runs model configurations against one or more ARC-AGI-3 environments, records each run, and produces an online scorecard.

When to use it

Compare model versions or prompt strategies on the same game set.
Detect regressions after code or prompt changes.
Generate official scorecards and replays for sharing.
Experiment with model configurations and agent architectures.

Quickstart

Prerequisites

Python: 3.12+
uv: package manager. Follow the uv installation guide
ARC-AGI-3 API key: required to talk to the ARC server. Sign up for a key here

Install

Clone the repository:

git clone https://github.com/arcprize/arc-agi-3-benchmarking.git
cd arc-agi-3-benchmarking

From repo root:

uv venv
uv sync

Copy the example environment file:

cp .env.example .env

Setting up your environment

Set the ARC API key and your provider keys. You can put them in a .env file (see .env.example) or export them in your shell. The checked-in model configurations currently use these providers:

Set ARC_API_KEY and the provider key required by your chosen model configuration in .env. For example:

ARC_API_KEY=your_arc_api_key_here
OPENAI_API_KEY=your_openai_key_here
ANTHROPIC_API_KEY=your_anthropic_key_here
GOOGLE_API_KEY=your_google_key_here
XAI_API_KEY=your_xai_key_here
OPENROUTER_API_KEY=your_openrouter_key_here

Select your game

uv run main.py --list-games

Pick your model

uv run main.py --list-configs

Benchmark

uv run main.py \
  --game=ls20 \
  --config=openai-gpt-5-4-2026-03-05

The --game option accepts comma-separated game ID prefixes. Omit it to run all available games:

uv run main.py --config=openai-gpt-5-4-2026-03-05

Use --tags=experiment,v1 to add custom scorecard tags.

Action output

The harness accepts either free-text responses containing an available action name or structured JSON containing exactly one action:

{"actions": [{"action_type": "ACTION1"}]}

For ACTION6, include integer x and y values from 0 through 63:

{"actions": [{"action_type": "ACTION6", "x": 32, "y": 32}]}

Invalid structured responses are retried rather than parsed as free text. Before action metadata is sent to ARC, the harness trims large output and reasoning fields to keep the serialized payload within 16 KB. Full model responses remain available in local step recordings.

Scorecards

When you run a benchmark, a scorecard is saved on the ARC server. If you are logged in, you can view them at arcprize.org/scorecards.

Learn More

See the benchmarking repository for the latest model configurations and implementation details.

Get Started

Core Concepts

Further Reading

Learn More

Community

Benchmarking Tooling (BETA)

ARC-AGI-3 Benchmarking Harness

When to use it

Quickstart

Prerequisites

Install

Setting up your environment

Select your game

Pick your model

Benchmark

Action output

Scorecards

Learn More

​ARC-AGI-3 Benchmarking Harness

​When to use it

​Quickstart

​Prerequisites

​Install

​Setting up your environment

​Select your game

​Pick your model

​Benchmark

​Action output

​Scorecards

​Learn More

ARC-AGI-3 Benchmarking Harness

When to use it

Quickstart

Prerequisites

Install

Setting up your environment

Select your game

Pick your model

Benchmark

Action output

Scorecards

Learn More