FlutterBench Methodology

01

FlutterBench is our evaluation framework designed to measure how AI coding agents perform within the Dart and Flutter ecosystem. The evaluation system consists of four core components:

Component	Description
Dataset	Real-world development tasks derived from critical user journeys (CUJs).
Test matrix	Multidimensional testing framework across models, agents, tooling configurations, and SDK versions.
Scoring system	Unified grading approach evaluating functional outcomes, code quality, and developer experience, paired with diagnostic telemetry.
Harness	Containerized automation infrastructure that executes evaluations at scale.

02

Evaluation tasks derive directly from Flutter's canonical Critical User Journeys (CUJs), which are the core workflows that developers perform regularly. This approach ensures evaluations reflect real-world developer needs rather than synthetic puzzles.

Each CUJ represents a combination of:

A Flutter developer persona (e.g., app developer, plugin developer, full-stack developer)
A high-level goal
The specific sequential steps required to achieve that goal

As an application developer

I want to resolve layout overflow errors in UI component trees

I use the error messaging to identify widgets causing the layout overflow in the widget tree.

I refactor the layout using flexible scrolling or bounding widgets to resolve the overflow error.

Implement widget tests to prevent regressions.

Note: We can't open-source the full evaluation tasks without contaminating the benchmark dataset, but we publish our canonical list of CUJs. With this list, along with the example task below, you can understand our evaluation philosophy for FlutterBench.

These CUJs are converted into Harbor tasks. Harbor is the framework used to run containerized evaluation tasks.

CUJs and Harbor tasks don't map cleanly one-to-one. Instead, the CUJ list serves as a guide to verify that core developer workflows are evaluated. In some cases, several CUJs combine into a single task, and vice-versa. Our most ambitious evaluations combine multiple Harbor tasks, and thus cover many CUJs.

Task Structure

Directory layout, instructions, environments, and verification criteria.

Each task lives in its own directory and contains four elements:

Element	Description
Instruction	A realistic prompt written the way developers talk to agents (typically 1–2 sentences, behavior-focused rather than prescriptive).
Target codebase & environment	A containerized Docker environment preseeded with a Dart or Flutter codebase, testing greenfield generation or existing codebases with bugs or debt.
Verification criteria	Verification scripts ( `tests/graders.dart` and `test.sh` ) executing `package:eval_scoring` .
Configuration & metadata	The `task.toml` configuration defining associated CUJs, priority tiers, expected skills, and MCP tools.

Task Categories

Primary agent capabilities and workflows evaluated across benchmarks.

Tasks are categorized by the primary agent capability being evaluated:

Category	Description
Greenfield generation	Creating new features or applications from scratch.
Hill climbing	Iterative debugging, test repair, and multi-turn problem-solving.
Refactoring	Restructuring existing code while maintaining functionality.
Migration	Upgrading deprecated APIs or transitioning between architectural patterns.
Integration	Adding platform-specific features, native plugins, or third-party packages.

Task Tiers & Prioritization

Execution cadences and priority levels balancing coverage and speed.

To balance comprehensive coverage with evaluation speed, tasks are organized into execution tiers and priorities:

Execution tiers

Tier	Description
Tier 1	Core benchmark tasks executed monthly across the complete evaluation matrix.
Tier 2	Maturing tasks slated to graduate into Tier 1 once calibrated.
Tier 3	Experimental tasks used for ad-hoc investigations and targeted questions.

Task priority

Priority	Description
P0	Critical production workflows and high-frequency productivity tasks
P1	Core functionality and API consistency verification
P2	Standard features and application maturity tasks
P3	Edge cases and cosmetic polish

Quality Assurance

Engineering audits and human reviewer calibration for dataset integrity.

Before a task graduates into the core benchmark suite, it undergoes an engineering audit. Human reviewers inspect initial trial runs and label results as:

Audit label	Definition
True positive	Agent correctly passed the task.
True negative	Agent correctly failed the task.
False positive	Agent passed incorrectly due to overly lenient checks.
False negative	Agent failed incorrectly due to brittle or flaky tests.

This process ensures that the dataset produces reliable, actionable signals rather than noise.

Each task contains an instruction, a target codebase and environment, verification criteria, and metadata. Using the CUJ example above, a corresponding Harbor task looks like this:

environment/
- lib/
- test/
tests/
solution/

Task

environment/

Containerized Flutter workspace pre-seeded for the agent

The target codebase is an isolated, containerized Flutter workspace.

When a benchmark run starts, the evaluation harness boots an ephemeral Docker container pre-seeded with this project. The agent is given access to tools (such as reading files, editing code, and running terminal commands) to investigate and resolve the issue.

The agent only sees the files inside this directory. Grading scripts and reference solutions remain strictly isolated outside the container until the agent completes its run.

Contents

Task/environment

lib/

Application source code

Contains the Flutter application source code.

In this evaluation task, the agent inspects lib/main.dart to locate the source of the RenderFlex layout errors and applies appropriate widget modifications.

Contents

Task / environment / lib

main.dart

Flutter app containing horizontal and vertical overflow bugs

The starter code for the Flutter application. Alternatively, the docker file may point to a remote codebase.

Task/environment

test/

Target directory for agent-authored widget tests

The test directory for the target project.

Initially, this directory is empty. The task instruction directs the agent to create test/main_test.dart to verify its bug fix with automated widget tests.

Contents

Task / environment / test

main_test.dart

Widget tests the agent must write to verify its fix

The widget test file that the agent is expected to author.

Task/environment

pubspec.yaml

Project manifest declaring Flutter and lint dependencies

Declares dependencies and environment constraints for the Flutter project.

Task/environment

analysis_options.yaml

Linter configuration enforcing const and style rules

Defines the static analysis rules enforced across the project.

During grading, StaticAnalysisGrader runs flutter analyze against the modified codebase. Solutions that introduce analyzer errors, warnings, or lint violations (such as missing const constructors) lose quality points.

include: package:flutter_lints/flutter.yaml

linter:
  rules:
  # Strict lints for this task

Task/environment

Dockerfile

Container setup, SDK caching, and baseline commit

Defines the container image for the task environment.

Workflow: 1. Inherits from the pre-warmed flutter-linux:latest base image, which contains the Flutter and Dart SDKs. 2. Copies project manifests and initial source files into /workspace. 3. Runs flutter pub get so dependencies are pre-fetched. 4. Executes init-baseline to snapshot the clean repository state. This allows the evaluation harness to measure the agent's work as a precise Git diff against the starting baseline.

Task

tests/

hidden from agent

Multi-dimensional evaluation harness hidden from the agent

The automated verification harness used to grade the agent's performance.

This entire directory is hidden from the agent during task execution. It is mounted into the evaluation container only when the grading phase starts.

The harness evaluates the agent's output across three weighted dimensions:

Outcome: Functional correctness, test pass rates, and layout fixes. Quality: Static analysis, formatting, and code craftsmanship. DX: Developer experience and tool interaction efficiency.

Contents

Task/tests

graders.dart

Outcome, quality, and craftsmanship scoring rules via eval_scoring

Defines the multi-dimensional scoring pipeline using a separate scoring package.

// Heavily edited
AggregateGrader grader(context) =>
    AggregateGrader.result(
      outcome: _outcomeGraders,
      quality: _qualityGraders,
      dx: _dxGraders,
    );

Task/tests

test.sh

Verification entry point running the eval_scoring suite

The test harness entrypoint script executed inside the grading container.

#!/bin/bash
eval_scoring run [tasks]

Task

solution/

Oracle reference solution used to validate the eval task

Contains the ground truth reference implementation maintained by the benchmark authors.

Before a task is added to the benchmark suite, the reference solution is executed through the grading harness to confirm that it achieves a full 1.0 reward. This ensures the task is solvable, unambiguous, and calibrated correctly.

Contents

Task/solution

solve.sh

Reference script fixing layout bugs and adding layout tests

The reference shell script that applies the canonical fix and generates comprehensive tests. Not safe to share publicly.

Task

instruction.md

input

The prompt the agent receives

The instruction is the task prompt provided to the agent. It mimics real-world workflows from developers, and is written in a way that real developers interact with agents.

Design principles:

Prompts state symptoms and expected outcomes without naming exact remedy widgets. Prompts require the agent to write regression tests, measuring both implementation skills and testing rigor.

We currently keep receiving `RenderFlex overflowed` errors both horizontally and vertically. Implement the correct widgets to resolve these layout issues.

Write widget tests in
`test/main_test.dart` that verifies the changes.

Task

task.toml

Task definition, difficulty notes, and timeouts

The task configuration file specifies execution bounds, metadata, and target artifacts evaluated during grading.

Key sections:

artifacts: Lists files that must be present in the workspace after the run. task definition: Task-related data, like name, author and tags. infra and agent configurations, such as timeout limits and retries.

03

4-Axis Evaluation Matrix

Select an evaluation axis to explore its configurations, supported targets, and methodology.

Axis 1: Tooling Configurations

3 Setups

FlutterBench tests three configurations to isolate the impact of specialized Dart and Flutter AI tools (skills, MCP tools, compiler diagnostics, and sandboxes).

Configurations Tested:

Baseline: No specialized Dart or Flutter tools loaded — tests raw model capability and baseline reasoning.
Enhanced: Full suite of Dart and Flutter skills, MCP server tools, and compiler diagnostics loaded.
Enhanced Minus N: Full suite with a specific tool or skill group ablated to measure its isolated delta.

This A/B testing approach isolates whether adding specific tools (such as widget inspection or analyzer fixes) measurably improves task success rates.

Axis 2: Frontier Model Families

3 Families

Evaluations span major frontier model families across high-capability reasoning tiers and low-latency production tiers.

Model Families Evaluated:

Gemini: Evaluated across Gemini Pro (deep reasoning and complex architecture) and Gemini Flash (low-latency generation) tiers.
Claude: Evaluated across Claude Opus (heavy multi-file refactoring) and Claude Sonnet (daily developer workflows).
ChatGPT: Evaluated across the GPT-4o series and specialized reasoning checkpoints.

Additional providers and local open-weights models are integrated continuously based on community feedback and developer adoption.

Axis 3: Agent Runtime Harnesses

2 Harnesses

Different agents employ distinct system prompts, context management techniques, and tool-loading strategies. We evaluate the CLI agents Flutter developers use most.

Agent Harnesses Tested:

Antigravity CLI: Google's developer agent harness featuring optimized tool discovery, compact context budgets, and deep IDE integration.
Claude Code: Anthropic's terminal-based agent harness with autonomous multi-turn problem-solving capabilities.

Testing across multiple harnesses isolates model intelligence from agent runtime orchestration, measuring how execution environments influence success.

Axis 4: SDK Release Channels

2 Channels

Evaluations run against both Flutter release channels to verify developer-ready reliability and detect framework regressions early.

Release Channels Tracked:

Stable: The current production SDK release — measures developer-ready reliability on the version used by most teams.
Beta: Monthly beta releases — catches deprecation migrations, API evolutions, and analyzer changes before stable rollout.

Ensures AI tooling and skills maintain precision as the Flutter framework and Dart language evolve.

04

When evaluating AI coding agents, execution friction—such as tool failures, endless retries, and hallucinations—is often attributed entirely to model capability. However, AI coding systems follow a core equation:

Agent = Model + Harness

While the Dart and Flutter teams do not train the underlying LLMs, we build and maintain the Dart and Flutter AI Harness (skills, MCP tools, compiler diagnostics, and sandboxes). Therefore, Developer Experience (DX) is directly within our engineering control and belongs in our primary benchmark score alongside functional outcomes and code quality.

Primary focus: The quality and correctness of the final code artifact.
First-class signal: Developer experience friction, tool accuracy, and recovery efficiency.

Each evaluation produces three independent dimensions that compute the composite Result Score in Harbor's reward:

60%Outcome

30%Code Quality

10%Developer Experience

1. Outcome

Evaluates whether the generated code compiles, runs, and fulfills all functional UI and task requirements correctly.

Build, Tests, Visual, HeuristicsFunctional

2. Code Quality

Measures code maintainability, static analysis lints, architectural health, and idiomatic Dart 3 standards.

Lints, DCM, Structure, PatternsMaintainable

3. Developer Experience

Tracks agent execution friction, tool call accuracy, error recovery loops, and step efficiency.

Tooling, Trajectory, RecoveryEfficiency

FlutterBench deploys a mix of deterministic, LLM-as-a-judge, and heuristic graders across all three dimensions:

Grader Matrix

Explore deterministic, LLM-as-a-judge, and heuristic evaluation graders.

Outcome Deterministic

Build & Run

Binary pass/fail checking if code compiles and launches cleanly.

Outcome Deterministic

Unit & Widget Testing

Executes test suites and scores passing test ratio across affected code.

Outcome LLM-as-Judge

Visual Validation

Compares DevTools screenshot captures against visual UI expectations.

Outcome Heuristic

Task Heuristics

Task-specific checks verifying dependency additions, required config, and files.

Quality Deterministic

Static Analysis

Runs dart analyze enforcing strict project lint rules.

Quality Deterministic

Dart Code Metrics (DCM)

Detects dead code, widget complexity, undisposed controllers, and memory leaks.

Quality LLM-as-Judge

Idiomatic Dart Review

Evaluates modern Dart 3 pattern compliance and adherence to Effective Dart.

Quality Deterministic

Structural Validation

Verifies directory structure, naming conventions, and required file placement.

DX Deterministic

Tool Usage Analysis

Checks whether appropriate MCP tools and CLI utilities were called effectively.

DX Deterministic

Step & Token Economy

Measures trajectory length, token burn, and redundant execution steps.

DXLLM-as-Judge

Error Recovery

Rates how fluidly the agent adapts to compiler, analysis, and test errors.

Tier	Grader Type	Evaluation Role
Code-based	Compilers, test runners, `dart analyze`, DCM, structural checkers	Deterministic, unambiguous source of truth for syntax, compilation, and functional test assertions.
LLM judge (BINEVAL)	Frontier model rubric evaluation	Evaluates qualitative dimensions (visual UI, idiomatic review, trajectory, recovery) using binary yes/no questions.
Human audit	Flutter engineer manual review	Ground truth calibration, failure root-cause analysis, and conflict resolution.

To avoid penalizing capability scores on complex tasks that naturally require more reasoning steps or tokens, FlutterBench tracks diagnostic telemetry separately from the Result Score:

Metric	Measurement Target
Token usage	Tracks total input, cache, and output tokens consumed to measure efficiency deltas and verify token reductions from skill optimizations.
Expected tool calls	Compares actual tool invocations against expected tools. If an agent succeeds without using an expected tool, it is not penalized; this telemetry helps evaluate whether the tool is necessary for that user journey.

05

06

Single-run trials only sample luck. True agent trust requires measuring multi-trial stability across repeated runs:

Ceiling Metricpass@k

Capability

The probability that an agent succeeds at least once across k attempts. Shows what the model can achieve under ideal sample runs.

Our North Star pass^k

Consistency

The probability that an agent succeeds every single time across all k attempts. Solves unreliability friction for real developer workflows.

Click a score tier to view its grading criteria and actionable engineering triage steps:

Score Triage & Action Matrix

Select a score range to view its criteria and immediate engineering triage actions.

Score 1.00 — Perfect Success

Criteria & Triage

Code compiles, runs, and passes all unit and widget tests. Zero analyzer or DCM warnings. Idiomatic Dart 3 and flawless DX.

Engineering Triage Actions:

Telemetry check: Validate whether all expected tools were utilized.
Skip optimization: If expected tools were skipped successfully, re-evaluate if the tool is redundant.
Token burn review: If token consumption was high, optimize prompt context and create fast-path shortcut tools.

Score 0.75 – 0.99 — Minor Flaws

Criteria & Triage

Code works and functional tests pass, but exhibits minor lint warnings, slightly unidiomatic patterns, or minor DX friction.

Engineering Triage Actions:

Prompt tuning: Tune skill prompt instructions for better Dart 3 style enforcement.
Schema refinement: Refine tool parameter schemas and validation logic to prevent minor parameter retry hiccups.
Model collaboration: Collaborate with model team to polish code generation formatting.

Score 0.50 – 0.74 — Partial Success

Criteria & Triage

Core requirements work but some widget or unit tests fail, lint warnings are significant, or the agent entered noticeable recovery loops.

Engineering Triage Actions:

Assertion triage: Investigate specific test assertion failures.
Context gap analysis: Analyze whether the agent lacked key framework or package context.
Diagnostic feedback: Refine compiler diagnostic feedback to help the agent self-correct faster.

Score 0.25 – 0.49 — Poor Implementation

Criteria & Triage

Fails to compile, ignores constraints, or encountered severe developer experience breakdown during execution.

Engineering Triage Actions:

Tool discoverability: If expected tools were ignored, fix tool discoverability, frontmatter, and prompt instructions.
Tool output clarity: If tools were used but failed, improve helper tool output clarity and error messaging.
Loop prevention: If token burn was high due to endless fix loops, improve compiler errors for single-step healing.

Score 0.00 — Total Failure

Criteria & Triage

No working code produced, severe runtime crash, or agent caught in an infinite loop.

Engineering Triage Actions:

Human root-cause audit: Engineers inspect reasoning traces, plan adherence, and test harness logs.
Behavior categorization: Identify premature surrender versus infinite loop traps.
Constraint refinement: Update benchmark task prompt constraints and bounding parameters.

When an evaluation task receives a low Result Score, human expert reviewers inspect the diagnostic process data:

Reasoning trace review: Inspect the agent's internal thoughts to identify where misunderstandings of Dart/Flutter APIs occurred.
Plan adherence audit: Check whether the agent derailed due to ambiguous task prompts or missing context.
Harness diagnostics: Audit error messages returned to the agent during failed compile/test steps to see why recovery failed.

07

To prevent model training contamination, raw datasets and reference solutions cannot be open-sourced. However, the Flutter team maintains transparency by:

Publishing the comprehensive evaluation methodology on this page.
Sharing task prompts and Critical User Journey (CUJ) lists.
Publishing regular blog posts with analysis and insights.
Open-sourcing verification tooling that does not risk dataset compromise.

To run benchmark tasks locally using the Harbor evaluation runner:

# Run an individual trial with Harbor
harbor run \
  --task dataset/flutter/manage-state-with-bloc \
  --agent antigravity-sdk \
  --model google/gemini-3.5-flash \
  --mcp dart

This methodology will evolve as more data is gathered and analyzed. Expect updates and refinements in future blog posts and documentation.

For questions or to contribute new CUJ evaluation tasks, visit the Flutter repository on GitHub.

FlutterBench Methodology

Overview

Dataset & tasks

Task derivation

Task specifications

Task Structure

Task Categories

Task Tiers & Prioritization

Quality Assurance

Interactive task anatomy

Evaluation test matrix

4-Axis Evaluation Matrix

Axis 1: Tooling Configurations

Axis 2: Frontier Model Families

Axis 3: Agent Runtime Harnesses

Axis 4: SDK Release Channels

Scoring architecture

Scoring philosophy

Three core evaluation dimensions

1. Outcome

2. Code Quality

3. Developer Experience

Grader matrix

Grader Matrix

Build & Run

Unit & Widget Testing

Visual Validation

Task Heuristics

Static Analysis

Dart Code Metrics (DCM)

Idiomatic Dart Review

Structural Validation

Tool Usage Analysis

Step & Token Economy

Error Recovery

Grader implementation tiers

Diagnostic telemetry (excluded from Result Score)

Evaluation harness

Reliability & triage

Multi-run reliability metrics

Capability

Consistency

Score interpretation & triage

Score Triage & Action Matrix

Score 1.00 — Perfect Success

Score 0.75 – 0.99 — Minor Flaws

Score 0.50 – 0.74 — Partial Success

Score 0.25 – 0.49 — Poor Implementation

Score 0.00 — Total Failure

Human root-cause audits

Transparency & dataset integrity