Introducing FrontierEE Bench

Evaluating frontier AI models on real-world EE tasks at a physical lab bench

For frontier models to advance electrical engineering, they need to be trained on the realities of a physical lab: imperfect measurements, unexpected behavior, and decisions made with incomplete evidence. Building that scientific judgment starts with understanding where models struggle today. We built FrontierEE Bench to investigate those limits, putting models in control of real instruments and experiments across 10 domains of electrical engineering.

On this page

Results

ModelPass@1Cost per trial ($)Mean API spend per trial in US dollars, over all 10 tasks.
Claude Fable 5.2high36.7%$4.44
GPT-6 Astrahigh30.0%$7.06
Claude Opus 5.5high30.0%$1.57
Gemini 4 Carbonhigh26.7%$1.60
GPT-6.1 Solhigh23.3%$1.05

As of 17 November 2026. Five models, three runs per model–task pair, using native harnesses at high reasoning effort. Pass@1 covers 10 tasks. Cost is mean API spend per run over all 10 tasks.

Best pass rate
36.7%
Avg pass rate
29.3%
Avg time per trial
51 min
Performance vs. cost
FrontierEE Bench cost-performance Pareto frontierCost per run, USD on a log scale versus Pass@1 on all 10 tasks, with 95% credible intervals. GPT-6.1 Sol, Claude Opus 5.5, Claude Fable 5.2 form the Pareto frontier.0%20%40%60%80%$1$2$5$10Claude Fable 5.2: $4.44 per run, 11/30 passed (37%, 95% interval 28–53%), on the Pareto frontierClaude Fable 5.2GPT-6 Astra: $7.06 per run, 9/30 passed (30%, 95% interval 24–46%)GPT-6 AstraClaude Opus 5.5: $1.57 per run, 9/30 passed (30%, 95% interval 23–47%), on the Pareto frontierClaude Opus 5.5Gemini 4 Carbon: $1.60 per run, 8/30 passed (27%, 95% interval 21–44%)Gemini 4 CarbonGPT-6.1 Sol: $1.05 per run, 7/30 passed (23%, 95% interval 19–42%), on the Pareto frontierGPT-6.1 SolCost per run, USD (log scale)Pass@1 on all 10 tasks
On the Pareto frontierOff the frontierPareto frontier95% credible interval
  • Claude Fable 5.2$4.44 · 36.7%
  • GPT-6 Astra$7.06 · 30.0%
  • Claude Opus 5.5$1.57 · 30.0%
  • Gemini 4 Carbon$1.60 · 26.7%
  • GPT-6.1 Sol$1.05 · 23.3%

Means over each model’s runs on all 10 tasks (30 runs per model). A model is on the frontier when no cheaper model scores as high. The intervals overlap, so the frontier can change with more trials.

Behind each trial is a physical lab bench, where models investigate real components through experiments and measurements.

Inside the tasks

Tasks span a wide range of EE disciplines, including mixed-signal circuits, power electronics, electromechanical systems, and precision instrumentation. Each starts with a job an engineer might bring to a lab, with competing requirements and physical constraints that the model must work through experimentally.

Technical illustration of a protection diode on a circuit board, with a temperature sensor attached to its body.

Diode runs hot

Establish how much transmit current an unchanged panel can sustain without overheating or losing too much voltage, accounting for the gap between the diode’s body-sensor reading and its junction temperature.

Cutaway illustration of a level sensor mounted above a sump with foam covering the liquid surface.

Sump level measurement

Configure a sump sensor that falsely reports empty under foam, establishing its highest reliable alarm and usable range while accounting for shaft geometry, air temperature and reading frequency.

Technical illustration of a mounted gearmotor with an output shaft coupling, wiring, and a breadboard on a baseplate.

Motor identification

Characterize an unlabeled gearmotor for drive configuration, separating winding properties from brush-contact and heating effects while establishing its back-EMF constant and exact gearbox ratio with the coupling fitted.

Observed failure modes on the bench

Models solved the simulated tasks but struggled with their physical counterparts. The failures revealed how they handled unexpected results and incomplete evidence.

  1. Changing variables without isolating effects

    Changing several conditions at once sometimes improved a result without establishing why. Models credited one adjustment even when the experiment left other explanations open.

  2. Ignoring the effects of previous experiments

    Starting a new test doesn’t reset the hardware. Models sometimes treated a warm component or a partly charged circuit as though it had returned to its original state, carrying errors from one experiment into the next.

  3. Misinterpreting a measurement

    An instrument can give an accurate reading without answering the question that matters. Models used surface temperatures to infer internal temperatures without establishing how closely the two tracked each other.

  4. Leaving relevant physics out of the calculation

    The arithmetic could be right while the assumed circuit or thermal model was incomplete. Overlooking an existing current path or a route for heat to escape led models to infer the wrong properties from real measurements.

We also report safety concerns observed during runs, though no run failed solely for a safety issue: the benchmark focuses on scientific judgment.

AgentDUT, our custom lab harness

We built agentdut, a custom harness that lets AI agents run electrical engineering experiments on real hardware. Through the bench CLI, models control networked lab instruments and request camera images of the setup. They choose what to test, collect measurements and decide how to proceed when results challenge their assumptions.

For physical changes, the model submits an operator ticket. A person moves the probe, replaces the component or changes the wiring, then reports completion so the model can continue.

The harness checks requests against configured operating limits and records instrument commands, camera images, measurements and operator actions in a shared experimental trace, so each run can be reviewed step by step.

How we built the benchmark

We built tasks around real-world decisions an engineer faces at a lab bench, like how much current a panel can sustain, which sensor readings a controller can trust, or whether an embedded model meets its accuracy, timing and power requirements. Each task specifies a deliverable, the hardware and conditions it must work under, and what the engineer is allowed to change.

  1. The brief

    The model receives an engineering objective, requirements, constraints, available equipment, and execution budget. It chooses its measurements, develops an experimental approach and decides when it has enough evidence to submit a result. We leave those decisions open so the task tests experimental judgment.

  2. The physical environment

    Through our custom harness, agentdut, models use the bench CLI to control instruments, collect measurements, and request camera images. Tasks can also expose device-specific actions, like flashing a firmware configuration. When an experiment requires physical changes, an operator carries out the model’s instructions. The model is responsible for choosing the experiment and interpreting its results.

  3. The verifier

    Each task has a deterministic verifier that assesses the submission using protected bench records and task-specific acceptance tests. Depending on the task, the harness can test a module at its claimed operating limit, check a sensor correction against known physical positions, or evaluate deployed firmware on a held-out audio session. Scores come from recorded measurements and explicit checks; the pass/fail decision does not use an LLM judge.

  4. Quality control

    We develop reference solutions alongside the tasks to test whether verifiers reject plausible wrong answers and shortcuts while accepting valid alternative approaches. We check the correspondence between the brief and the scoring logic, audit the files and tools available to the model, and investigate failed runs for task, harness and hardware defects.

    Physical commissioning checks the assumptions behind the experiment, including instrument uncertainty, acquisition timing, fixture geometry and repeatability. We retain versioned validation evidence and distinguish missing or corrupted bench evidence from demonstrated failures to meet the engineering requirements.

Data and evaluations

We work with AI labs on training data and evaluations grounded in real engineering work. Get in touch to evaluate your model on physical hardware, build new tasks with us, or discuss training data from real experiments.

Get in Touch

Interested in the benchmark or building with us? We’d love to hear from you.