Benchmark

Every figure and every number on this page is rendered from one file, results/benchmark_summary.json. Nothing here is typed into Markdown by hand, so the table, the chart and the home page cannot disagree with each other.

Published run — WM-811K, 12 wafers × 5 seeds

Written by python -m nano.benchmark --ablation against a local copy of WM-811K. The raw dataset is not committed; data/subsets/wm811k_eval.json names the twelve wafers, so the evaluation set can be rebuilt exactly. Two things below are worth knowing before reading the rest: NANO separates from Random and Grid on the primary metric and not on plain MAE, and the ablation finds a reduced rule beating the published one.

A results file is committed only for a run against the dataset. A run on generated stand-in wafers (--synthetic) produces the same file locally and is labelled as such — the table prints the wafer source — but stand-in numbers are not published as dataset results. See Reproduce.

What is being compared

All three arms receive the same wafers, the same seeds, the same initial observation mask and the same additional measurement budget. The only difference is which die each one picks next.

Strategy	Selection rule
Random	Uniformly sample an unmeasured die
Grid	Pick the unmeasured die closest to the next point on a spatially uniform lattice
NANO	Pick the unmeasured die with the highest acquisition score

The rule each arm ran is recorded in the results file, and all three are driven by the same loop in nano/agent.py through the same select(estimate, observed_mask, coords, rng) signature. Grid sizes its lattice to the budget and consumes it in a fixed order; Random draws uniformly from the unmeasured dies with the episode's seeded generator.

A fair comparison here is a matter of protocol, not of intent: if NANO started from a different initial mask or spent a larger budget, any improvement would be uninterpretable.

Headline results

Final reconstruction Balanced MAE ↓ after the full measurement budget, averaged over 12 wafers × 5 seeds. All strategies start from the same initial observations and spend the same budget.

Strategy	Selection rule	Balanced MAE mean ↓	± std	Improvement vs NANO
NANO	highest acquisition score	0.3479	0.0524	reference
Random	uniform over unmeasured dies	0.3812	0.0527	NANO better by 8.7% 95% CI [6.6%, 11.1%], NANO better in 53/60 episodes
Grid	spatially uniform lattice	0.3714	0.0494	NANO better by 6.3% 95% CI [4.6%, 8.1%], NANO better in 49/60 episodes
Uncorrected prior spends no budget	the simulation prior, with no measurement applied	0.4376	0.0567	NANO better by 20.5% 95% CI [18.6%, 22.5%], NANO better in 59/60 episodes
Constant spends no budget	predicts 0.0 everywhere — "no die fails" — and reads no measurement	0.5000	0.0000	NANO better by 30.4% 95% CI [27.8%, 33.0%], NANO better in 60/60 episodes

The same run scored on every other metric. A number marked * is one where the paired interval spans zero, so the arms are not separated on that metric.

Strategy	MAE ↓ all in-wafer dies	MAE on failing dies ↓ dies whose hidden value is 1.0	MAE on passing dies ↓ dies whose hidden value is 0.0	RMSE ↓ all in-wafer dies
NANO	0.2292	0.4626	0.2332	0.3319
Random	0.2337 *	0.5346	0.2278 *	0.3421
Grid	0.2299 *	0.5314	0.2115	0.3402
Uncorrected prior	0.2946	0.7009	0.1744	0.3875
Constant	0.3325	1.0000	0.0000	0.5409

Metrics disagree here by construction. Plain MAE over every in-wafer die is led by the majority class when failures are rare, which is why the primary metric weights the two classes equally and why both are printed.

• Wafer source: WM-811K
• Initial measurements: 20
• Additional measurement budget: 60
• Simulation prior error before any correction: 0.4376
• Commit: 85804cf
• Generated: 2026-09-06T10:27:49Z
• Source: results/benchmark_summary.json

How relative improvement is defined

relative improvement = (baseline error − NANO error) / baseline error × 100

Positive means NANO reconstructed the wafer more accurately than the baseline at equal cost. Negative means it did not, and the table prints that with the same prominence.

Why every improvement carries an interval

A difference of means is not a result. Each arm is paired with NANO episode by episode — same wafer, same seed, same prior, same initial mask, so the difference within an episode isolates the selection rule — and the paired differences are bootstrapped (2000 resamples, seed recorded in the file) into a 95% interval.

If that interval spans zero, the table prints not separated instead of a percentage, however good the mean looks. The win count is printed next to it, because "better on average" and "better on most wafers" are different claims and a benchmark should not let them be confused.

Why the primary metric is not plain MAE

Most dies pass. On a wafer where 12% of dies fail, a predictor that answers "no die fails" everywhere and reads no measurement at all scores an excellent MAE — better than any strategy here. That is not a property of the strategies; it is class imbalance being scored as if it were knowledge.

So the primary metric is balanced MAE: the mean of the error on failing dies and the error on passing dies. A predictor that knows nothing scores exactly 0.5 on it, whatever the failure rate. The criterion is that chance must look like chance — not that any particular arm wins.

Plain MAE is still computed, still written into the results file, and still printed on this page in the second table. Where the two disagree, both are shown. Override the choice with python -m nano.benchmark --metric mae; the underlying per-episode numbers do not change, only which one leads.

The two rows that spend no budget

Two predictors appear in the table marked spends no budget:

Reference	What it answers	Why it is there
Uncorrected prior	The simulation prior, with no measurement applied	The floor the whole exercise is meant to beat
Constant	0.0 everywhere on a binary target — reads nothing	The trap plain MAE walks into, made visible instead of hidden

A selection rule that cannot beat a predictor which reads nothing has not earned the tool time it spent. Putting both in the same table as the strategies is the cheapest way to keep that check from being skipped.

Measurements versus error

Reconstruction Balanced MAE ↓ against measurement count. Shaded bands show ±1 standard deviation across seeds. The y-axis starts at zero. Generated from results/benchmark_summary.json.

The interesting part of this curve is its shape, not just its endpoint. A selection rule that is merely lucky converges at the same rate as Random and separates only at the end. A rule that is genuinely choosing informative locations should separate early, while budget still remains.

Does each term of the acquisition rule earn its place?

The selection rule is a product of three terms, and a product of three terms is a claim: that each one changes the decision for the better. The way to find out is to drop them and re-run, so python -m nano.benchmark --ablation runs the reduced rules as extra arms — same loop, same budget, same initial observations, only the terms differ.

Final Balanced MAE for the acquisition rule with terms dropped. Every arm ran the same loop, the same budget and the same initial observations; only the terms of the product differ.

Acquisition rule	Balanced MAE ↓	± std	Against the published rule
uncertainty × disagreement × novelty published rule	0.3479	0.0524	reference
uncertainty	0.3403	0.0555	beats the published rule by 2.2%this rule better in 35/60 episodes
uncertainty × disagreement	0.3503	0.0564	not separated this rule better in 25/60 episodes
uncertainty × novelty	0.3574	0.0524	loses to the published rule by 2.7% this rule better in 11/60 episodes
disagreement × novelty	0.3534	0.0551	loses to the published rule by 1.6% this rule better in 24/60 episodes

A row that beats the published rule is a finding, not a footnote: the term it drops is costing accuracy on this wafer source rather than buying it.

This is the table most likely to embarrass the rest of this site, which is exactly why it is here rather than in a notebook. A reduced rule that beats the published one means the term it dropped is costing accuracy rather than buying it — and on this run one does: uncertainty alone. The published rule stays what the approach page documents, for a reason Honest Scope states in full: twelve wafers on a binary target are not grounds to redefine the method, and the continuous run reverses the finding. --terms uncertainty runs the reduced rule today.

Prior versus corrected reconstruction

Both wafers below are single episodes, not results. The scored table above covers every evaluation wafer; these two are drawn because the mechanism is clearest on one and weakest on the other, and each image states the criterion that picked it. The pair is published together on purpose — one flattering wafer on its own is a claim nobody can check.

Run benchmark to generate results

Prior → reconstruction → ground truth — The wafer whose prior error the measurements reduced most: biased prior, NANO reconstruction, and hidden ground truth on a common scale.

Expected at: docs/public/results/wafer_comparison.webp, produced by scripts/plot_wafer_comparison. Nothing is drawn until the real figure exists.

Run benchmark to generate results

The same three panels, weakest wafer — The wafer where choosing what to measure earned the least over Random and Grid. Nobody selected it; it is whichever wafer the benchmark scored worst, and it is here so the figure above cannot be read as typical.

Expected at: docs/public/results/wafer_comparison_weakest.webp, produced by python -m nano.benchmark. Nothing is drawn until the real figure exists.

Which wafer each figure shows is recorded in results/benchmark_summary.json under figures, together with the rule that chose it, so the caption cannot drift from the image. --figure-wafer overrides the first one; the weakest wafer is deliberately not overridable.

Uncertainty before and after

Run benchmark to generate results

Uncertainty map, start versus end of budget — Where the agent knew it was blind at the start, and what remained unknown after the budget was spent.

Expected at: docs/public/results/uncertainty_before_after.webp, produced by scripts/plot_uncertainty. Nothing is drawn until the real figure exists.

Uncertainty and prediction are drawn on deliberately different colour scales. They answer different questions and should never be read off the same legend.

Per-wafer improvement

Run benchmark to generate results

Paired improvement distribution — Per-wafer difference between NANO and each baseline, so wins and losses are both visible rather than averaged away.

Expected at: docs/public/results/paired_improvement.svg, produced by scripts/plot_paired_improvement. Nothing is drawn until the real figure exists.

A mean improvement can hide a rule that helps most wafers a little and hurts a few a lot. The paired distribution is the figure that would expose that, which is why it is here rather than a single bar.

Does the uncertainty map rank anything?

NANO ranks locations by uncertainty, so the map has to order the dies by how wrong the estimate is there — otherwise the acquisition rule is sorting on noise. The run checks it directly: on the dies the agent never measured, bin by reported uncertainty and report the error actually made.

Run benchmark to generate results

Reported uncertainty against the error actually made — Equal-count bins over the unmeasured dies at the end of the budget, with the Spearman rank correlation printed.

Expected at: docs/public/results/calibration.svg, produced by scripts/plot_calibration. Nothing is drawn until the real figure exists.

Measured dies are excluded on purpose. They have zero uncertainty and zero error by construction, so including them would manufacture a correlation out of the tool's own bookkeeping.

This is a ranking check, not a calibration claim

A rising curve says uncertainty orders the dies usefully. It does not say the values are probabilities, and nothing here evaluates whether a stated interval covers the truth at its stated rate — no interval is stated. The results file names the field rank_correlation for that reason. See Honest Scope.

Where NANO fails

Every pattern class in the evaluation set is scored, including the ones that go the wrong way, and this is where they are rather than in a footnote.

Final Balanced MAE per failure pattern, worst margin first. Each row is the mean over that class's episodes; the margin is against whichever baseline did better on that class.

Failure pattern	Episodes	NANO	Random	Grid	Margin over the better baseline
Random	5	0.4483	0.4508	0.4444	−0.0040
Donut	10	0.3385	0.3504	0.3468	+0.0083
Center	10	0.2934	0.3044	0.3077	+0.0109
Scratch	5	0.3983	0.4100	0.4124	+0.0117
Near-full	5	0.2939	0.3706	0.3073	+0.0134
Edge-Loc	10	0.3867	0.4268	0.4118	+0.0252
Edge-Ring	10	0.3351	0.3951	0.3876	+0.0525
Loc	5	0.3266	0.3899	0.3854	+0.0588

The margin column is a difference of means, not a separated result: the paired bootstrap on this page runs over all episodes, and a single pattern class holds too few to carry an interval. A row at or below zero is a class where choosing what to measure bought nothing.

Three things in this run deserve more attention than the headline number:

• Random defects. On the one wafer whose failures have no spatial structure, NANO ends behind Grid. A rule that ranks locations by coverage and prior disagreement has nothing to exploit when failures are scattered independently, and that is the honest shape of the claim: choosing where to measure helps where reality has structure to find.
• Plain MAE. The advantage over Random and Grid does not survive the switch away from the balanced metric — both intervals span zero, and the table at the top of this page says so. What balancing rewards is getting the rare failing dies right, which is what the budget is being spent on.
• Seed spread. On the Near-full class the seed-to-seed spread is wider than the margin over the baselines, so that row's mean is not a conclusion. Five episodes is five episodes.

The wafer where selection earned least is published as a figure above. The per-episode records behind this table are in episodes, one entry per (wafer, seed), and the per-pattern means come from strategies.*.by_pattern.

Results file schema

results/benchmark_summary.json is the single source of truth. The site validates it on build and fails loudly on a malformed file rather than rendering something plausible.

{
  "schema_version": 1,
  "generated_at": "2026-01-01T00:00:00Z",
  "git_commit": "<short sha of the run>",
  "dataset": {
    "name": "WM-811K",
    "subset_file": "<path to the versioned subset index>",
    "n_wafers": 0
  },
  "experiment": {
    "seeds": [0, 1, 2],
    "initial_measurements": 0,
    "measurement_budget": 0,
    "grid_shape": [0, 0]
  },
  "metric": {
    "name": "MAE",
    "direction": "lower_is_better",
    "unit": "failure probability"
  },
  "prior": { "initial_error": { "mean": 0.0, "std": 0.0 } },
  "strategies": {
    "nano": {
      "label": "NANO",
      "description": "highest acquisition score",
      "final": { "mean": 0.0, "std": 0.0 },
      "curve": [{ "measurements": 0, "mean": 0.0, "std": 0.0 }]
    },
    "random": {
      "label": "Random",
      "description": "uniform over unmeasured dies",
      "final": { "mean": 0.0, "std": 0.0 },
      "curve": []
    },
    "grid": {
      "label": "Grid",
      "description": "spatially uniform lattice",
      "final": { "mean": 0.0, "std": 0.0 },
      "curve": []
    }
  },
  "assets": { "error_curve": "results/error_curve.svg" }
}

Field	Required	Notes
schema_version	yes	Build fails without it
strategies	yes	Keys nano, random, grid get dedicated colours and line styles
strategies.*.final	yes	Drives the table and the home-page metric cards
strategies.*.curve	for the chart	Omit it and the chart shows its pending state; the table still renders
metric.direction	recommended	Controls the ↓ / ↑ marker printed next to every metric
prior.initial_error	recommended	Lets the site report how much prior error the correction removed
dataset.name	recommended	Printed under the table as the wafer source
dataset.note	when not WM-811K	A stand-in run writes a note here and the table prints it prominently
metric.key	recommended	Which entry of metrics leads the table, the curve and every comparison
metrics	for the second table	Every metric the run scored, with its name and scope
strategies.*.metrics	for the second table	Per-arm values for each of those metrics
references	recommended	The measurement-free rows: prior and constant
comparisons	for the intervals	comparisons[metric][arm] — paired mean, 95% interval, win count and separates

The file the harness writes carries more than the minimum: experiment.acquisition_rule and experiment.prior_bias and experiment.model record the rule and the parameters that produced the numbers, strategies.*.by_pattern breaks results down by failure pattern, and episodes lists every (wafer, seed) pair so per-episode wins and losses stay recoverable from the published file.

Figures referenced by ResultAsset are read from docs/public/results/. The sync script copies them there from the canonical results/ directory — see Reproduce.

Next: walk through a single run →