Comparing runs and confidence intervals

When two runs can be compared, and what “within sampling error” means.

On this page

After an optimization, the obvious question is: "the number moved — did it really change?" AI answers vary on their own, and the measurement scope can shift between runs. When PeakGEO compares two runs, it only pairs answers collected under the same conditions and uses confidence intervals to decide whether a difference goes beyond normal fluctuation. This page explains the rules and how to read the results.

What can be compared

The smallest unit of comparison is a comparable cell: the same question, the same model, the same question type and the same sampling conditions (such as region and collection method). Only when all of these match are answers from two runs compared.

A cell becomes non-comparable when:

the question text was edited, or replaced by a newer version from the question library;
a different model was used, or a model is being sampled for the first time;
sampling conditions differ;
in one of the runs the cell has no valid answer (failure, timeout or empty answer).

Paired cells

Among comparable cells, only those with valid results in every run being compared enter the comparison — these are paired cells. This prevents fake changes such as "this run happened to include more questions where the brand is easy to mention".

In Retest comparison and between monitoring runs, every repeated sample inside a paired cell counts.
The Impact comparison in Visibility also balances repeat counts so both sides have the same number of valid answers.

The console shows both what was included and what was left out, for example "3 matched cells · 12 valid answers on each side" and "Excluded: 2 before and 4 after". If only some cells are comparable, the change applies to those cells only — a change in sample scope must not be read as an optimization effect.

How a change is judged

Each side's rate is recalculated from the paired answers, with a 95% confidence interval (see Metric definitions). The result is then determined in this order:

Result	Condition
Can't compare yet	Either side has no comparable valid answers
Unchanged	Both rates are exactly the same
Within sampling error	The rates differ, but the confidence intervals overlap
Improved	The intervals don't overlap and the change goes in the better direction
Worsened	The intervals don't overlap and the change goes in the worse direction

"Better" depends on the metric: higher is better for mention and recommendation rates, while for a metric such as the verified fact-error share, lower is better.

Example

On one model, 3 of 20 comparable answers recommended your brand before the optimization (15%), and 8 of 20 after (40%):

The intervals are about 5.2%–36.0% before and 21.9%–61.3% after;
They overlap, so the result is Within sampling error, even though the point estimates are 25 percentage points apart.

With 40 answers on each side at 30% (12) and 70% (28), the intervals are about 18.1%–45.4% and 54.6%–81.9%. They no longer overlap, so the result is Improved.

To make differences easier to tell apart, increase "Repeats per question" in the monitoring plan or widen the question scope — see Monitoring, retests and alerts.

Where to compare

Visibility › Impact comparison: compare by "Measurements", "Periods" or "Optimization event", and break the change down by model and topic. Question-by-question comparison lets you open the answers on each side.
Retest & monitoring › Retest comparison: places the first diagnosis next to every retest. Each metric is marked Improved, Worsened, Unchanged, Within sampling error or Can't compare yet, with an overall verdict: Improved, Not improved yet, No clear change (normal fluctuation), Mixed or Can't tell yet.

When both sides have data but question versions, models or conditions don't match, the console says plainly that no reliable change can be given instead of forcing a difference.

Publication records and first-cited time

Once you register a published link in Content — or add one manually with "Record published content" in Visibility — the record appears under "Optimizations and observed impact":

First cited / detected: the time of the first valid answer after publication that actually cites this URL. If no answer has cited it yet, it shows "Not detected yet"; the publication time is never used as a substitute.
Page citations: before → after: counted by exact URL, not by the whole domain.
When comparing by "Optimization event", the action's recorded baseline is used when available, otherwise the 14 days before. The 14 days after are then observed, and results show as they arrive.

Change is not causation

Comparisons show observed changes. They do not establish that a particular optimization caused them.

Even when intervals don't overlap and a result is marked Improved, it only means that within the comparable scope there was a difference beyond normal fluctuation. Model updates, content changes on other sites and competitor activity can all move the numbers. In the same way, an article being detected in citations does not prove it raised your recommendation rate. PeakGEO does not promise that any optimization will lead to an improvement.