Sample ratio mismatch: check allocation before outcomes

What does a sample ratio mismatch check ask?

An experiment may have two variants without receiving exactly equal counts. The useful question is not whether their counts differ at all. It is whether the supplied allocation counts cross a preselected statistical discrepancy threshold under the experiment's stated assignment model. Sample size matters alongside the apparent split.

Microsoft Research describes sample ratio mismatch detection separately from finding its cause. Its explanation uses configured versus counted variant populations before interpreting experiment outcomes. That is the context for this exercise, not evidence that a particular advertising account or website has been tested.

We will use a wholly fictional card with no sales, engagement or conversion outcomes. Your deliverable is a saved allocation-check note containing the assumptions, expected counts, two calculation rows, a table lookup and a limited finding. You will not choose a winner, diagnose a logging defect or make a live launch decision.

The distinction matters for a small team reviewing a report. A striking count imbalance can invite an immediate story about what happened. This worksheet stops earlier: preserve the stated model, calculate the discrepancy and retain the questions that actual measurement owners would need to answer. A flag is not an explanation of its cause.

Write the assignment contract before calculating

The synthetic card contains one hundred unique eligible assignment units. Each appears once in exactly one of A or B, and all one hundred are included. No unit is selected afterward because it converted, viewed a page or produced an outcome. These conditions are stipulated teaching inputs, not facts demonstrated by two aggregate totals.

Call the comparison model H0, or the null model. It specifies independent assignments with a fixed probability of one half for A and one half for B after every previous assignment history. The planned probabilities do not change. We do not estimate them from the sixty-five and thirty-five counts below.

The contract permits one prespecified check after one hundred units. It supplies alpha=0.01 in advance as the significance level used to choose a reference cutoff. Alpha is not the probability that the dataset is wrong or that H0 is true. This is not a procedure for repeatedly peeking at a live dashboard until a flag appears.

The observed-role counts are A=65 and B=35. They are deliberately invented, not collected in an experiment. Put that sentence beside the input. “Observed” labels the count's role in this hypothetical calculation; it must not turn into a claim about actual customers or a randomisation process we ran.

The segment and pooled rate audit explicitly uses A/B as table names without randomised assignment. Here assignment probabilities and assumptions are supplied separately. If a real export lacks those conditions, record the missing information instead of treating any pair of columns as this experiment.

Compute expected counts and both contributions

Start with the total: 65+35=100. Under the fixed half-and-half model, each expected count is 100×1/2=50. Expected means what the supplied model allocates on average, not a command to replace the actual-role counts. Preserve sixty-five and thirty-five unchanged in their own columns.

Pearson's chi-square goodness-of-fit statistic adds one contribution per category: squared observed-minus-expected difference divided by the expected count. NIST documents that statistic and its comparison with an upper-tail reference cutoff. In this two-arm exercise, the arithmetic needs only a basic calculator.

Calculate 65−50=15 and 35−50=−15. Retain the signs in your note. Both differences have magnitude fifteen, so each square is 15×15=225. Divide by the expected fifty: 225÷50=4.5 for each arm. Add the contributions: 4.5+4.5=9. Nine is the test statistic, not a percentage or a measure of conversion lift.

Check that the observed total is one hundred and the expected total is also one hundred. The smallest expected count is fifty. It exceeds NIST's minimum expected-frequency condition of five for this approximation. That does not authorize applying the same example to a different card with tiny or zero expected counts.

The standard deviation denominator audit also uses squared differences, but answers another question. It describes numeric spread around a mean. Here categorical assignment counts are compared with model-implied counts; the shared operation does not give the two statistics the same interpretation.

Find the correct reference cell, not a new threshold

There are two categories and no probability parameter is fitted from the observed counts. Degrees of freedom therefore equal 2−1−0=1. This number selects the reference-table row. It is neither another participant count nor an instruction to discard one of the arms.

Open NIST's critical-values page and locate the upper-tail table. For the supplied alpha of 0.01, choose the column labelled 1−alpha=0.99. In the row for one degree of freedom, the rounded critical value is 6.635. The column is labelled probability below the reference value, not the upper-tail probability itself.

Compare your statistic with that value: 9>6.635. Under this exercise's prespecified model and cutoff, flag the allocation-count discrepancy for review. The reference distribution is an approximation, not an exact finite-sample probability calculation. No p-value calculator is necessary to finish this bounded comparison.

This goodness-of-fit statistic is checked in the upper tail. Do not halve alpha merely because A and B are two arms. Squared deviations already contribute for either allocation direction. Also do not select a cutoff after seeing the result just to obtain the preferred finding.

Microsoft's opened article reports its own conservative threshold of p<0.0005. That is not our exercise's alpha=0.01. Keep the fictional choice visibly separate from the platform's stated policy. Neither should be described as a universal setting that every business must adopt. If the reference table and its attributed excerpt are unavailable, preserve the calculation and leave cutoff verification unresolved.

Keep an integrity flag separate from a business verdict

A bounded conclusion reads: “The supplied counts produce a chi-square statistic of nine, above the selected approximate cutoff of 6.635; flag the allocation discrepancy under the stated model.” Keep the model and cutoff attached to that sentence. Without them, the finding can sound like an unconditional diagnosis.

Do not replace it with “A wins,” “B lost users,” “the system is definitely broken” or “stop the campaign.” The card provides no outcome values and establishes no cause. It also does not calculate the probability that the null model is true. A nonflag in another case would not certify every aspect of measurement quality.

The conversion double-counting guide investigates particular tracking actions and transaction evidence. Aggregate assignment totals do not perform that investigation. Record an open question for a real measurement owner rather than guessing a tag defect, missing-user story or corrective account setting.

The at-least-one probability audit is different too: it calculates possible-history weights under a supplied model. This exercise instead compares supplied count roles with model-implied expected counts. Nine must not be relabelled a success probability or multiplied by one hundred as a rate.

An AI assistant is optional. The artificial intelligence starting guide emphasizes choosing a concrete task. Here you can ask an assistant to preserve the fictional contract and every arithmetic step, then compare its answer with your calculator. If no assistant was used, write that the optional comparison was not performed.

Save the calculation and its limits together

On Windows, open Start, type Notepad and launch the application. Then create a new document. Copy the assumptions and source card before adding the following calculation record. A useful note must let another person follow your work without needing a conversation transcript, an experiment account or a specialist statistical application.

Allocation review V1: wholly fictional, no experiment performed.
All100 unique eligible units counted once in A or B; no outcome filtering.
H0: independent assignments; fixedA=1/2,B=1/2; no fitted probability.
One planned check after100units; preselectedalpha0.01.
A: O65 E50; difference15; square225; contribution4.5.
B: O35 E50; difference−15; square225; contribution4.5.
Totals100/100; minimumE50; chi-square9; degrees of freedom1.
NIST upper-tail table: column0.99,row1,roundedcutoff6.635;9>6.635.
Finding: flag under this model, not a cause, winner or live action decision.
Open question: who establishes real unit uniqueness, independence and complete inclusion?

Press Ctrl+S, choose Documents and save as allocation-review-V1.txt. Close the editor. Open File Explorer, go to Documents and double-click that saved file. Recheck the sixty-five/thirty-five inputs, fifty/fifty expectations, statistic nine, row one and cutoff 6.635. The open assumption question and no-live-action boundary must survive too.

In earlier public writing about perfection and effectiveness, Anar described projects delayed while pursuing perfection. Applied here, the point is to complete one explainable note, not to claim a personal marketing experiment or a customer result. Allow three minutes for assumptions, five for arithmetic, three for lookup and interpretation, and four for saving and reopening. That fifteen-minute division is a proposed cognitive allowance, not a measured participant result.