{"id":1005534,"date":"2026-04-28T13:43:38","date_gmt":"2026-04-28T11:43:38","guid":{"rendered":"https:\/\/www.iterates.be\/?p=1005534"},"modified":"2026-06-11T16:11:37","modified_gmt":"2026-06-11T14:11:37","slug":"the-7-pitfalls-that-invalidate-your-a-b-tests-and-how-to-avoid-them","status":"publish","type":"post","link":"https:\/\/www.iterates.be\/en\/the-7-pitfalls-that-invalidate-your-a-b-tests-and-how-to-avoid-them\/","title":{"rendered":"A\/B testing: the 7 pitfalls that invalidate your tests"},"content":{"rendered":"
\n

L\u2019A\/B testing<\/strong> is reassuring. We launch two versions, wait for the figures, then choose the winner. Scientific. Data-driven<\/em>. Objective. Except that, in practice, the majority of A\/B tests are invalid not because the teams are incompetent, but because very specific errors creep into the process. The result: bad decisions made with false confidence. Here are the seven most common pitfalls, and how to avoid them in practice.<\/p>\n\n\n\n

A\/B test design: mistakes that skew your results before you even start<\/h2>\n\n\n\n

Even before looking at your data, the damage is often already done. Two structural errors can invalidate a test as soon as it is set up.<\/p>\n\n\n\n

Stop the test too soon (Peek & Stop)<\/h3>\n\n\n\n

You launch your test on a Monday. On Wednesday dashboard<\/strong> posted +14 % in conversions with a p-value<\/strong> of 0.03. You stop and deploy. Classic mistake.<\/p>\n\n\n\n

During a test, results naturally fluctuate. If you check frequently and stop as soon as the threshold is crossed, your false positive rate<\/strong> (errors in type I<\/strong>) is exploding. Simulations show that this rate can exceed 50 % - even with a theoretical threshold of 95 %.<\/p>\n\n\n\n

The solution: before launching the test, define the duration of the test and the sample size<\/strong> then stick to it. If you need to keep an eye on things along the way, use suitable methods such as the Sequential Testing<\/strong> or the\u2019Always Valid Inference<\/strong>.<\/p>\n\n\n\n

Testing too many variants simultaneously<\/h3>\n\n\n\n

Testing five variants at the same time seems effective. In reality, each additional comparison creates its own probability of false positive<\/strong>. With five comparisons at p < 0.05, the probability of obtaining at least one false positive exceeds 22 %. This is the problem with multiple comparisons<\/strong> also known as Type I error rate<\/strong>.<\/p>\n\n\n\n

The solution: test one or two variants at a time if your traffic is limited. If you test several, apply a statistical correction such as the Bonferroni correction<\/strong> or the Benjamini-Hochberg<\/strong>. And above all: formulate a clear hypothesis before<\/em> to test. A test without a hypothesis is exploration disguised as experimentation.<\/p>\n\n\n\n

Statistical traps that inflate your results<\/h2>\n\n\n\n

Even with a well-designed test, statistical errors can lead you to believe in gains that don't exist, or miss those that do.<\/p>\n\n\n\n

A sample that is too small or miscalculated<\/h3>\n\n\n\n

La statistical power<\/strong> measures the probability of detecting a real effect if it exists. A test with 50 % of power misses half the real effects - it's a coin toss with an extra step.<\/p>\n\n\n\n

To calculate the right sample size, you need three parameters: the base rate<\/strong> of your metric, the Minimum Detectable Effect (MDE)<\/strong> and the power level<\/strong> (generally 80 %). An undersized test has two perverse effects: it misses the real positive effects (false negatives<\/strong>), and when it detects something, the estimated effect is often inflated. winner's curse<\/strong>.<\/p>\n\n\n\n

Confusing statistical significance with practical importance<\/h3>\n\n\n\n

The solution: calculate the sample size before<\/em> to launch, with tools such as Evan Miller's calculator or the Python library statsmodels<\/code>.<\/p>\n\n\n\n

A result can be statistically significant<\/strong> and practically useless. p = 0.01 with a gain of +0.3 % in conversion rate: is this worth the development cost and the associated technical debt? Probably not.<\/p>\n\n\n\n

La statistical significance<\/strong> answers : \u00abIs this effect due to chance? It does not answer: \u00bbIs this effect big enough to matter?\u00ab To do this, use\u00bbeffect size<\/strong> Cohen's d<\/strong>, relative lift<\/strong>, The absolute impact projected onto your user base. Define a Minimum Business Impact<\/strong> What is the smallest result that would justify implementing the change? It's this bar that should guide your decisions, not just the p-value<\/strong>.<\/p>\n\n\n\n

3. Behavioural and measurement biases that lead to misinterpretation<\/h2>\n\n\n\n

A test can be technically valid and statistically correct and still mislead you. This is where user bias and metrics come in.<\/p>\n\n\n\n

The novelty effect and Sample Ratio Mismatch<\/h3>\n\n\n\n

Two distinct phenomena can corrupt your conclusions at this stage. The first is the\u2019novelty effect<\/strong> a variant performs better simply because it is different<\/em>. Users explore out of curiosity and click more, but this effect is transitory. If you measure too early, your apparent gain will disappear once the novelty wears off. Conversely\u2019Hawthorne effect<\/strong> causes users who feel they are being watched to behave differently. To distinguish a real gain from an artefact, monitor the evolution of the metric over time and wait at least two weeks before concluding on a major interface change.<\/p>\n\n\n\n

The second phenomenon is the Sample Ratio Mismatch (SRM)<\/strong> The ratio of users between the variants does not correspond to the expected ratio. If your test is designed on a 50\/50 basis and you obtain 52\/48, all your conclusions are invalid. Before any analysis, carry out a chi-square test<\/strong> on your group sizes. Common causes include tracking<\/strong>, s, misconfigured redirects and bots<\/strong> filtered after assignment, or a cross-contamination<\/strong> between groups.<\/p>\n\n\n\n

Poorly chosen metrics<\/h3>\n\n\n\n

This is the most strategic trap. A perfectly executed test can lead to the wrong decision if you measure the wrong thing. Classic example: you optimise the click-through rate<\/strong> on an add to basket button. It goes up. But the purchase completion rate<\/strong> fall. You've created friction downstream without realising it.<\/p>\n\n\n\n

All good tests must include guardrail metrics<\/strong>, These are metrics that you're not trying to improve, but that you don't want to deteriorate: revenue per user, retention rate<\/strong>, NPS, loading time. If a guardrail metric deteriorates, the \u00abwinning\u00bb variant on the main metric<\/strong> may not be a victory. Before you launch, ask yourself this question: \u00abIf this metric increases by X %, am I sure that's good for business?\u00bb If the answer is \u00abnot necessarily\u00bb, look for a better metric.<\/p>\n\n\n\n

<\/p>\n<\/div>","protected":false},"excerpt":{"rendered":"

There's something reassuring about A\/B testing. You launch two versions, wait for the figures, then choose the winner. Scientific. Data-driven. Objective. Except that, in practice, the majority of A\/B tests are invalid - not because the teams are incompetent, but because very specific errors creep into the process. Results<\/p>","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[1238,1245,1239],"tags":[],"class_list":["post-1005534","post","type-post","status-publish","format-standard","hentry","category-developpement-dapplication","category-developpement-de-logiciel","category-developpement-web"],"acf":[],"aioseo_notices":[],"aioseo_head":"\n\t\t\n\t\n\t\n\t\n\t\n\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t\n\t\t