Significance threshold
The significance threshold is the confidence level a result has to reach before it counts as conclusive.
The significance threshold is the confidence level you set to determine when an experiment has enough statistical evidence to identify a winner. It's one of four requirements an experiment must meet before Elevate A/B Testing considers the result conclusive.
To understand the concept behind this setting, read our documentation on statistical significance.
What can go wrong on Shopify store?
- Lowering the threshold after seeing the results. Changing your significance threshold because a variation looks promising can lead to biased decisions. Set your threshold before starting the experiment and keep it consistent throughout the test.
- Setting the threshold too high. A very strict threshold can make it difficult for an experiment to reach a conclusive result, especially if your store has limited traffic. Choose a realistic level based on your store's traffic and testing goals.
- Relying on the threshold alone. Reaching your significance threshold doesn't automatically mean the results are reliable. You also need enough visitors, conversions, and running days to make an informed decision.
- Choosing the wrong goal metric. The significance threshold tells you how confident you can be in the result, but your goal metric determines what you're measuring. Make sure the metric matches what you want to improve, whether that's conversion rate, revenue per visitor, or another outcome.
- Using different thresholds for every experiment. Constantly changing your significance requirements makes it harder to compare results across tests. Establish a standard threshold and adjust it only when there's a clear reason.
- Ignoring other changes during the experiment. Theme updates, overlapping tests, or other store changes can affect your results. Even a high significance threshold can't guarantee reliable conclusions if the experiment wasn't run under consistent conditions.


