A/B testing glossary
Plain-English definitions of the A/B testing, conversion and statistics terms behind every Shopify experiment.
36 terms
A
4 terms
A/B testing
Showing two versions of a page or offer to separate groups of visitors at the same time, then keeping the version that earns more.
A/B/n testing
An A/B test with more than one variant, so the control competes against two or more alternatives at the same time.
Add-to-cart rate
The share of visitors or sessions in which a shopper adds at least one product to the cart.
Average order value (AOV)
Total revenue divided by the number of orders over the same period.
B
1 term
C
4 terms
Cart abandonment rate
The share of shoppers who add products to their cart but leave without completing checkout.
Control group
The visitors who see your store exactly as it is today, used as the baseline every variant is measured against.
Conversion rate
The share of visitors who complete a goal, usually a purchase: conversions divided by visitors, multiplied by 100.
Conversion rate optimization (CRO)
The practice of raising the share of visitors who buy through research, testing and iteration, without buying more traffic.
F
2 terms
H
3 terms
Headless commerce
Running a store with a custom storefront that is separate from the commerce platform behind it, connected through APIs.
Heatmap
A visual overlay that shows where visitors click, move and scroll on a page, with warm colors for the most activity.
Hypothesis
A testable prediction that states what you will change, for whom, and which metric you expect to move.
L
2 terms
M
3 terms
Micro conversion
A smaller step toward a purchase, such as an add-to-cart or an email signup, that a single page can realistically own.
Minimum detectable effect (MDE)
The smallest change in a metric that a test is designed to detect reliably.
Multivariate testing
Testing several changes on one page at once to find which combination of elements performs best.
N
1 term
P
6 terms
P-value
The probability of seeing a difference at least this large if the change you tested had no real effect.
Peeking
Checking a test while it runs and stopping it as soon as one version looks ahead, which inflates the chance of a false win.
Price elasticity of demand
How strongly the quantity people buy responds to a change in price.
Price testing
Showing different prices for the same product to comparable groups of shoppers to find the price that earns the most.
Probability to win
How likely a variation is to be the best performer on your goal metric, given the data collected so far.
Profit per visitor
Gross profit after product costs, shipping and fees, divided by unique visitors.
R
1 term
S
5 terms
Sample size
The number of visitors or orders each variant needs before a test result can be trusted.
Social proof
Signals that other people bought and liked a product, such as reviews, ratings, testimonials and user-generated photos.
Split URL testing
An A/B test where each version lives at its own URL, used for redesigns and pages too different to build as edits of one page.
Statistical power
The probability that a test detects a real effect of a given size when one exists.
Statistical significance
The point at which the gap between a variant and the control is too large to be explained by random chance alone.
T
2 terms
U
1 term
V
1 term
