How long does it take to run a test?
It depends. At a minimum, you should let your test run for two weeks. Depending on your traffic you may need to run your test longer. Aim for 500 product impressions for each variant at a minimum. 1,000 or higher is better. You're looking for the statistical significance of the metrics you care about to be at 90% or higher. This is shown on the dashboard below your test. If you have exposed a lot of traffic to the test and you're not seeing statistically significant results, it's possible that the result is flat - meaning that the change you made has not affected your metrics in a meaningful way. In that case, choose the version you prefer.
What if I don't have a lot of traffic?
Ultimately, you want sales. Sales = more money and more time laying on a beach somewhere. But, depending on your traffic level, sales might not be the best metric to test. Let's look at an example to see why.
Bob's shop averages 5,000 visits per month and he has a healthy conversion rate of 2%, meaning he makes about 100 sales. He wants to test a different image for his only product; Bob's Widget. He tests for a month and ends up with the following results.
If we look at the conversion rates and significance levels we get:
|Add to carts||-1%||55%|
If we only cared about sales, we could not act on this information. The significance level of the sales has not reached a point yet where we can safely draw a conclusion. In fact, without a large increase in traffic or a much larger change in sales, we would need to test a lot longer before we reached significance with sales. This is normal on a store without a huge amount of traffic. If we take a look at product views though, we see a different story. People are clicking the product with the new image 15% more and we are 99% sure that is due to the change we made. At 5,000 visits per month, we expect an additional 293 people to view this product if we make the change permanent. That's huge!
It's still important to consider the other metrics, even if we haven't reached 90% significance. In this case, the add to carts are flat. We can be relatively sure that this test has not affected add to carts. Sales are looking good. We're at +8% with 66% certainty. This means that there is a 66% chance that this test has improved our sales. This, combined with the significant increase in product views makes us confident that making the change permanent will improve our shop.