Interpreting test results
When you have tests running and you open Neat A/B Testing in your Shopify admin console, you will see a listing for each of your running tests like this:
- View level gives you the option to view your results by visit or visitor. Visit is the default view and is usually the best. It shows every impression, view, add to cart, and sale without limit. Visitor will only show the first impression, view, and add to cart for a site user. For example, if the same user views a product three times, you will see three views in the visit view level and one in visitor.
- Denominator gives you the option to calculate your conversion rates and success based on either impressions or views. Usually you will want to leave this set at impressions, but if you are sending traffic directly to the product page you are testing (you have more views than impressions), select views.
- Success metric lets you choose which metric determines the success of your test. You can choose from views, add to carts or sales. When you select a success metric the text at the bottom (12) will change to reflect the chosen metric. You should be looking at all the metrics when evaluating your test but, depending on your traffic and sales, it may be more practical to evaluate the success of your test based on metrics that are higher in your sales funnel, like views.
- The variant column displays each version of your product. Control is always your original product, and test is the product with the changes you made. If you're not sure which is which, just click see next to the variant name to see that version of your product.
- The Impressions column shows you how many times a user was exposed to your product. An impression is counted when a user sees your product on any page except the checkout/cart pages and the product page of the product being tested. For example, when your product shows on the homepage, in your categories, search listings or in a recommended products widget on a product page, these will all count as impressions.
- The Views column shows you how many times the product page was viewed for the product being tested. When you're testing primary image, title, or price, views is a great success metric to use because you can see if your test version is enticing users to click on your products more. This metric is not always applicable if you are only changing the description because the description does not usually show up on product pages.
- The Add to carts column shows you many times each variant was added to a user's shopping cart.
- The Sales column shows you how many times each variant was sold.
- The product row shows you all the statistics for each test variant.
- The conversion rate is shown in parentheses after each metric. Conversion rate is calculated by dividing the metric by number of product impressions. For this example 39.45% of the time a user saw the control product variant, they clicked on it.
- Clicking the Set as winner button will stop the test and set the changes in your shop so that variant you chose is the only one that shows up for all visitors from now on.
- The winning variant (based on the success metric you have chosen) will be highlighted in green.
- Here is a summary of the results of your test. It shows you the lift between the two variants, the certainty we have about the results and the change you can expect to see based on your current traffic. When we say we are 79% sure, that means that there is a 79% chance that the results you are seeing are due to the changes you made. There is a 21% chance that the change is just random chance. the longer your test runs, the higher this number will get. Professionals typically aim for above 90% certainty before ending a test, but we will talk about some instances where you may not need to wait that long.