Amazon A/B testing: what it is and how it works

An A/B testing (or Split testing), in statistics, is a form of a controlled experiment that compares two versions – version A and version B. It is a widely used test in online marketing because it can capture trends of a specific KPI: in this sector are often used two different versions of the same content (for example, a landing page or a paid campaign) each one to a single group of users.

What Amazon A/B tests are and how they work

Manage your trials is an Amazon feature that allows Sellers and Vendors to perform A/B tests on the content of their brands’ catalogues. Doing A/B tests allows you to compare two versions of content with each other, in order to identify which one gets the preference of users. Creating and using a more engaging content based on user expectations and preferences contributes significantly to the increase in conversion rates.

During the test period, users landing on the product page are randomly divided into two groups: one group displays the A version of the content, while the other displays the B version.

The contents of the test do not alternate over time: this means that during the test the two versions of the content are always shown to different groups of customers.

Customers of the “experimental” group will see the content in each available organic placement (e.g. a title of the experimental product will be shown in the search results, ASIN product page, shopping cart and payment page).

What content can A/B be tested on

There are five contents on which brand owners can experiment: images, titles, bullet points, descriptions and A+ content of products.

Users rely on the titles and on the main image to decide whether or not to visit a specific product page: in this regard, performing A/B tests on these two elements allows brand owners to optimize them and thus increase traffic to the product itself.

In the same vein, when browsing the product page, customers rely on the main images and A+ content associated with the article to decide whether or not to purchase: for this purpose, testing these two elements can help to determine which content can record a higher conversion rate.

Finally, bullet points and product descriptions help to highlight product features and benefits, so they can help determine what content helps customers make more informed purchasing decisions.

Eligibility

To perform A/B tests on Amazon, the merchant must:

1. Be registered in the Amazon Brand Register;
2. Have at least one eligible ASIN with sufficient traffic in the previous weeks.

If an ASIN is not eligible because it does not have high traffic, it is advisable to increase traffic using advertising or other means.

Witailer Tips

• Make sure that the “test” content is consistent with Amazon guidelines and different from the existing one: a version that is too similar may have little impact on user behavior, and consequently the test may not get a clear winner;
• Leave the test in action for all its duration, even if the first results are not promising: stopping a test in advance based on partial results increases the possibility of overestimating the impact of the test or even selecting the wrong version;
• Do not check results too often before the end of the test: provisional results can be misleading.
• Perform other tests to constantly improve the content strategy (e.g. testing different AINs, modifying the content assumptions for the same ASIN, and carrying out tests during different seasonal periods): understanding consumers’ needs is the key to positioning as the first player in the reference category.

Case study: A/B tests on main images

We performed an A/B test on the main image of a famous brand of personal care & hygiene products, in particular on a pack of 3 bottles of shampoo.

Specifically, a second version of the main image was created, adding some graphic details that highlighted specific distinctive characteristics of the type of product offered. The test lasted 10 weeks and produced the following results:

Contact us for further information!