Schedule a call

The Value of AB Testing UX Designs

In this article

Should the landing page include a hero image? Would a menu bar on the app’s homepage lead to more clicks? Would having a more engaging copy generate more sign-ups?

These are questions routinely asked by UI/UX designers when creating a design. Although UX research and user testing allow for broad insight into aspects such as the user’s ability to navigate through an app or their thoughts on an interface, it can be difficult and costly to constantly test smaller changes when trying to find a solution or after the research has concluded.

This is where AB testing comes into play. The method has been gaining popularity in recent years, as online services and websites become more and more essential to companies. Google and Facebook have also further popularised the methodology, with Google taking it the extra mile by testing over 40 different shades of blue to see which ones people like the most.

Source: UX Booth

So what exactly is AB testing? Is it just mind-numbing data tracking and tedious, complex calculations? Keep on reading as we explain the basics of AB testing, its importance, and how even you can utilize AB testing for your works.

What is AB Testing?

AB testing, or sometimes also referred to as split testing, is similar to an experiment, where two versions of a design such as a website homepage or an app screen are compared side-by-side to determine which one is more successful. To ensure fairness, both versions are made available at the same time and exposed to the same target audience, with inbound traffic split between each. A portion of the audience will be directed to the first version and the rest to the second one.

The test measures the design which is more successful based on a certain set of criteria such as conversion rate or bounce rate for a webpage. The criteria set is based on what the test is trying to achieve, for example, if a company wants to use AB testing to see if a more prominent call-to-action (CTA) button can lead to more users clicking on it, they can measure the click rate between different designs.

Source: Optimizely

So why is AB testing so important? Won’t having proper UX research cover the same thing by ensuring that the initial UX designs made are optimal? Let’s look at the value that proper AB testing can bring to answer these questions.

The Importance of AB Testing

AB testing is a great UX design strategy as it provides people with valuable data on whether certain features or designs have an effect on the users, in order to create a better user experience journey. It forces designers to look past their own biases, preferences, and ego, and instead focus solely on the results and behaviour of the users.

The data obtained from each group between the versions allows companies to make informed decisions backed by evidence. It allows companies to test out a new design approach or change, before fully committing to it. AB testing is also versatile in the sense that it can be applied to many different aspects such as marketing, user experience, and design.

Source: Medium

However, what makes AB testing so useful compared to UX user testing is that it allows companies to continually improve a website or application that has already been developed, without going through the research and testing process again. Furthermore, it is expensive and difficult for companies to conduct a proper, full-scale user test for a large number of users, thus AB testing is a viable user-centric alternative, as it allows for testing with real users in an organic setting.

Example of Amazon doing an AB test with vehicle colors. Source: Medium

Benefits of AB Testing

1. Better user experience

Besides user research and testing, AB testing can further help to figure out the most effective solution by using user’s response to different designs. The data collected through AB testing helps in making better decisions that would improve the user’s experiences.

2. Better return on investment (ROI)

An improvement in a customer’s user experience journey also correlates to better ROI as key metrics such as bounce rate and conversion rate improves as they are able to navigate easier or understand the value of the company better. Constant re-iteration and updates through AB testing ensures a design’s effectiveness.

3. Quick and easy

AB testing is fairly straightforward as determining the results is a matter of comparing both page’s metrics against the set goal e.g. bounce rate. The tests themselves do not take long to set up as they can be implemented through an app or webpage that is already live. Furthermore, even a small sample size from AB tests is able to provide effective, actionable results. There are also many easy-to-use, accessible AB testing platforms available on the market such as ABTasty.

Here are some examples of situations where AB testing is used to improve the user experience strategy.

Source: Hubspot

In this example, a digital agency is testing two different variations of their homepage copy to see which one works better in decreasing their bounce rate. In version 2, they decided to make their copy less vague in terms of what they are offering to their customers. They discovered that version 2 had an increase in click-through rate compared to version 1.

Source: crazyegg

In the next example, the company, WallMonkeys, wanted to test the optimisation of its homepage conversions by changing the image to be more relatable and positioning its search bar and copy more prominently. Through an AB test, they found that the second version had a higher conversation rate. These examples of how real companies make use of AB testing in their UX interface design to obtain meaningful insights and measurable results.

AB Testing Methodology

Here is a guideline you can follow on how to use AB testing on your designs, in order to create a good user experience for your users.

1. Identify your goal

You should have a clear objective for your AB tests, such as increasing the homepage conversion rate or increasing time spent on a certain page. Having a goal helps you to establish which metrics to measure and how to create your different designs to test against one another.

Work Illustrations by Storyset

2. Formulate a hypothesis

A hypothesis acts as the basis for the AB test as it allows testers to focus on certain factors to test and compare the different designs. To generate a valid hypothesis, one should consider why and how a certain design variation would be more effective. An example hypothesis could be: Changing the call-to-action (CTA) button text from “Download” to “Download Career Guide” would increase downloads as users are more clear on what they would receive.

3. Set testing criterias

Identify the key metric to track in line with your set hypothesis and goals. For example, when looking to test the effectiveness of a different copy, you will probably be tracking bounce rate or time on page. However, when tracking effectiveness of a CTA button text, tracking button clicks may be more relevant.

Data illustrations by Storyset

4. Design and carry out the test

Create two variations of your design with your AB testing tool. Be sure to have a control design and refrain from changing too many factors when starting out as it can be difficult to pinpoint what affected the metrics.

5. Analyze and implement

Lastly, carry out and run the test over a set period of time. The timeframe required is dependent on your site’s traffic. Once completed, collect the data, analyse your results, and implement the necessary changes. Do note that having a larger sample size is more representative of the users.

Business illustrations by Storyset

AB Testing Tips

Now that you have learned the basics of AB testing, here are some additional key tips to help bring your AB test even further!

Multiple variant testing

When running a standard AB test between two variants, your observations and results are usually limited to whether your new variant does better or worse than the original. However, with multiple variant testing, you can test more than two designs at any time, simultaneously testing multiple hypotheses.

Source: Cortex

For example, rather than just testing if an image of a person would do better compared to an image of a product on the landing page, you can take it further and additionally see how images of a person using the product or a person smiling might perform. Do note that multiple variant testing requires sufficient traffic to obtain effective results, thus would usually take a longer time.

Start simple

When first starting out an AB test, it can be difficult to pinpoint areas that can affect the user’s behaviour drastically. Our tip is to start simply with tweaking copy, images, and colour (low-hanging fruits). You can then further adapt from the results received and adjust accordingly.

Source: adoric

Effectiveness with higher traffic

A/B testing is faster and more effective if you have higher traffic — ideally more than 1000 unique visitors a month. The amount of traffic on the pages you are AB testing for is important to obtain better results and maximise your returns.

For sites with lower traffic, a tip could be to test things with a higher impact instead, in order to maximise the results and obtain actionable insights. When testing an aspect with a higher impact, the difference in results can reduce the time taken to get statistically significant results. Additionally, you should also avoid multiple variant testing with limited resources.

Calculating statistical significance

This brings us to our next point, calculating statistical significance. Statistical significance is important as it proves the level of certainty of the test results, making sure that they are not due to sampling errors or random chance.

For example, if your test has a 97% significance level, you can be 97% confident that the differences in your results are from the changes you have made.

Ideally, your AB tests should reach 95% statistical significance, or at the very least 90%, as this ensures that the changes made will impact the site’s performance when implemented.

Be sure to calculate the statistical significance of your test using a calculator such as this by SurveyMonkey.

Data illustrations by Storyset

Popular software

Last but not least, some popular software that you can consider when AB testing includes:


I hope that this article has provided you with some insight into why AB testing UX designs and why it is so useful in creating good UX designs as well as a brief idea on how to establish an AB test. If you want to know more about user experience and its industry, do check out our other resources available on our website, such as our articles, weekly webinars, and podcasts.

CuriousCore offers both a 2-day UX course and an intensive 4-month career accelerator course for those keen on transitioning into the industry. Click the buttons below to find out more.