CC BY
6Оптимизация пользовательского опыта с помощью итеративного A/B-тестирования
Чумаченко Аксиния
руководитель группы аналитики продукта, Simpals, aksinia.chumachenko@999.md
В этой статье обсуждается оптимизация пользовательского опыта с помощью итеративного А/В-тестирования как средства повышения эффективности веб-страниц и пользовательских интерфейсов. А/В-тестирование, также известное как сплит-тестирование, представляет собой методологию, которая позволяет сравнивать две версии тестового объекта, чтобы определить, какая из них более эффективна для достижения ваших целей. В статье анализируется роль А/В-тестирования в современных маркетинговых стратегиях и дизайне пользовательского интерфейса и подчеркивается важность итеративного подхода в непрерывной оптимизации и адаптации к предпочтениям пользователя. Основное внимание уделяется анализу ключевых показателей, которые включают показатели конверсии и показатели вовлеченности пользователей, а также изучению эффектов внедрения изменений с помощью последовательных итераций тестирования.
Ключевые слова: пользовательский опыт, А/В-тестирование, программирование, ИТ, цифровизация.
Introduction
Two key components play a significant role in the process of adapting to change: the system and the user. This is especially evident in adaptive user interface design for mobile applications, where the desire to maximize automation of adaptation on the part of the system is critical. This means that the system or application must organize information processing in a way that simplifies its perception and makes the computing system itself as little visible to the user as possible.
The goal of user interface adaptation for mobile applications is to create an intuitive and user-friendly interface that is also able to adapt to changes in the operating environment and compensate for the limitations of mobile devices.
Interface development is included in each cycle of the iterative application development model. At each stage of this model, either new functionality is introduced with an appropriate interface or existing functionality is improved. The main objective of such enhancements is to prioritize interface properties, which include identifying critical aspects that need to be optimized. In this context, alternative approaches to interface improvement are developed, and technical requirements are formed, which are then implemented at the stages of software development and testing [1].
In turn, if we talk about the relevance of this topic, it is due to the growing competition in the online space, where the quality of user experience directly affects the retention and attraction of customers.
1. General characterization
Product analytics is an approach to product development that is based on collecting, analyzing, and interpreting data. In product analytics, a company studies how users interact with a product and assesses how conveniently and effectively it solves their problems. Product management considers all aspects of the product life cycle, from initial ideas, their evaluation, and business case, through prototyping, development, pricing, market introduction, managing market introductions, measuring performance, and ultimately withdrawal from the market. Initially, a product management organization is provided, often providing the backbone or closest thing to the customer. However, as the organization grows, a more formal stage becomes necessary to maximize the minimum p-value of the product, increase market share, create benefits, and grow competitiveness and profits. In this ninth process of product management, a structure is put in place to achieve these goals and increase the chances of success and development [2].
If we talk about product analytics, refers to a methodical approach to studying the interaction of users with the product, providing user management, product usage patterns, and the effectiveness of product features. This analysis provides valuable data that contributes to improving product development, sustainable user experience, and informing strategic decisions based on different usage patterns [3].
Product analytics enables decision-making based on objective data. Instead of basing your decisions on intuitive hunches or choices, you can make informed choices backed by reliable data. Which in turn ensures that serious errors are avoided and progress is maintained in the highest priority aspects [4].
Controlled experiments, often called A/B tests, also called A/B/n-tests (for multiple choices), split tests. Controlled experiments allow you to determine the impact of any changes from the original brand. These experiments are the simplest method to:
• Establishing cause-and-effect relationships with a certain degree of probability
• Detect even small changes
• Detecting unexpected consequences (e.g., decreased performance or increased failures)
Common mistakes
1. Insufficient number of users and statistical power: A common error is that the test does not identify enough users to detect significant changes. This leads to low statistical power and increases the likelihood of unreliable results.
2. Continuous observations of p-value: Constant observation of p-values during the experiment can cause bias and misinterpretation of the p-value. It is optimal to determine the p-value only after the completion of a predetermined period of the experiment.
3. Considering some indicators: Analyzing the number of indicators other than core indicators (OEC) and control indicators can cause bias and erroneous conclusions [5,6].
Over the past five years, the percentage of companies utilizing A/B testing has grown significantly. Today, 58% of companies use A/B testing to optimize conversion rate optimization (CRO), and 77% of organizations run A/B tests on their websites. The global A/B testing software market was p-valued at $485 million in 2023 and is projected to reach $1.08 billion by 2025 [7].
o o o
77%
Website
60%
Landing page
59%
Figure 1 - The most common areas of A/B testing [7 ]
Despite the increased investment in A/B testing, the importance of an iterative approach is often overlooked. Setting up an effective A/B test requires not only an initial run but also subsequent iterations that many specialists consistently neglect. For example, only one out of eight A/B tests results in significant change, highlighting the need to retest and optimize for better results [8].
An iterative process is critical for improving and optimizing product strategies. Microsoft Bing, for example, conducts over 1,000 A/B tests per month, which allows them to continuously improve user experience and increase revenue. In one such test, a small change in the display of ad headlines resulted in a $100 million increase in annual revenue [9].
The goal of A/B testing is to collect data on user behavior and analyze differences in key performance indicators between the two groups. These metrics can range from conversion rates to engagement levels, depending on the goals of the testing.
As part of the testing, each group is given a different version of the interface or web page. The control group works with the original version, while the experimental group has access to modifications that may include new designs, content structures, or controls (Figure 3).
Click rate: 52 %
Figure 3 - Example of A/B testing of a web page
During the testing process, user behavior is closely monitored, which includes tracking clicks, visits, and other important activities. Analyzing the collected data allows companies to determine which version performs more effectively.
The application of A/B testing covers a wide range of products, including websites, landing pages, email campaigns, and user interfaces, and promotes data-driven decision-making rather than assumptions. Effective use of this method requires strict adherence to a scientific approach, including random assignment of users to groups and statistical analysis to achieve accurate results. A/B testing is key to optimizing products and improving user experience.
Next, in Table 1, let's look at the main advantages inherent in A/B testing.
Table 1
Sales/ leads/ Total monthly Click-through Search traffic Bounce rates conversion rates visitors rates
Figure 2 - The most important web analytics metrics for websites [9]
A/B testing, also known as split testing, is a methodology used to compare two versions of web pages, user interfaces, or marketing campaigns to determine the more effective version in achieving a given goal. This method involves dividing the audience into two groups: a control group (Group A) and an experimental group (Group B).
Advantages Description
Data-Driven Decision Making A/B testing provides companies with concrete data and insights, enabling informed decisions. Unlike intuition-based or subjective opinion-based decisions, this testing allows for the assessment of actual user behavior and performance metrics, contributing to more effective strategic optimization.
Improved User Experience A/B testing identifies changes that positively impact user interaction. Experimenting with various elements such as layout, design, and content helps establish the best engaging aspects, enhancing user interaction and increasing customer satisfaction levels.
Increased Product Metrics Optimizing elements directly impact sales volume and other key business metrics.
Risk Minimization A/B testing helps minimize risks associated with implementing significant changes by testing innovations on a subset of the audience before large-scale application. This reduces the likelihood of negative consequences and allows for a more accurate assessment of the impact of new implementations.
Efficient Resource Allocation This testing method helps organizations use resources wisely, investing in the most effective changes. This approach prevents spending on ineffective initiatives and promotes more productive budget use.
Continuous Optimization Continuous testing and improvement of products create a culture of ongoing optimization in the company. This helps maintain competitiveness by adapting to changing customer preferences and market demands.
Cost- Effectiveness of Iterations A/B testing provides an opportunity for economically justified iterations, allowing changes to be tested with a small number of users to assess their effect before full-scale implementation [10].
X X O OD A C.
X
0D m
o
io 00
2 O
IO ■p»
CN
0
cs
03
01
o
HI
m
X
<
m
o
X X
2. Practices for A/B testing
To effectively apply A/B testing, there are some key guidelines to follow that will help optimize the process and improve results:
Define specific goals: It's important to define exactly what metrics or results you're looking to improve through A/B testing, such as conversion rates, user interaction rates, or click-through rates. Clearly articulating your objectives ensures that the study is focused and aligned with your company's overall strategy.
Testing one variable: To accurately assess the impact of changes, it is crucial to test one variable at a time. This allows you to accurately isolate the effects of changes and correctly attribute any differences in performance. In order to do A/B testing you need to identify your main business objectives. Then you need to identify the metric you will be looking at to see if the new version of the site is more successful than the original. After that, you need to develop a hypothesis about what exactly will change and therefore what you want to test. Next comes the preparatory phase before conducting the experiments. The next step is to conduct the experiment and then analyze the results.
Ensuring a statistically significant sample size: A sufficient sample size is necessary to obtain reliable results. The use of statistical tools and calculators can help determine the necessary sample size based on the desired level of confidence, the power of the test, the magnitude of the expected effect, and the variance of the data.
Random assignment: To minimize bias and ensure representative results, users should be randomly assigned between the control and test groups. This allows for individual differences between users and ensures the objectivity of the results.
Sufficient test duration: It is important to allow the A/B test to continue long enough to collect an adequate amount of data for reliable conclusions. Parameters such as traffic volume, expected effect size, and significance level should be considered when determining the duration of the test.
Assess statistical significance: Use statistical methods to determine the significance of the findings. Statistically significant results confirm that the observed differences in performance are unlikely to be due to random chance.
Implementation and iteration: Once the most successful option has been identified, implement it. However, A/B testing is an iterative process that requires continuous improvements and further testing for optimization [11].
In turn, the main differences between A/B/n-testing, and multivariate tests will be reflected in Table 2.
Table 2
Differences General Characteristic
A/B/n testing A/B/n testing includes controlled experiments, where the main focus is on comparing the conversion rates of the original page with one or several of its variations. This allows for determining which changes are most effective.
Multivariate Testing Multivariate tests also compare several versions of a page, but the goal here is to find out which specific attributes of the page influence the results the most. Unlike A/B/n testing, where changes may be limited to one or two elements, multivariate tests explore the impact of various combinations of design elements.
Each of these methods has its advantages and is suitable for different situations depending on the research objectives and available resources.
The following strategies can be used to optimize the site and achieve maximum impact:
• Using A/B testing: This identifies the most effective page design options by directly comparing two versions among the selected audience.
• Using multivariate testing: This method explores the interaction of different page elements with each other, with the aim of improving the design and increasing functional consistency.
Before starting multivariate testing, it is necessary to ensure a significant number of users to the page under test. If there is sufficient traffic volume, it is recommended to use both types of testing for comprehensive website optimization. A/B testing is often chosen by product companies because it allows them to test significant changes that can significantly impact the page while being easier to implement.
In turn, to make A/B testing as productive as possible, the following testing structure is recommended:
• Research: Collecting and analyzing data to identify key areas for improvement.
• Prioritization: Determining the order of experiments based on potential impact on results.
• Experimentation: Implementing tests, and strictly controlling variables for data purity.
• Analyze, Learn, Repeat: Analyze results to identify successful changes and integrate them into business processes, and continually learn and repeat successful experiments for continuous improvement.
Adopting this structured approach not only helps to improve the testing process but also contributes to a better understanding of the mechanisms at work, increasing the chances of successful implementation of optimizations [12,13].
3. Methodology for Conducting A/B Testing
The methodology of A/B testing is pivotal for achieving reliable and actionable insights that can drive the optimization of user experience. This section outlines a comprehensive approach to conducting A/B tests, focusing on the stages of goal setting, hypothesis formulation, preparation and execution of tests, and result analysis. The methodology is enriched with specific examples, including code snippets and visual elements, to cater to professionals with a medium level of expertise.
1. Setting Goals and Hypotheses
The first step in any A/B testing process involves defining clear, measurable goals. These goals should align with the broader business objectives and user experience enhancements. Common goals include increasing conversion rates, enhancing user engagement, or reducing bounce rates.
Minimum Detectable Effect (MDE) is the smallest improvement in the conversion rate of the existing object (baseline conversion rate) that you aim to detect during the experiment. MDE is a crucial parameter for evaluating the cost and potential profitability of conducting A/B experiments. From a practical perspective for mobile app marketers, choosing an appropriate MDE p-value means achieving a balance between the cost of acquiring paid traffic for the experiment and attaining a meaningful return on investment (ROI) [14].
Calculate Sample Size. The calculator allows for the evaluation of various statistical schemes when planning an experiment (trial, test) where conclusions are drawn using a null hypothesis statistical test. It can be used both as a sample size calculator and as a statistical power calculator. Typically, it is necessary to determine the required sample size considering the specific power requirement. However, in cases where a predefined sample size exists, the calculator can instead compute the power for the given effect size of interest [15,16].
Once goals are set, the next step is to formulate hypotheses. A well-structured hypothesis should identify the variable to be tested, the expected outcome, and the rationale behind the expectation. For example, a hypothesis could be: "Changing the color of the call-to-action button from blue to green will increase the click-through rate by 10% because green is perceived as more inviting and action-oriented."
Minimum Detectable Effect (MDE) is the smallest improvement in the conversion rate of the existing object (baseline conversion rate) that you aim to detect during the experiment. MDE is a crucial parameter for evaluating the cost and potential profitability of conducting A/B experiments. From a practical perspective for mobile app marketers, choosing an appropriate MDE p-value means achieving a balance between the cost of acquiring paid traffic for the experiment and attaining a meaningful return on investment (ROI) [17].
Calculate Sample Size: The calculator allows for the evaluation of various statistical schemes when planning an experiment where conclusions are drawn using a null hypothesis statistical test. It can be used both as a sample size calculator and as a statistical power calculator. Typically, it is necessary to determine the required sample size considering the specific power requirement. However, in cases where a predefined sample size exists, the calculator can instead compute the power for the given effect size of interest [18,19].
2. Preparation and Execution
Preparation for A/B testing requires careful planning and setup to ensure that the test will provide reliable data. This involves selecting the appropriate metrics, creating variants, and segmenting the audience. The key metrics should directly relate to the goals, such as click-through rates, conversion rates, or average session duration, or other product metrics.
Here is an example of setting up an A/B test using a popular testing tool, Optimizely, with a focus on testing a call-to-action button color change:
variation_conversions = [1, 1, 1, 1, 1, 1, 0, 1, 1, 1]
# Perform t-test t_stat, p_value variation_conversions)
stats.ttest_ind(control_conversions,
print(fT-statistic: {t_stat}, P-value: {p_value}')
In this example, conversion data from the control and variation groups are analyzed using a t-test to determine if the difference in conversion rates is statistically significant. A low p-value (typically less than 0.05) would indicate that the observed difference is unlikely to have occurred by chance, thus validating the hypothesis.
4. Visualization
Visualizing the results can provide additional insights and facilitate decision-making. Graphs and charts, such as conversion rate comparisons and confidence intervals, are valuable tools. Here is an example of how to visualize A/B test results using matplotlib in Python:
// Initialize Optimizely SDK
const optimizelyClient = require('@optimizely/optimizely-sdk');
// Define the experiment
const experimentKey = 'cta_button_color_test';
const userId = 'user123';
import matplotlib.pyplot as plt # Conversion rates
control_rate = sum(control_conversions) / len(control_conversions) variation_rate = sum(variation_conversions) / len(variation_conversions)
// Variation keys
const variationBlue = 'blue_button'; const variationGreen = 'green_button';
# Plotting the results
labels = ['Control (Blue)', 'Variation (Green)'] rates = [control_rate, variation_rate]
// Start the experiment
optimizelyClient.activate(experimentKey, userId);
// Determine the variation to show
const variation = optimizelyClient.getVariation(experimentKey, userId);
if (variation === variationGreen) {
// Show green button
document.getElementById('cta-button').style.backgroundColor = 'green';
} else {
// Show blue button (control)
document.getElementById('cta-button').style.backgroundColor =
'blue';
}
In this code snippet, the Optimizely SDK is used to activate the experiment and determine which variation the user will see. This ensures that users are randomly assigned to either the control (blue button) or the variation (green button).
3. Analyzing Results
Once the test is executed and sufficient data is collected, the analysis phase begins. This involves comparing the performance of the control and variation groups using statistical methods. Key metrics are analyzed to determine whether the observed differences are statistically significant.
A common approach to analyzing A/B test results is to use statistical significance tests, such as a t-test or chi-square test, or other tests. Here is an example of using Python to perform a t-test on conversion rates from two variations:
from scipy import stats
# Conversion data from the control (blue button)
control_conversions = [0, 1, 0, 1, 0, 0, 1, 1, 0, 1]
# Conversion data from the variation (green button)
plt.bar(labels, rates, color=['blue', 'green']) plt.xlabel('Group') plt.ylabel('Conversion Rate') plt.title('A/B Test Results') plt.show()
This code snippet creates a bar chart that compares the conversion rates of the control and variation groups, providing a clear visual representation of the test outcomes.
By following these methodological steps, professionals can conduct robust A/B tests that yield reliable data, driving informed decisions and continuous improvements in user experience. This structured approach, supported by concrete examples and visualizations, ensures that A/B testing is not only effective but also accessible to those with intermediate expertise.
4. Tools for A/B Testing
Selecting the right tools for A/B testing is crucial for obtaining accurate and actionable insights. 1. GrowthBook
GrowthBook is a robust and user-friendly tool designed for A/B testing and feature flagging, providing a comprehensive suite for experimentation and personalization. It supports various types of experiments, including A/B, multivariate, and feature tests [20,21]. Features:
• Integration with Analytics Tools: Allows for detailed user behavior analysis and segmentation.
• Visual Editor: Enables non-technical users to create and modify experiments without coding.
• Targeting and Personalization: Facilitates the creation of highly targeted and personalized experiences.
Example of Setting Up an Experiment:
Here is an example of setting up an experiment using GrowthBook:
<script src="https://cdn.growthbook.io/growthbookjs"></script> <script>
X X
o 00 A c.
X
00 m
o
2 O
ho ■p»
es o es
CO
o
HI
m
X
<
m o x
X
growthbook.init({
apiHost: "https://api.growthbook.io", clientKey: "your-client-key", user: {
id: "user-id"
}
});
growthbook.run({ key: "experiment-key", variations: [
{ id: "control", name: "Control" },
{ id: "variation", name: "Variation" } ],
callback: (result) => { if (result.variation === "variation") { // Code for the variation } else {
// Code for the control }
} });
</script>
This script integrates GrowthBook with your website, enabling the experiment to run.
2. Optimizely
Optimizely is a powerful and versatile platform widely used for A/B testing, multivariate testing, and personalization. It offers advanced features suitable for more complex testing scenarios and supports serverside and client-side experiments. Features:
• Multi-Armed Bandit Experiments: Optimizes traffic allocation to the best-performing variations in real-time.
• Feature Management: Allows feature flagging and experimentation on new features before a full rollout.
• Audience Segmentation: Provides robust targeting options based on user behavior and attributes.
Example of a Multi-Armed Bandit Experiment: Optimizely's SDK can be used to implement multi-armed bandit experiments. Here is a code snippet illustrating this:
const optimizelyClient = require('@optimizely/optimizely-sdk');
const experimentKey = 'pricing_test'; const userId = 'user123';
const variation = optimizelyClient.activate(experimentKey, userId);
if (variation === 'variationl') { // Show first pricing option } else if (variation === 'variation2') { // Show second pricing option } else {
// Show default pricing option }
This code demonstrates how Optimizely can dynamically adjust traffic allocation to maximize the performance of different pricing strategies.
3. VWO (Visual Website Optimizer)
VWO is an all-in-one platform that provides tools for A/B testing, multivariate testing, and conversion rate optimization. It is known for its ease of use and comprehensive set of features tailored for both beginners and advanced users. Features:
• Heatmaps and Session Recordings: Provides visual insights into user behavior, enhancing experiment design.
• Conversion Funnels: Analyzes drop-off points in user journeys to identify areas for improvement.
• Personalization Engine: Creates customized user experiences based on visitor data.
Example of Setting Up an A/B Test:
Here is a code snippet demonstrating how to set up a simple A/B test using VWO's JavaScript API:
// Include the VWO SmartCode on your website (function(a,b,c,d,e,f,g){a['vwoSettings']=e;a[e]=a[e]||[];a[e].push(fu nction(){var
d=c.createElement('script');d.async=true;d.src=('https:'==c.location.proto col?'https://':'http://')+f;var
b=c.getElementsByTagName('head')[0];b.appendChild(d)});a[e].push(fu nction(){window._vwo_code=f;window._vwo_code_version=g})})(win dow,document,'script',0,'vwoCode','cdn.vwo.com/ads/smartcode.js',1);
// Define the variation code _vwo.push(['applyCode', { testId: 1, segment: '1', version: 1, code: function() {
document.getElementById('cta-button').style.backgroundColor =
'green'; }
}]);
This code snippet includes the VWO SmartCode to integrate with your website and sets up an A/B test where the background color of the call-to-action button is changed to green. 4. Convert
Convert is a sophisticated solution that offers a wide range of capabilities for A/B testing, personalization, and automated optimization. It is known for its ease of use and powerful feature set [22]. Features:
• Automated Personalization: Uses machine learning to deliver personalized experiences at scale.
• Rules-Based Targeting: Allows for the creation of complex targeting rules based on user attributes and behaviors.
• Comprehensive Reporting: Provides in-depth analysis and reporting tools integrated with various analytics platforms.
Example of Setting Up an A/B Test:
Here is an example of setting up an A/B test using Convert's JavaScript API:
<!-- Include the Convert library --> <script>
(function(e, t, n, r, i, s, o) { e['ConvertBoxObject'] = i; e[i] = e[i] || function() { (e[i].q = e[i].q || []).push(arguments) }, e[i].l = 1 * new Date();
s = t.createElement(n), o = t.getElementsByTagName(n)[0]; s.async = 1; s.src = r; o.parentNode.insertBefore(s, o) })(window, document, 'script', 'https://cdn-
3.convertexperiments.com/js/10000001-1000001js', 'convert'); </script>
<!-- Define the variation code -- > <script>
convert('experiment1', function(response) { if (response.variation === 'variation1') {
document.getElementById('cta-button').style.backgroundColor = 'green'; } else {
document.getElementById('cta-button').style.backgroundColor =
'blue'; }
});
</script>
This code snippet includes the Convert library and sets up an A/B test where the background color of the call-to-action button is changed based on the test variation.
The selection of an A/B testing tool depends on specific needs, including the complexity of experiments, integration requirements, and desired features. GrowthBook is ideal for flexible and developer-friendly experimentation, while Optimizely offers advanced experimentation capabilities. VWO stands out with its user-friendly interface and comprehensive features, and Convert excels in providing robust testing and targeting options. By leveraging these tools effectively, professionals can conduct robust A/B tests, driving continuous improvements in user experience and achieving significant business outcomes.
Conclusion
Iterative A/B testing is a critical tool for optimizing user experience and improving products. This approach not only allows us to test the effectiveness of specific changes in real-world settings but also promotes a culture of continuous optimization within a company. The results of the study confirm that the systematic application of A/B testing helps improve user interfaces and the performance of websites and applications. The importance of this approach is particularly relevant in the face of ever-changing technology and user expectations. Based on the data obtained as a result of testing, companies can effectively manage changes, minimize risks, and optimally use resources to achieve strategic goals.
Optimizing the user experience through iterative A/B testing Chumachenko A.
Simpals
JEL classification: C10, C50, C60, C61, C80, C87, C90
This article discusses optimizing the user experience through iterative A/B testing as a means of improving the effectiveness of web pages and user interfaces. A/B testing, also known as split testing, is a methodology that allows you to compare two versions of a test object to determine which is more effective in achieving your goals. The article analyzes the role of A/B testing in modern marketing strategies and user interface design and emphasizes the importance of an iterative approach in continuous optimization and adaptation to user preferences. The main focus is on analyzing key indicators, which include conversion rates and user engagement metrics, and examining the effects of implementing changes through successive iterations of testing. Keywords: user experience, A/B testing, programming, IT, digitalization.
References
20.
Pavlovich Yu.G., Kirinovich I. F. A/V testing as an effective means for adapting the user interface in the iterative model of application development for mobile devices // REPORTS of BGUIR. 2021. No.1. pp. 30-36.
Ireland R., Liu A. Application of data analytics for product design: Sentiment analysis of online product reviews //CIRP Journal of Manufacturing Science and Technology. -2018. - T. 23. - C. 128-144.
Rodrigues J. Product Analytics: Applied Data Science Techniques for Actionable Consumer Insights. - Addison-Wesley Professional, 2020.
Kumar R. Data-Driven Design: Beyond A/B Testing // Conference on Human Information Interaction and Retrieval. 2019. pp. 1-2.
Ron Kohavi Online Controlled Experiments and A/B Tests. [Electronic resource] Access mode: https://www.semanticscholar.org/paper/Trustworthy-Online-Controlled-
Experiments-Kohavi-Tang/d7577815504a551ce7af4f03ab35987543fc2d32 (accessed 05/08/2024).
Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. [Electronic resource] Access
mode:https://www.researchgate.net/publication/339914315_Trustworthy_Online_Contr olled_Experiments_A_Practical_Guide_to_AB_Testing (accessed 05/8/2024). Sheng J., Liu H., Wang B. Research on the Optimization of A/B Testing System Based on Dynamic Strategy Distribution // Processes. 2023. №11 (912). pp.1-12. Analysis of A/B testing trends. [Electronic resource] Access mode: https://trends.google.com/trends/explore?date=today%205-y&q=%2Fm%2F0284x50&hl=ru (accessed 05/8/2024).
Agarwal Shashank Optimizing Product Choices through A/B Testing and Data Analytics: A Comprehensive Review// International Journal of Advanced Research in Science, Communication and Technology. 2023. №3. pp. 550-555.
The Importance of A/B Testing for User Experience. [Electronic resource] Access mode: https://www.freedomtoascend.com/marketing/marketing-tactics/a-b-testing/a-b-testing-for-user-experience / (accessed 05/8/2024).
A/B Testing: Optimizing Success Through Experimentation. [Electronic resource] Access mode: https://dzone.com/articles/ab-testing-optimizing-success-through-experimentat (accessed 05/8/2024).
Somit Gupta, Ronny Kohavi, Alex Deng, Jeff Omhover, and Pawel Janowski. A/B Testing at Scale: Accelerating Software Innovation // In Companion Proceedings of The 2019 World Wide Web Conference (WWW '19). 2019. pp. 1299-1300. P. Mahajan, D. Koushik, M. Ulavapalle and S. Shetty, "Optimizing Experimentation Time in A/B Testing: An Analysis of Two-Part Tests and Upper Bounding Techniques // IEEE International Conference on Contemporary Computing and Communications (InC4), Bangalore, India. 2023. pp. 1-4
Minimum Detectable Effect (MDE). [Electronic resource] Access mode https://splitmetrics.com/resources/minimum-detectable-effect-mde/ (accessed
8.05.2024).
Sample Size and Power Calculation. [Electronic resource] Access mode https://www.researchgate.net/publication/319442443_Sample_Size_and_Powr_Calculat ion (accessed 8.05.2024).
E. Claeys, P. Gançarski, M. Maumy-Bertrand and H. Wassner Dynamic Allocation Optimization in A/B-Tests Using Classification-Based Preprocessing // IEEE Transactions on Knowledge and Data Engineering. 2021.№.35.pp. 335-349. Minimum Detectable Effect (MDE). [Electronic resource] Access mode https://splitmetrics.com/resources/minimum-detectable-effect-mde/ (accessed
05/8/2024).
Sample Size and Power Calculation. [Electronic resource] Access mode https://www.researchgate.net/publication/319442443_Sample_Size_and_Power_Calcul ation (accessed 05/08/2024).
Victoria Pokhilko, Qiong Zhang, Lulu Kang, D'arcy P. Mays D-Optimal Design for Network A/B Testing // Journal of Statistical Theory and Practice. 2019. №13 (31). [Electronic resource] Access mode https://goldinlocks.github.io/Introduction-to-A-B-testing-in-python / (accessed 05/8/2024).
W. S. De Souza, F. O. Pereira, V. G. Albuquerque, J. Melegati and E. Guerra A Framework Model to Support A/B Tests at the Class and Component Level // IEEE 46th Annual Computers, Software, and Applications Conference (COMPSAC), Los Alamitos, CA, USA. 2022. pp. 860-865.
Salam M., Taha S., & Hamed M. Advanced Framework for Automated Testing of Mobile Applications // 4th Novel Intelligent and Leading Emerging Sciences Conference (NILES).2022. pp.233-238.
Ramesh Johari, Pete Koomen, Leonid Pekelis, David Walsh Peeking at A/B Tests: Why it matters, and what to do about it // KDD '17: Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. 2017. pp. 15171525.
I I
o
03
A J=
I 03
m
o
2
O
ro •pt
