Editor/Yilanjun
In the era of rapid development of mobile Internet, the scale of users and the amount of network information have shown explosive growth, and information overload has increased the difficulty of user selection. In the continuous iteration of the recommendation system, its algorithms, strategies, features, functions and user interface are often updated and optimized, and the adjustment of the recommendation algorithm is particularly crucial. However, due to the wide application of deep learning models, it is difficult to directly judge the user experience and effect of the adjusted recommendation algorithm through experience.
In order to more accurately evaluate and optimize recommender systems, A/B experimentation has become an indispensable tool. A/B experiments can quantify the changes of various indicators, so as to scientifically evaluate the effect of the recommendation system and provide data support for subsequent optimization. In this article, we will take the A/B testing platform (DataTester) under VeDI, a digital intelligence platform of Volcano Engine, as an example to introduce how Douyin Group can use its capabilities to continuously achieve accurate optimization of the recommendation system.
In the process of optimization and exploration of recommender system, different algorithms superimpose different strategies or functional effects, and it is the most efficient way to find the optimal strategy by doing A/B experiments of functional combination through experimental parameters. Taking the Volcano Engine A/B test DataTester as an example, it currently supports the configuration of experimental parameters of Number, String, Boolean, and Json to help users directly implement A/B experiments in different dimensions of recommender system policies.
Taking the short video app e-commerce recommendation scenario as an example, assuming that the timing of product content display is different, it will have an impact on the user's video consumption time and e-commerce GMV, and the A/B experiment for this strategy can be designed as follows:
• Control group: Show the product card as soon as the video starts playing
•Experiment Group 1: Display the product card after 5 seconds of video playback
•Experiment Group 2: The product card will be displayed after 10 seconds of video playback
In the above experiments, the Volcano Engine DataTester can directly realize the grouping of control group, experimental group 1, and experimental group 2 through the adjustment of experimental parameters. The experiment is completed by parsing the parameters in the code and displaying the product card after X seconds of video playback. If you want to add an experimental effect such as "display the product card after 8 seconds of video playback", you don't need to modify the code, you just need to continue to add an experiment with new experimental parameters. On this basis, dozens or even more sets of experiments with different parameter values can be created and the optimal strategy can be obtained with little to no additional development manpower in the process.
It should be noted that since the experimental parameters are a functional control configuration, it is necessary to avoid a misunderstanding when designing A/B experimental parameters: do not design experimental parameters according to the dimension of experimental design, but according to the dimension of functional control. This is especially important in mobile APP experiments, because APP products usually have a long release cycle and low frequency of changes, and with the experimental parameters of the functional control dimension, you can open multiple sets of A/B experiments with different parameters at any time without issuing a version, and screen the optimal combination of parameters to take effect online.
As the core product of VeDI, the Volcano Engine Digital Intelligence Platform, DataTester originates from ByteDance's long-term technology and business precipitation. At present, DataTester has served hundreds of enterprises, including well-known brands such as Midea, Get, BSH Home Appliances, and Leke Fitness. These enterprises have benefited from DataTester's scientific decision support in the business process, and have achieved continuous business growth and optimization.