Recently, Guo Yongyi, an assistant professor at the University of Wisconsin-Madison, and his collaborators proposed a personalized just-in-time adaptive intervention (pJITAI) method based on reinforcement learning.
图 | 郭永祎（来源：郭永祎）
This approach can be used in digital health to reduce cannabis intake in young people aged 18-25 years in a personalized way through flexible formats such as mobile apps and questionnaires.
The actual clinical trial has been registered (NCT05824754) in ClinicalTrials.gov and has started in March 2024.
In the clinical trial, about 120 young people who had a marijuana habit and wanted to quit participated. For each of these users, they will decide twice a day whether or not to display the intervention information on their phones during the 30-day engagement period, helping them reduce their cannabis intake.
At the same time, researchers will also use questionnaires and other methods to collect relevant information about users simultaneously, so as to better make personalized optimization decisions for different users.
Compared with other application scenarios of reinforcement learning, the field of digital health will face practical problems such as insufficient sample size, user differentiation, and loss of user engagement.
Based on comprehensive considerations, the research team used contextual bandit to model the problem: that is, at each decision point within a certain time span, based on previously observed data, to decide whether to "intervene" with the user.
The specific form of intervention here is to push appropriate intervention information. These decisions will be combined with each user's current state and model parameters to determine the user's return.
In the study, the research team worked together with experts in the medical field to develop a return indicator to ensure that it is highly correlated with the reduction of cannabis intake. At the same time, one of the goals of this study is to maximize the cumulative return of users over a period of time.
What's special about this payback model is that given the limited number of users and the number of decisions, researchers need to strike a balance between efficient data use and consideration of user variability.
Specifically, in the design of the model parameters, they hope that different users will report that the parameters in the model are both different and have common parts.
In this way, for each user in the decision-making process, the data of other users can help them quickly learn the common parts of the user return model.
At the same time, they will be able to learn how Ta differs from other users, especially based on that user's historical data.
Among them, the research group used a mixed-effect model to describe the return model of different users. Fixed effects are common to users, and random effects are different for each user.
For the above return model, the team used the Thompson sampling algorithm to make adaptive decisions, and optimized the quality of decisions while learning user returns.
Unlike the previous method, they used projected gradient descent and empirical Bayesian to update the learned model information to ensure that the algorithm could still operate autonomously and stably under a large number of parameters.
At the same time, from the data of previous clinical trials, the research group determined the prior division of specific parameters in the algorithm.
In addition, from previous clinical trials, they also extracted and refined simulated users, and based on this, they established a variety of simulation environments, such as different degrees of intervention effects, different degrees of user habituation, etc., to test the performance of the algorithms used.
In a variety of environments, the algorithm can effectively identify the heterogeneity of users and exploit the commonalities of users, so as to achieve the purpose of optimizing returns.
Compared with other algorithms, this algorithm can especially show advantages in the case of high user heterogeneity.
This is important in the application of this work, as studies have shown that the intake habits of addictive substances and the relationship between intake behavior and psychological state vary greatly among different people [1].
During the study, in order to ensure the user experience of the clinical trial, some members of the team also downloaded and tested the mobile app during the trial period and gave a lot of feedback.
For example, they realized that user habituation is almost inevitable in practice, and that the impact of a single intervention on the user is almost always positive (i.e., it can help the user reduce cannabis intake).
Intervening too often with information can make users desensitized to information and even uninstall programs. Therefore, they added a variety of user habituation scenarios to the experiment and used them as important indicators to test the performance of the algorithm.
最终，相关论文以《reBandit：基于随机效应的在线 RL 算法减少大麻的使用》（reBandit：Random Effects based Online RL algorithm for Reducing Cannabis Use）为题发在 arXiv[2]。
Figure | Related papers (source: arXiv)
At the end of the clinical trial, they need to perform a statistical analysis of the data, the most important of which is the statistical inference of the effect of the intervention, so as to verify the effectiveness of the intervention program.
Compared with traditional statistical analysis, the data in this study are not independent for both time points and different users. So, there is a need to develop new methods of statistical inference, which they are currently working on.
Resources:
Benson， Lizbeth， et al. "Associations between morning affect and later-day smoking urges and behavior." Psychology of Addictive Behaviors （2023）
2. Hattapus://Arxiv.org/PDF/2402.17739.PDF
Operation/Typesetting: He Chenlong