Real-world evaluation of a chatbot for promoting the reach and accessibility of a smoking cessation service

Abstract

Background: Although evidence-based smoking cessation interventions are widely accessible at minimal or no cost, their utilization remains low across various regions. “Dr. Wise” is a chatbot launched by Tung Wah Group of Hospitals Integrated Centre on Smoking Cessation (ICSC) on 31st May 2021, aiming to assist with regular services, particularly recruitment. This study aimed to evaluate whether “Dr. Wise” can promote the reach of ICSC services, and further explore the interaction between users and counselors in live chat channel embedded in “Dr. Wise”. Methods: This thesis contains two studies. The first is a longitudinal study of ICSC service users with follow-ups at 8 weeks, 26 weeks, and 52 weeks after registration, aiming to compare characteristics, retention rates and abstinence outcomes between chatbot users and non-users. From June 2021 to June 2022, 3074 smokers (123 chatbot users) enrolled in ICSC. Baseline characteristics were compared via bivariate analyses. Robust Poisson regression was used to estimate the relative risk (RR) of self-reported 7-day point prevalence tobacco abstinence (7-day PPA) by chatbot use status. Those with missing outcome data were assumed to be non-abstinent. In the second study, 122 valid live chat dialogues sessions were retrieved to explore the prominent topics and sentiment tendency shown in users’ and counselors’ messages. Word frequency analyses were used to profile the words. Latent Dirichlet Allocation (LDA) topic modeling was used to identify the prominent topics. Sentiment analyses were conducted using Linguistic Inquiry and Word Count (LIWC) dictionary. Linear regression was used to examine the association between sentiment patterns and user’s engagement. Results: In study 1, chatbot users were younger (mean±SD age 38.0±10.5 vs 43.5±11.7; p<0.001) and more likely to receive pharmacotherapy (73.8% vs 59.5%; p=0.002) than non-users. The retention rates of chatbot users and non-users showed no significant difference at 8 weeks, 26 weeks, and 52 weeks (all p > 0.05). Chatbot users reported higher abstinence rates at 52 weeks than non-users (RR=1.38; 95% CI 1.05-1.81; p=0.02), adjusting for demographic factors, baseline predictors of quitting, and treatment received. In study 2, 1680 users’ messages and 1814 counselors’ messages were analyzed. Three themes, including 26 topics, emerged from users’ messages: relapse prevention, access to ICSC services and brief information consultation. Four themes, including 11 topics, emerged from counselor’s messages: supportive response, facilitating the use of services, pharmacotherapy, and withdrawal symptoms. Negative sentiment words were less prevalent than positive sentiment words and correlated with body words and health words in both users’ and counselors’ messages. The prevalence of negative sentiment words in both groups were positively associated with users’ engagement (p<0.05). Conclusion: This study found that “Dr. Wise” could reach different populations compared to other recruitment channels in ICSC service. The built-in live chat channel supports users by providing relapse prevention support, connecting users with clinics, and addressing the concerns of ongoing pharmacotherapy. Further trials are warranted to test the effectiveness of the chatbot in promoting cessation outcomes and investigate how the expression of negative sentiment in live chat engages users.