Evaluating JITAI Effectiveness: Techniques and ML Tools for Measuring Digital Engagement
Thoughtful application of adaptive and data-driven communications not only elevate user engagement but also play a crucial role in:
- driving meaningful behavior change
- boosting adherence
- enhancing care coordination
- fostering improved health literacy
- shared decision-making
- empowering patients and caregivers
At MEMOTEXT, we thought we’d share about the importance of how to measure the effectiveness of tools such as JITAI’s and Microrandomization.
A Just-in-Time Adaptive Intervention (JITAI) is an adaptive digital health approach designed to deliver engagement in an evidence-informed sequence precisely when an end-user needs it, based on their context as determined by health-data such as behaviors, engagement patterns and health data to name a few.
Microrandomizations are instrumental in refining JITAIs by enabling continuous, data-driven learning about user engagement. By introducing slight, randomized variations to intervention components, we can gain insight into which conditions best promote user interaction. For instance, varying message timing across user wake and activity patterns may reveal optimal windows for engagement, while adapting message content (e.g., a motivational quote versus a practical tip) can allow for a nuanced understanding of user preferences.
Below, we provide a list of measurement techniques, which give you an idea of both the types of variables to consider and the importance of measurement. See both advanced Machine Learning (ML) techniques and non-ML JITAI measurement techniques for your reading pleasure.
Non-ML Techniques… Just a few ideas for your consideration.
- Engagement with Content: Track CTRs, goal completions, time-to-action, and basic KPIs to understand what drives engagement.
- Day-Specific Trends: Monitor which days show higher or lower engagement.
- User Interaction Flow: Study the sequence of actions (e.g., link clicks, survey completions) to optimize message structure.
- Drop-Off Points: Identify where users disengage in a message series to adjust cadence or content.
- Frequency-Based Engagement: Compare engagement across message frequencies to find the ideal rate.
- Media and Content Metrics: Analyze engagement differences across media types (SMS, voice, email) and content length/type.
- User feedback: Collect feedback for context on engagement patterns.
- Retention and Drop-Off Rates: Track when users disengage to adjust timing and content, reducing message fatigue.
Machine Learning/Advance Techniques
- NLP for Topic & Sentiment Analysis: Use NLP to gauge sentiment and identify topics resonating with users.
- Clustering: Segment users based on engagement, utilizing insights from non-ML techniques.
- Free Text Feedback: Analyze open-ended responses for emotional indicators to refine tone and content.
- Reinforcement Learning for Timing: Use real-time feedback to adjust message timing for optimal engagement.
- A/B Testing with ML Insights: Use ML insights to refine A/B tests, validating personalized message variations.
- Time-Series Analysis: Predict trends in behavior to identify ideal intervention times based on past engagement data.
- Pattern Recognition: Detect engagement trends by demographics or behavior, customizing strategies.
- Context-Aware Recommendations: Tailor content based on context (location, activity) to improve relevance.