AI coaching should be judged by behavior after support ends

In evaluations of conversational coaches, metrics such as registration, conversation completion, satisfaction, and intention to change are gathered quickly. However, these are indicators of having accessed the intervention or showing psychological responses, not the real-world behaviors themselves (e.g., walking, getting vaccinated, taking medication, submitting work). A review integrating RCTs among young adults noted that while engagement relates to effects, repetitive content and technical issues hinder continued use, and many behavior metrics relied on self-reports. [2]

Evidence is divided into contact, attitude/self-efficacy, self-reported behavior, behavior confirmed by external records or devices, and maintenance after support is reduced. It is not always true that higher stages are more important; rather, it is crucial to choose the stage corresponding to the decision at hand. For example, for appointment reminders, actual attendance is the primary outcome; for executive coaching, specific actions after meetings are key; for habit formation, repetition after notifications cease is the main result.

Collapsing everything into a single composite score hides those who dropped out or did not change behind the satisfaction of those with high conversation counts. Define one primary outcome before starting, and report contact and intention as process metrics, while reporting real-world behavior and maintenance as outcome metrics separately. Do not call conversation logs "results"; instead, link them to the time when the next behavior occurred.

• Contact: Measure invitation reach, first-time use, and conversation completion
• Cognition: Measure intention, self-efficacy, understanding, and satisfaction
• Self-reported behavior: Measure frequency and amount as reported by the individual
• Objective behavior: Confirm via public records, business logs, devices, or deliverables
• Maintenance: Measure whether behavior persists after reducing notifications or conversation frequency

Singh et al.'s formal meta-analysis on lifestyle behaviors integrated 19 trials and 3,567 participants. In the analysis limited to RCTs, there was a small positive effect on physical activity, with a standardized mean difference (SMD) of 0.27 and a 95% confidence interval (CI) of 0.07–0.46. In contrast, for step count, the SMD was 0.18 with a 95% CI of 0.00–0.36, and the P-value was 0.06. [1] Rather than a blanket statement that it "works on lifestyle behaviors," the certainty of results varies by metric and comparison design.

This review includes not only RCTs but also single-arm pre-post comparisons, with 14 of the 19 trials rated as having weak quality. The median intervention duration was 6 weeks, with 58% lasting 6 weeks or less. [1] Improvements immediately post-intervention are an important starting point but do not directly guarantee that habits persist after the coach is gone or that they replicate under different life conditions.

A 2025 meta-analysis targeting young adults included 31 RCTs and 29,637 participants, but only 6 trials could be integrated for health behaviors, yielding an effect size of SMD 0.11. Twenty-five of the 31 studies had an overall high risk of bias, and GRADE certainty ranged from very low to low across outcomes. Interventions ranged from minutes to 4 months, and additional follow-ups were limited to 2 weeks–6 months in some cases, with reliance on self-reports also noted. [2] We must retain the boundary that while the average is positive, evidence for sustained real-life behavior is thin.

A preregistered RCT conducted in Argentina randomized 249,705 participants to a behavior-science-integrated WhatsApp chatbot, one-way reminders, or no contact. Confirming the next COVID-19 vaccination within 4 weeks via Ministry of Health records, the chatbot group had a rate 1.56 percentage points higher than the no-contact group and 0.99 percentage points higher than the one-way message group. [3] A strength is that it measured actual vaccination rather than intention.

Although this appears large relatively, it is an absolute difference under conditions where the no-contact group's vaccination rate was 0.70%. [3] The target was limited to those who had received at least one dose, were eligible for the next dose, and had a unique WhatsApp number. This is not evidence of persuading strong anti-vaccine groups nor of forming habits lasting several months.

Furthermore, this trial did not isolate bidirectionality alone. The chat included multiple behavior-science functions such as facilitating appointments, planning, and social norms; the difference from one-way messages reflects this combination. [3] In practice, do not interpret this as "it worked because we conversed with AI"; instead, specify whether to replicate friction reduction, implementation intentions, individualized guidance, or bidirectional confirmation.

• Primary outcome: Fix one target behavior that occurred within the deadline, rather than intention
• Comparison: Compare against no intervention as well as one-way support with the same information volume
• Absolute difference: Show the base number and percentage point difference, not just the ratio
• Mechanism: Separate conversation, personalization, friction reduction, and reminders to the extent possible

A 2026 pragmatic RCT conducted in primary care facilities in Hong Kong randomized 627 adults aged 45–75 with hypertension, diabetes, or risk thereof, into a 12-week motivational interviewing chatbot add-on group and a usual care group. In the modified ITT of 460 participants, self-reported physical activity improved by 576 MET-min/week and fruit/vegetable intake by 0.27 servings/day compared to usual care at 12 weeks. [4] This is new field evidence connecting dedicated coaches to daily clinical practice.

However, the primary outcome was self-reported via an open-label design, and missing data excluding those outside the modified ITT were not random. A sensitivity analysis including all 627 participants (including 78 excluded for lacking compatible smartphones post-randomization) was performed, but reachability due to device conditions remains an implementation outcome. [4] Do not substitute the conversation completion rate for those who could use it with the effect for the entire target population.

At 9 months, only changes within the intervention group were tracked, with no simultaneous comparison to usual care. While one can observe that improvements in physical activity etc. persisted from baseline, this does not constitute a sustained causal effect excluding seasonality, additional treatments, or regression to the mean. [4] If displaying "9-month persistence," one must state in the same sentence that the control is absent.

A 2024 review of diabetes self-management chatbots included 25 studies, integrating a 0.30 improvement in glycemic-related values from baseline, while weight change was not significant. [5] The existence of metrics that appear more objective than self-reports is progress, but objective measurement targets do not automatically strengthen causal inference.

Of the 25 studies, only 9 were intervention studies, mostly non-randomized comparisons, with only 1 RCT. The meta-analysis primarily aggregated pre-post intervention data rather than differences from control groups, making it difficult to separate natural progression, treatment changes, and measurement familiarity. [5] Distinguish between the fact that biomarkers moved and the attribution that the chatbot caused it.

The same applies to business behavior change. Even using external logs such as submission counts, deal conversions, absenteeism, or processing time, explanations compete if busy periods, supervisor changes, reward systems, or concurrent training change. Score measurement validity and comparison validity in separate columns, and do not approve deployment based on only one.

• Measurement Validity: Can the target behavior be confirmed without relying on the individual's memory?
• Comparison Validity: Can it be compared with an appropriate control from the same period?
• Attribution Possibility: Were concurrent initiatives, seasonality, and policy changes recorded?
• Maintenance Possibility: Was the same metric tracked after support was reduced?

Current evidence is not a monolith called "conversational AI." Among 31 RCTs on young adults, 18 were rule-based, 10 were search-based, and only 3 studies tested generative types. [2] In a 2023 lifestyle behavior review, only 7 of 19 trials involved chatbots with AI or natural language processing. [1] Many effects are derived from dedicated systems that reliably deliver fixed behavior techniques, rather than free generation.

Interventions included in the lifestyle behavior review repeatedly used goal setting, self-monitoring, goal review, barrier problem-solving, feedback, and social support. [1] However, since multiple elements are bundled, it is impossible to identify empathetic conversational tone or generative capability as the cause of effects. In practice, define interventions not by model name, but by who, when, which behavior technique, and within what safety boundaries.

Using generative AI increases intervention variability alongside flexibility. Evaluate separately the risks of giving different advice under identical user conditions, making unfounded psychological interpretations, or missing crisis signals. Existing dedicated chatbot studies serve as a reference for behavior design but do not prove the safety, accuracy, or long-term effects of general LLMs.

It may appear as a dose-response of the intervention that those who completed more conversations improved more. In a 2026 RCT, the estimated effect for compliers was larger than for the modified ITT. [4] However, those with stronger initial motivation, familiarity with devices, or stable health may have continued conversations more; thus, usage is both a cause of effect and a result of participant characteristics.

Punishing dropout as failure loses sight of who did not receive the intervention. Distinguish between non-start, technical inability, early dropout, graduation due to goal achievement, stopping due to burden, and safety-related interruptions. A review of young adults noted dropout and incomplete sessions as issues, with repetitive content and technical problems as barriers. [2] Keep results with the entire allocated population as the denominator, separate from completers' averages.

Initiatives to increase conversation frequency may create notification dependency even if short-term contact rates rise. Therefore, incorporate randomization to reduce reminders after a certain period or phased withdrawal to verify if behavior is maintained. Success is defined not by maximizing usage, but by the point where results are maintained even as the required support amount decreases.

• Denominator: Do not mix allocated, reached, started, and completed populations
• Dropout reasons: Distinguish technical, burden, safety, indifference, and graduation
• Withdrawal trials: Gradually weaken notifications, conversation frequency, and personalization
• Maintenance judgment: Verify that the target behavior remains even as usage decreases

Before implementation, fix the target behavior, comparison conditions, observation sources, and follow-up dates into a single evaluation contract. Initially limit to low-impact behaviors that can be reversed, comparing AI coaches, one-way support with the same information, and current procedures. Rather than making conversations human-like, identify friction hindering execution and hypothesize which technique moves which stage.

Do not limit the judgment date to immediately post-intervention. Measure the same primary outcome during intervention, immediately post-intervention, when reminders are halved, and after support cessation to create a maintenance curve. If long-term no-intervention controls are difficult due to ethics or operations, choose waitlist, phased introduction, or interrupted time series of objective logs, and display how much can be claimed causally.

Deployment decisions should simultaneously view average improvement, minimum safety/quality standards, reach gaps, dropout, and operational costs. If short-term effects disappear after withdrawal, reassess including costs and dependency for permanent service. Conversely, even a small average difference can justify limited deployment if it increases a critical one-time action via public records and the target/conditions are clear. [3]

• Business Owner: Owns the behavior to change and the path by which change delivers value
• Evaluation Owner: Pre-registers controls, primary outcomes, missing data handling, and follow-up periods
• Behavior Designer: Records goal setting, friction reduction, and feedback as intervention components
• Safety Officer: Monitors inappropriate advice, crisis signals, and handover to humans
• Deployment Meeting: Approves short-term responses and post-withdrawal maintenance separately