Not whether nudges work, but how much they shrink

Mertens et al. reported a moderate average effect in a broad meta-analysis integrating choice architecture interventions, while also demonstrating heterogeneity by intervention type and domain.[1] This is a crucial synthesis across nudge research.

A correction was later published for this paper, involving revisions to study classification and data.[2] Do not refer only to the original paper while ignoring the correction.

The average effect aggregates different interventions such as defaults, information presentation, placement changes, and social norms. Do not extrapolate a total average including different mechanisms to your company's specific wording changes.

To convert effect sizes into absolute numbers, baseline rates are required. The business value of the same relative effect differs depending on whether the original behavior rate is high or low.

We examine not only the existence of effects but also prediction intervals and heterogeneity. We do not hide the possibility of zero or adverse effects in other settings behind the significance of the average.

• Interventions: Separate defaults, wording, order, and social norms
• Target behavior: Define confirmed actions, not just clicks
• Baseline rate: Fix the absolute rate before the intervention
• Heterogeneity: Report both the average and the range in other settings

Maier et al. re-analyzed Mertens et al.'s meta-analysis by adjusting for publication bias, concluding that there is no evidence supporting the average effect of nudges after adjustment.[3] Interpretations change based on model selection even with the same literature.

This is not proof that all nudges are ineffective. Publication bias correction also depends on assumptions and does not negate registered large-scale trials for specific interventions.[3]

We do not select only the positive original meta-analysis or the negative re-analysis, but show how conclusions change under different assumptions, allowing decision-makers to see uncertainty.

The same problem arises if only good results are shared in our own small-scale trials. Register all trials and keep records of no difference, adverse effects, and interim stops in the ledger.

Pre-fix the primary behavior, period, and analysis to avoid testing multiple metrics and selecting only the good ones as primary outcomes. Send exploratory results to the next confirmatory trial.

DellaVigna and Linos analyzed numerous large-scale randomized trials by administrative nudge organizations, showing that the average effect in practical trials is considerably smaller than the average reported in academic papers.[4]

Differences arise from publication selection, intervention optimization, target population, implementation quality, and scale.[4] When transferring academic effects to practice, do not use the same rates for budget planning.

Small absolute differences can be valuable if the target population is large and implementation costs are low. Conversely, even with a high effect rate, adoption is impossible if it does not lead to important actions and maintenance costs are high.

Separate intensive operations during the trial phase from full-scale deployment. If individual follow-up, skilled staff, or special data supported the effect, include these in the conditions for replication after scaling up.

Monitor effects through phased introduction with control groups or alternating implementation even after deployment. Effects cannot be separated from seasonality, policy changes, or publicity impacts by looking only at before and after introduction.

• Absolute difference: How many percentage points the behavior changed
• Target scale: How many people/cases are reached
• Implementation cost: Includes design, delivery, maintenance, and verification
• Sustainability: Does the difference remain after expansion and repetition?

Clicking a notification or starting a form is an intermediate action. Track until the confirmed target action such as application completion, preventive action, payment, or continuation of learning.

Even if behavior occurs immediately after the nudge, there is a possibility of returning later, forgetting, or avoiding other paths. Measure the duration of the effect and habituation upon repetition.

Even if the target behavior increases, quality may decrease. Check for side effects such as rushed consent, not reading content, or increased inappropriate applications.

Check if increasing one behavior reduces another important behavior. Measure attention displacement, such as important warnings being buried by increased notifications.

Check if choices that individuals later regret are increasing by tracking cancellations, complaints, and re-choices. Do not estimate welfare based solely on behavior rates.

The effect of the same nudge varies by language, digital skills, existing knowledge, and time constraints. Average improvements may widen access gaps.

While defaults help choices, they strongly affect those who find it difficult to change. Audit who benefits from the default and whose choices are fixed.

Do not consider no difference in small sample groups as equitable. Pre-determine important groups and limit expansion if power is insufficient, marking them as unconfirmed.

Personalized nudges may increase effects but increase the need for sensitive inference and the risk of manipulation. Design necessary data, consent, explanation, and opt-out.

Do not use subgroup effects to strongly guide only vulnerable people. Separate the need for support from commercial conversion rates and set lower bounds for rights and autonomy.

Nudges that make choices clear are not the same as dark patterns that hide rejection, use guilt, and hinder cancellation. Review purposes and means.

Check if it aligns with the individual's long-term goals, if information is hidden, if rejection is equally easy, and if it can be withdrawn.

Social norm messages must be fact-based. Do not change behavior using fictitious majorities or misleading comparisons.

When indicating urgency or scarcity, verify deadlines and grounds. Do not call permanent countdowns nudges.

Ethical review should be conducted before the experiment, not after effects are observed. Even if high conversion rates are found, deceptive means will not be adopted.

• Transparency: Understandable what is being prompted
• Symmetry: Friction between acceptance and rejection is not unfairly different
• Withdrawal: Easily returnable later
• Alignment: Separate individual benefit from provider benefit

In business plans, use cautious effects close to practical trials rather than academic averages as the benchmark.[4] Show points where judgments change under zero, cautious, and baseline scenarios.

Even if production costs are low, include legal verification, data integration, delivery, inquiries, and side effect response. Screen changes that appear free also have operational costs.

Do not overvalue the value of a single behavior change. Adjust for conversion from clicks to confirmed outcomes, losses from erroneous actions, and overlap with other measures.

When applying small effects to a large target population, consider target overlap and contact fatigue. Do not simply add the effects of multiple nudges.

Set lower bounds for effects and maximum allowable costs in advance. Even if statistically significant, do not expand if the absolute difference is below the lower bound.

Pre-register target behavior, primary metrics, period, groups, minimum value, side effects, and stopping conditions before the experiment. Do not change success definitions after seeing results.

When testing multiple options, perform simultaneous comparison and correction, and re-verify the winner in a separate experiment. Do not permanently adopt the accidental best option.

In production, monitor effect decay, habituation, complaints, and subgroup differences. Do not stop measurement just because of initial success.

Withdraw if the lower bound is not met, effects are adverse, there is deception, or gaps widen. Also verify the need to return settings or choices to participants after withdrawal.

The value of nudges is not the magnitude of average effects. It is verifying conditions under which autonomy is maintained at low cost and confirmed behaviors are improved, assuming uncertainty and small practical effects.

The preregistration record separates intervention version, target behavior, target group, and withdrawal date from decision criteria: minima for practical effect and autonomy and maxima for cost and group disparity. An intervention stops and returns to the pre-intervention neutral condition if either minimum is missed, either maximum is exceeded, or the withdrawal date passes. Restart requires a redesigned preregistered test approved by an ethics reviewer not involved in the original implementation and by the analytics owner.