Design-first practice for causal thinking without RCTs

Hernán and Robins provided a framework for designing observational data as a mimic of a target trial in comparative effectiveness research where RCTs are unavailable.[1] By pre-specifying eligibility criteria, intervention strategies to compare, assignment, start of follow-up, outcomes, causal contrasts, and analysis plans, you can see the danger of post-hoc question fitting to available records.

Time zero is particularly critical. Designs that include only those who survived until treatment initiation in the treatment group, use future information to determine eligibility, or have misaligned follow-up start times for comparison groups cannot be fixed by data volume. In business initiatives, if the start point is not aligned across target identification, proposal, acceptance, and implementation, effects and selection processes will be conflated.

After analysis begins, log any deviations from the target trial. If eligibility was narrowed to fit the data, follow-up start was shifted, or intervention definitions were changed after outcome confirmation, re-examine whether these changes affect comparison groups asymmetrically. If necessary time-point information is not recorded, do not change the estimator; instead, determine that the question cannot be identified with the current data.[1]

Rosenbaum and Rubin demonstrated that propensity scores, defined as the probability of treatment conditional on observed covariates, allow balancing observed covariate distributions through matching or stratification under certain assumptions.[2] The ability to summarize numerous background variables into a single assignment probability is useful in practice for constructing comparison groups.

However, similar scores do not mean people are essentially the same. Confounding remains if unrecorded factors such as decision-makers' intuition, symptom severity, or sales representatives' expectations influence both intervention and outcome. Do not extrapolate to individuals with poor overlap, check balance after adjustment, and conduct sensitivity analyses to see how far conclusions change with unmeasured confounding.

Propensity score acceptance criteria should be based on covariate balance and overlap after adjustment, not predictive accuracy. Forcing weights in regions where only the treatment group exists produces results based on model extrapolation rather than comparability. Narrow the analysis scope to areas with ensured overlap; if conclusions reverse under realistic alternative explanations for unmeasured factors, they should not be established as effects for decision-making.[2]

Imbens and Lemieux practically organized regression discontinuity designs where treatment is determined by thresholds such as scores or age.[3] If the points immediately above and below the threshold are continuous relative to other points, local causal effects can be estimated from the outcome difference in that vicinity. The institutional assignment rule provides the basis for comparison.

Validity requires that subjects cannot precisely manipulate the threshold, other institutions do not switch at the same threshold, and results do not overly depend on bandwidth or functional form selection. The results primarily reflect effects near the threshold; generalizing to distant subjects requires additional assumptions. Do not declare identification based solely on clean graphs.

In regression discontinuity, check beforehand whether variables other than the intervention are discontinuous at the threshold, and examine the distribution of predetermined attributes or subject counts. If signs or conclusions change significantly when varying bandwidth or functional form, specification choice, rather than local effects, becomes the alternative explanation. If the threshold assignment rule changes to a different institution, do not transfer previous estimates to the new system; reconstruct them from identification conditions.[3]

Target trials explicitly design comparisons but do not exclude unmeasured confounding, and propensity scores require assumptions of observed covariates and overlap.[1][2] Therefore, the following negative controls are not tests to prove causality, but auxiliary diagnostics to find associations that the current design cannot explain and to cautiously weaken claims.

As a hypothetical example, if an outcome that should only change after intervention appears in the pre-intervention period, suspect time misalignment, subject selection, or common factors. Also, if outcomes predetermined to be unaffected by the intervention move in the same direction, broad confounding or measurement changes are alternative explanations. Controls must be selected before viewing results, and selection rationale and expected direction must be registered.

Even if no anomalies appear in controls, do not conclude that unmeasured confounding is absent, as the types and sensitivity of detectable biases are limited. If anomalies appear, do not add estimators to eliminate them; instead, investigate in detail which aspect of time zero, subject definition, covariate acquisition timing, or assignment records moved the control, and return to the design table.

Once point estimates are obtained, list which assumptions, if slightly changed, would reverse the decision. Recommended categories are: usable only for direction, usable if limited to a specific scope, requires additional data, and reported as association only. Do not dichotomize causal claims based solely on significance; record approval in stages. Each stage corresponds to the type of judgment that may be used.

Reversal boundaries are recorded by changing one factor at a time: candidates for unmeasured factors, subjects excluded from overlap, missing data handling, follow-up start, and for regression discontinuity, bandwidth and manipulability. Conditions where multiple weak changes occur simultaneously are also listed separately. If judgments change under realistically possible modifications, increasing the precision of estimates never strengthens confidence. Report adoption deadlines alongside.

What is passed to approvers is not only the best-specification results, but also the weakest assumptions, negative control results, reversed specifications, and available subject scope. When new institutional changes or confounding candidates are found, re-evaluate based on whether this information crosses the reversal boundary. If it does, downgrade the causal claim and link the reason for change to past adoption decisions. Track pending initiative decisions as well.

• CLAIM: Fix subject, intervention, comparison, period, and how much of direction/magnitude to use
• NEGATIVE CONTROL: Define timepoints/outcomes that should not move and the types of bias detectable
• REVERSAL BOUNDARY: List points where judgments change due to changes in assumptions, subjects, missing data, or specifications
• ALLOWED LABEL: Retain how far claims can go: causal, limited causal, further investigation, or association only

In analysis reviews, first place the causal graph and target trial table. Next, confirm whether assignment can be explained by observed covariates, depends on institutional thresholds, or involves changes close to natural experiments. If the identification basis cannot be written here, even complex estimators likely only refine predictive associations rather than causal effects.

Final reports should list not only point estimates, but also subject population, comparison, follow-up period, exclusions, missing data, overlap, alternative specifications, negative controls, and sensitivity to unmeasured confounding. For decision-makers, show together what assumptions are needed to use this conclusion and under what conditions it should be withdrawn. The value of causal inference lies not in strengthening assertions, but in organizing uncertainty into actionable forms.

In decision-making, separate the person responsible for adopting estimates from the person responsible for auditing assumptions. If any of subject population, time zero, comparison, overlap, or sensitivity analysis fails to meet pre-specified criteria, withdraw the causal effect claim and revert to reporting associations. When new confounding candidates or institutional changes are found, restart analysis and do not maintain conclusions based solely on past significance. Record withdrawal reasons and restart conditions in the same report.

• Question: Fix subject, intervention, comparison, outcome, period, and effect type in one sentence
• Time: Align eligibility determination, assignment, and follow-up start to the same origin
• Identification: Explicitly state the institutions, records, and assumptions supporting comparability
• Robustness: Report overlap, alternative specifications, sensitivity analysis, and generalization scope