Before analysis:
- Fix what must stay constant.
- Quantitative: estimand, population, time window, support/inconclusive rule.
- Evaluation/ranking: decisive criteria, exclusions, tie-breakers.
- Research synthesis: claim and source/inclusion criteria.
- Identify the 2-4 defensible forks most likely to change the answer.
After the primary analysis:
- For each important fork, name the choice made and reasonable alternatives.
- Re-run alternatives that could plausibly change the conclusion.
- Report:
- STABLE: conclusion survives.
- SENSITIVE: magnitude/ranking changes but action does not.
- FLIPS: conclusion/action changes; name the fork responsible.
- Report the range/distribution where meaningful, not merely the preferred run.
- Distinguish computational reproducibility from analytical stability.