DID-ACTO / FIELD GUIDE
Tests, sensitivity and inference
Which checks exist, what they test, when they apply, and how to read uncertainty without overstating the evidence.
Available tests and diagnostics
| Check | Question it addresses | Availability and interpretation |
|---|---|---|
| Joint pretrend tests | Test whether eligible pre-treatment coefficients are jointly zero. | Source-matched Wald test for staggered CS; cluster F test for plain event study and the exact Sun–Abraham profile. Governed gating needs at least two supported leads. Read a non-rejection alongside the plotted leads and the test’s ability to detect a meaningful difference. |
| Placebo in time | Move the intervention into an eligible pre-treatment window as a falsification check. | Available when predeclared and supported. A substantive rejection gates the evidence; a structurally unavailable check is not_assessed. Timing assumptions still matter. |
| Unit-permutation and in-space placebos | Compare the observed result with reassigned treatment or donor-placebo results. | Procedure-specific and dependent on the assignment comparison being meaningful. Synthetic methods preserve the treated-group size and report RMSPE-ratio permutation diagnostics. |
| HonestDiD / Rambachan–Roth sensitivity | Calculate intervals under specified departures from parallel trends. | Formal relative-magnitude and smoothness procedures plus exact supported directional/composite cells. Requires a certified event path, full covariance and sufficient stable support; see the detailed families below. |
| Dynamic heterogeneity test | Jointly test equality across post-treatment event-time effects. | Uses supported coordinates and joint covariance. This concerns variation over event time; it is not automatically a test of differences between cohorts. |
| Cross-estimator equivalence and conflict | Compare eligible estimates of the same realized target using simultaneous difference intervals. | Confirmatory equivalence needs a predeclared margin and eligible uncertainty. Automatic comparison uses a conservative standard-error bound and Bonferroni adjustment. Overlapping marginal intervals do not establish equivalence. |
| Hansen overidentification | Assess compatibility of the overidentifying moment restrictions used by efficient staggered GMM. | Available on the eligible GMM path alongside clean-comparison support and convergence checks. Review the assignment assumptions separately from the moment-restriction test. |
| Propensity overlap and weight diagnostics | Check common support, effective sample size, separation and concentration of weights. | Relevant to DR, IPW and conditional procedures. Invalid support can block fitting; poor overlap calls for a design or sample review before estimation. |
| Synthetic donor and fit diagnostics | Review pre-treatment RMSPE, donor concentration, effective donor count, solver convergence and placebo feasibility. | Synthetic control also includes leave-one-donor and contamination/spillover diagnostics. Use these to identify fragile donor choices, then review possible spillovers using knowledge of the locations and intervention. |
| Panel, cohort and event support checks | Validate data structure and identify which comparisons support each requested effect. | Checks include treatment timing, baseline availability, contributing cells and event-window support. Thin or unsupported coordinates may retain points while losing inferential eligibility. |
| Randomization design checks | Validate the assignment contract, admissible reassignments and p-value resolution. | Separate randomized workflow. Lin adjustment adds covariate-rank checks; few clusters and a small assignment space limit what can be resolved. |
Read a pretrend test correctly
The staggered CS workflow tests source-matched pre-treatment coefficients jointly with their covariance. Plain event study and the exact Sun–Abraham profile use a cluster F reference. The source, controls, baseline and event support matter; a convenient regression with different comparisons is not an interchangeable gate.
At least two inferential leads are needed for the governed gating diagnostic. If support is inadequate, the result is not_assessed, rather than “passed.” A non-rejection can reflect limited power and never verifies the parallel-trends assumption. A failed gating pretrend prevents the stronger evidence interpretation.
Formal HonestDiD sensitivity
Rambachan–Roth sensitivity asks how an interval changes when specified violations of parallel trends are allowed. The canonical procedures are relative magnitude (conditional least-favorable, C-LF) and smoothness (fixed-length confidence intervals, FLCI). Additional directional and composite restrictions have narrower exact support.
The governed staggered source is a supported Callaway–Sant’Anna dynamic path with a universal event −1 baseline. It needs full covariance, at least two estimated pre-period coefficients and one post-period coefficient, and a stable supported cohort window. Sun–Abraham is not an authorized formal-sensitivity source. A common-adoption CS request does not inherit the staggered HonestDiD certification.
The default target is the first post-treatment effect, event_0. Its interval is not automatically a sensitivity interval for an overall ATT. The source and target must be stated explicitly.
All nine HonestDiD restriction families
| Family | Identifier | Restriction / inference |
|---|---|---|
| Relative magnitude | relative_magnitude | Bounds post-treatment trend violations relative to pre-treatment first differences; C-LF inference. |
| Relative magnitude + sign | relative_magnitude_sign | Adds a declared positive or negative bias direction; C-LF. |
| Relative magnitude + monotonicity | relative_magnitude_monotonicity | Adds a declared increasing or decreasing path; C-LF. |
| Smoothness | smoothness | Bounds changes in the differential-trend slope; fixed-length confidence intervals (FLCI). |
| Smoothness + sign | smoothness_sign | Adds a bias direction; C-LF. Some exact negative-direction cells remain refused. |
| Smoothness + monotonicity | smoothness_monotonicity | Adds a path direction; C-LF. Some exact decreasing-direction cells remain refused. |
| Smoothness + relative magnitude | smoothness_relative_magnitude | Applies a relative bound to second differences; C-LF. |
| Smoothness + relative magnitude + sign | smoothness_relative_magnitude_sign | Adds a declared bias direction to the composite restriction; C-LF. |
| Smoothness + relative magnitude + monotonicity | smoothness_relative_magnitude_monotonicity | Adds a declared path direction to the composite restriction; C-LF. |
Descriptive sensitivity is a different output
The low-level relative_magnitude_sensitivity helper provides a descriptive plug-in diagnostic. It is separate from formal honest_did_sensitivity; it does not promise conservative confidence coverage. Ask the assistant to identify which procedure generated the output.
Available uncertainty methods
| Inference method | Where it is used | What to know |
|---|---|---|
| Cluster-robust covariance | TWFE, event study, exact Sun–Abraham, ETWFE, BJS and other eligible procedures | Clustering convention, degrees of freedom and normal/t reference are method-specific. Few clusters remain a limitation. |
| Influence-function inference | CS panel and exact conditional repeated-cross-section profiles | Full covariance supports aggregate effects and contrasts; missing joint support prevents a whole-path inference claim. |
| Multiplier-bootstrap simultaneous bands | Supported CS vector outputs and exact conditional RCS vector profiles | Joint coverage across the declared vector is different from separate pointwise intervals. |
| Unit bootstrap | DR DiD, IPW DiD and low-level changes-in-changes | Resamples complete unit trajectories; unsuccessful required draws are not silently discarded. |
| Analytical GMM | Gardner did2s and efficient staggered GMM | Uses each procedure’s supported covariance and weighting contract. |
| Newey–West HAC | Interrupted time series | Addresses heteroskedasticity and serial correlation within the specified time-series model. |
| Exact / Monte Carlo randomization | Randomized difference in means and eligible Lin robustness | Two-sided sharp-null tests under the declared assignment design; not a blanket test of arbitrary average-effect nulls. |
| Placebo RMSPE inference | Synthetic control and synthetic DiD | Permutation p-values are diagnostic; placebo-SD normal intervals are explicitly heuristic. |
Estimator agreement is a separate assessment
DiD-Acto first checks that estimates share the same realized target: population, supported cells, sample and aggregation weights. Merely calling both estimates “ATT” is insufficient. Different targets can be complementary without being directly comparable.
Confirmatory equivalence requires certificate-eligible procedures, a predeclared margin and valid simultaneous difference intervals. Automatic comparison uses a conservative bound on the difference standard error with Bonferroni adjustment; it does not refit to estimate a cross-method covariance. Outcomes include equivalent, conflict, inconclusive and not assessed. A favorable point comparison or overlapping marginal confidence intervals cannot substitute for this assessment.
What runs, what skips, and what you receive
The immutable plan names the scheduled checks and optional work. Exploratory and confirmatory purposes impose different requirements; exact profiles can also fix their own resampling contract. Do not assume that every check on this page runs for every method.
Read each check’s status, statistic or estimate, uncertainty where available, support and reason for exclusion. Distinguish a substantive rejection from a computational failure and from a check that could not be assessed. Choose the checks before fitting, then report all scheduled findings together.
Before running, list every scheduled test and sensitivity check by name. For each, state its null or restriction, source estimator, target, support requirements and what a rejection or not-assessed result would mean. Afterward, report skipped checks with their reasons and distinguish formal inference from descriptive or heuristic diagnostics.