Research decision memo

Evidence tier: qualified claim; system evidence qualified.

QUALIFIED CLAIM. Exploratory report. Evidence status is QUALIFIED. Exact-target diagnosis is not_assessed.
Saved interpretation: two-way fixed effects diagnostic estimates the average treatment effect among treated units at -88.5 outcome units (95% confidence interval -92.6 to -84.5). 100 units over 2019-01-2022-12 (4,800 rows). 50 treated units in 1 cohort (2021-01). Each comparison uses never-adopting units against the last period before the intervention, summarized over the supported cohort-by-period comparisons residualized treatment variance. Cross-method comparison: no second estimate of the same target was available to compare, so nothing here corroborates the estimate. What you can say: Evidence status is passed with stated limits.

Evidence fingerprint: 74d19a606f518a05a4e01b0df0fe51dfbbd7c35d1578316f5002bfea0fff63b8

Estimator scalar comparison

Persisted estimator summaries are shown side by side; no pooled average is computed.

Estimator	Role	Estimand	Estimate	Interval	Certificate	Availability	Eligibility
twfe	headline	att_overall	-88.5285	95% [-92.5783, -84.4786]	passed (twfe)	available	not recorded
event_study	diagnostic	event_study	-82.6982	95% [-98.8842, -66.5123]	not_assessed (event_study)	available	not recorded

Typed dynamic effect paths

Intervals are shown only for coordinates recorded as inferential. Point-only, normalized-reference, and unsupported periods carry no interval.

event_study (diagnostic)

Coordinate	Estimate	Interval	Inference status	Support
-24	5.5402	95% [-16.625, 27.7054]	inferential	Coordinate has reportable point and uncertainty estimates.
-23	-8.0082	95% [-29.2712, 13.2548]	inferential	Coordinate has reportable point and uncertainty estimates.
-22	6.059	95% [-13.7875, 25.9055]	inferential	Coordinate has reportable point and uncertainty estimates.
-21	5.8978	95% [-12.6923, 24.4879]	inferential	Coordinate has reportable point and uncertainty estimates.
-20	16.2264	95% [-6.54807, 39.0009]	inferential	Coordinate has reportable point and uncertainty estimates.
-19	8.576	95% [-12.575, 29.727]	inferential	Coordinate has reportable point and uncertainty estimates.
-18	2.8172	95% [-18.068, 23.7024]	inferential	Coordinate has reportable point and uncertainty estimates.
-17	6.4218	95% [-12.918, 25.7616]	inferential	Coordinate has reportable point and uncertainty estimates.
-16	3.7454	95% [-18.5911, 26.0819]	inferential	Coordinate has reportable point and uncertainty estimates.
-15	8.4736	95% [-11.9778, 28.925]	inferential	Coordinate has reportable point and uncertainty estimates.
-14	11.4196	95% [-10.2358, 33.075]	inferential	Coordinate has reportable point and uncertainty estimates.
-13	-4.205	95% [-23.8484, 15.4384]	inferential	Coordinate has reportable point and uncertainty estimates.
-12	-7.7628	95% [-29.0118, 13.4862]	inferential	Coordinate has reportable point and uncertainty estimates.
-11	-2.295	95% [-22.8174, 18.2274]	inferential	Coordinate has reportable point and uncertainty estimates.
-10	4.5116	95% [-16.8639, 25.8871]	inferential	Coordinate has reportable point and uncertainty estimates.
-9	15.1576	95% [-4.61316, 34.9284]	inferential	Coordinate has reportable point and uncertainty estimates.
-8	14.774	95% [-6.13183, 35.6798]	inferential	Coordinate has reportable point and uncertainty estimates.
-7	6.4026	95% [-15.1451, 27.9503]	inferential	Coordinate has reportable point and uncertainty estimates.
-6	8.9494	95% [-12.0752, 29.974]	inferential	Coordinate has reportable point and uncertainty estimates.
-5	22.5228	95% [0.73055, 44.315]	inferential	Coordinate has reportable point and uncertainty estimates.
-4	-1.612	95% [-24.8223, 21.5983]	inferential	Coordinate has reportable point and uncertainty estimates.
-3	2.8722	95% [-17.8939, 23.6383]	inferential	Coordinate has reportable point and uncertainty estimates.
-2	13.4414	95% [-7.82538, 34.7082]	inferential	Coordinate has reportable point and uncertainty estimates.
-1	0	not available (normalized reference)	normalized_reference	Coordinate is the estimator's normalized reference and is fixed at zero.
0	-98.0058	95% [-120.212, -75.7996]	inferential	Coordinate has reportable point and uncertainty estimates.
1	-89.4592	95% [-110.229, -68.6892]	inferential	Coordinate has reportable point and uncertainty estimates.
2	-83.6944	95% [-103.063, -64.3256]	inferential	Coordinate has reportable point and uncertainty estimates.
3	-85.5308	95% [-107.732, -63.3294]	inferential	Coordinate has reportable point and uncertainty estimates.
4	-80.5682	95% [-100.27, -60.8664]	inferential	Coordinate has reportable point and uncertainty estimates.
5	-67.8248	95% [-88.1718, -47.4778]	inferential	Coordinate has reportable point and uncertainty estimates.
6	-91.3948	95% [-112.526, -70.2637]	inferential	Coordinate has reportable point and uncertainty estimates.
7	-84.5948	95% [-105.982, -63.2076]	inferential	Coordinate has reportable point and uncertainty estimates.
8	-85.116	95% [-106.087, -64.1451]	inferential	Coordinate has reportable point and uncertainty estimates.
9	-86.705	95% [-109.314, -64.0959]	inferential	Coordinate has reportable point and uncertainty estimates.
10	-75.7982	95% [-95.2809, -56.3155]	inferential	Coordinate has reportable point and uncertainty estimates.
11	-80.2948	95% [-101.887, -58.7023]	inferential	Coordinate has reportable point and uncertainty estimates.
12	-89.4034	95% [-112.313, -66.4934]	inferential	Coordinate has reportable point and uncertainty estimates.
13	-88.3326	95% [-111.284, -65.3811]	inferential	Coordinate has reportable point and uncertainty estimates.
14	-68.294	95% [-92.1311, -44.4569]	inferential	Coordinate has reportable point and uncertainty estimates.
15	-76.0724	95% [-96.97, -55.1748]	inferential	Coordinate has reportable point and uncertainty estimates.
16	-78.8628	95% [-100.868, -56.858]	inferential	Coordinate has reportable point and uncertainty estimates.
17	-74.3538	95% [-94.657, -54.0506]	inferential	Coordinate has reportable point and uncertainty estimates.
18	-81.2228	95% [-102.824, -59.622]	inferential	Coordinate has reportable point and uncertainty estimates.
19	-79.3016	95% [-101.256, -57.3469]	inferential	Coordinate has reportable point and uncertainty estimates.
20	-78.1952	95% [-98.3276, -58.0628]	inferential	Coordinate has reportable point and uncertainty estimates.
21	-87.3076	95% [-107.884, -66.7313]	inferential	Coordinate has reportable point and uncertainty estimates.
22	-85.8218	95% [-106.927, -64.7166]	inferential	Coordinate has reportable point and uncertainty estimates.
23	-88.6028	95% [-109.733, -67.4721]	inferential	Coordinate has reportable point and uncertainty estimates.

Diagnostic non-rejection does not establish parallel trends, conditional parallel trends, or homogeneity.

Evidence tier

Evidence field	Persisted value
planned_purpose	exploratory
disposition	exploratory
confirmatory_eligible	false
evidence_status	qualified
disagreement_status	not_assessed
confirmatory_agreement_status	not_assessed
blockers	none recorded
statement	QUALIFIED CLAIM. Exploratory report. Evidence status is QUALIFIED. Exact-target diagnosis is not_assessed.
claim_card_id	claim_d6ee8f851ddb3279e52e
target_claim_card_id	claim_d6ee8f851ddb3279e52e
claim_status	qualified
recorded_claim_status	qualified
next_action	Obtain a second exact-target estimate or document why none is available.
headline_inference_reporting	{"certificate_components": {"covariance": {"message": "The estimator reports identified support for nominal inference.", "reason_code": "inferential_support_identified", "state": "passed"}, "interval": {"message": "Confidence intervals use the declared inference level.", "reason_code": "typed_confidence_intervals", "state": "passed"}, "standard_error": {"message": "The estimator reports identified support for nominal inference.", "reason_code": "inferential_support_identified", "state": "passed"}}, "global_vector_certificate_limits_marginal_inference": true, "inference_scope": "marginal_effect", "inference_support": {"joint_status": "identified", "point_estimation": "retained_with_declared_target_weights", "rule": "Cluster-robust uncertainty is nonidentified when either the treated or the comparison arm has fewer than 2 clusters, and qualified when either arm has fewer than 30 clusters. Counts are cluster counts at the level used for inference, never total units or rows.", "schema_version": 1, "status": "identified"}, "joint_support_status": "identified", "joint_vector_limit_applies_to_marginal": true, "marginal_scope_override_applied": false, "marginal_support_status": null, "nominal_inference_available": true, "qualification": null, "reason_codes": [], "reporting_role": "nominal_inference", "status": "available", "support_status": "identified"}
scientific_target	{"anticipation_periods": 0, "assignment_unit": "zip_id", "clean_baseline_rule": "For anticipation L, each baseline must be strictly earlier than cohort - L; a target cannot lie in the cohort's anticipation window, and not-yet-treated controls must remain untreated and outside their own anticipation window through both compared periods.", "explicit_variant_parity_record": false, "outcome": "median_rent", "outcome_input_encoding": "native_outcome_units", "outcome_scale": "additive", "outcome_scale_factor": 1.0, "procedure": "twfe", "procedure_parity_passed": true, "procedure_variant": "default", "target_cells": [], "target_cells_count": 0, "target_cells_hash": null, "target_name": "att", "target_names": [], "target_role": "effect", "target_roles": [{"name": "att", "target_role": "effect"}], "variant_scope_status": "not_applicable"}
variant_scope	{"active": false, "exact_target_cells": false, "honest_did_sensitivity_status": "present_with_persisted_scope", "nonmatching_robustness_excluded_from_claim": [], "plan_excluded_methods": {"artifact_ref": "source.artifacts.json", "detail": "plan_excluded_methods retained in the full evidence artifact", "items": 15}, "planned_robustness": [{"estimator": "event_study", "role": "diagnostic"}], "planned_sensitivity": [], "positive_anticipation": false, "realized_sensitivity_count": 1, "sensitivity_source_relation": null, "statement": "The declared target does not use exact cells or positive anticipation.", "variant_scope_status": "not_applicable"}

Selected-primary estimand

Evidence field	Persisted value
headline_available	true
causal_claim_available	true
descriptive_headline_available	true
headline_availability	causal_claim
headline_available_semantics	legacy_alias_for_causal_claim_available
primary_effect_index	0
descriptive_effect_index	0
claim_status	qualified
estimator	twfe
role	headline
estimand	att_overall
scientific_basis	qualified
ordinary_finite_zero_is_evidence	true
caveats	Predeclared non-gating diagnostics were not assessed: staggered heterogeneity warning
next_action	Obtain a second exact-target estimate or document why none is available.
inference_reporting	{"certificate_components": {"covariance": {"message": "The estimator reports identified support for nominal inference.", "reason_code": "inferential_support_identified", "state": "passed"}, "interval": {"message": "Confidence intervals use the declared inference level.", "reason_code": "typed_confidence_intervals", "state": "passed"}, "standard_error": {"message": "The estimator reports identified support for nominal inference.", "reason_code": "inferential_support_identified", "state": "passed"}}, "global_vector_certificate_limits_marginal_inference": true, "inference_scope": "marginal_effect", "inference_support": {"joint_status": "identified", "point_estimation": "retained_with_declared_target_weights", "rule": "Cluster-robust uncertainty is nonidentified when either the treated or the comparison arm has fewer than 2 clusters, and qualified when either arm has fewer than 30 clusters. Counts are cluster counts at the level used for inference, never total units or rows.", "schema_version": 1, "status": "identified"}, "joint_support_status": "identified", "joint_vector_limit_applies_to_marginal": true, "marginal_scope_override_applied": false, "marginal_support_status": null, "nominal_inference_available": true, "qualification": null, "reason_codes": [], "reporting_role": "nominal_inference", "status": "available", "support_status": "identified"}
scientific_target	{"anticipation_periods": 0, "assignment_unit": "zip_id", "clean_baseline_rule": "For anticipation L, each baseline must be strictly earlier than cohort - L; a target cannot lie in the cohort's anticipation window, and not-yet-treated controls must remain untreated and outside their own anticipation window through both compared periods.", "explicit_variant_parity_record": false, "outcome": "median_rent", "outcome_input_encoding": "native_outcome_units", "outcome_scale": "additive", "outcome_scale_factor": 1.0, "procedure": "twfe", "procedure_parity_passed": true, "procedure_variant": "default", "target_cells": [], "target_cells_count": 0, "target_cells_hash": null, "target_name": "att", "target_names": [], "target_role": "effect", "target_roles": [{"name": "att", "target_role": "effect"}], "variant_scope_status": "not_applicable"}
effect	{"ci_level": 0.95, "ci_lower": -92.57829485014179, "ci_upper": -84.47863848319153, "df": 99.0, "estimate": -88.52846666666666, "name": "att", "p_value": 3.2291299862775403e-66, "p_value_display": "3.23e-66", "std_error": 2.0410208572390327}
claim_card_refs	{"card_ids": ["claim_d6ee8f851ddb3279e52e"], "content_hashes": ["d6ee8f851ddb3279e52e05df96efc28d1ab80bbabd1d2f60e1c7243a17f3939e"]}

Population, support, controls, baseline, and weights

Evidence field	Persisted value
estimand	att_overall
population	observed treated units
scientific_target	{"anticipation_periods": 0, "assignment_unit": "zip_id", "clean_baseline_rule": "For anticipation L, each baseline must be strictly earlier than cohort - L; a target cannot lie in the cohort's anticipation window, and not-yet-treated controls must remain untreated and outside their own anticipation window through both compared periods.", "explicit_variant_parity_record": false, "outcome": "median_rent", "outcome_input_encoding": "native_outcome_units", "outcome_scale": "additive", "outcome_scale_factor": 1.0, "procedure": "twfe", "procedure_parity_passed": true, "procedure_variant": "default", "target_cells": [], "target_cells_count": 0, "target_cells_hash": null, "target_name": "att", "target_names": [], "target_role": "effect", "target_roles": [{"name": "att", "target_role": "effect"}], "variant_scope_status": "not_applicable"}
support	{"baseline": "last_pre_period", "calendar_periods": {"artifact_ref": "source.artifacts.json", "detail": "calendar_periods retained in the full evidence artifact", "items": 48}, "cohorts": ["2021-01"], "control_group": "never_treated", "event_window": [-24, 23], "trimmed_cells": [], "unsupported_cell_policy": "fail_or_record_trim"}
control_group	never_treated
baseline	last_pre_period
weights	{"meaning": "none", "normalization": "none", "reason": null, "source": null, "supported": true}

Procedure certificate

Evidence field	Persisted value
item 1	{"estimator": "twfe", "parity_passed": true, "procedure": "twfe", "role": "headline", "state": "passed", "variant": "default"}
item 2	{"estimator": "event_study", "parity_passed": false, "procedure": "event_study", "role": "diagnostic", "state": "not_assessed", "variant": "default"}

Assumptions and waivers

Evidence field	Persisted value
records	[{"status": "asserted"}, {"status": "asserted"}, {"status": "asserted"}, {"status": "asserted"}, {"status": "asserted"}, {"status": "not_applicable"}, {"status": "asserted"}, {"status": "asserted"}]
waivers	none recorded
missing_assumptions	none recorded
not_yet_treated_assumption	Not-yet-treated membership is date-specific control eligibility, not evidence for cross-cohort parallel trends or no anticipation. Both identifying assumptions must be declared separately.

Source-bound diagnostics

Evidence field	Persisted value
interpretation_rule	Callaway-Sant'Anna pretrend diagnostics use the estimator-matched aggregated dynamic-lead zero-restriction Wald, not the native did::att_gt pretest. Nonrejection does not establish parallel trends or conditional parallel trends.
records	[{"evidence_gate": false, "interpretation": "Fewer than two finite estimates share the headline causal target. Target or specification differences establish non-comparability but do not explain the numerical gap without a quantified bridge or decomposition.", "name": "estimator_disagreement_diagnosis", "result": {}, "semantic_result": "not_assessed", "status": "assessed"}, {"evidence_gate": false, "name": "fit_free_panel_validation", "semantic_result": "not_assessed", "status": "passed"}, {"computation_status": "assessed", "diagnostic_family": "pretrend_joint_test", "evidence_gate": false, "falsification_outcome": "not_rejected", "interpretation": "The joint test does not reject; this is not proof of parallel trends.", "name": "pretrend_joint_test", "result": {"denominator_df": 99, "numerator_df": 5, "p_value": 0.1473251072920069, "reason": null, "reason_code": null, "reference_distribution": "F(5, 99)", "semantic_result": "not_rejected", "statistic": 1.6763039188981121, "status": "assessed"}, "semantic_result": "not_rejected", "status": "assessed"}, {"computation_status": "not_assessed", "evidence_gate": false, "name": "staggered_heterogeneity_warning", "reason": "Exactly one completed diagnostic-only TWFE result is required for this warning.", "reason_code": "diagnostic_twfe_unavailable", "semantic_result": "not_assessed", "status": "not_assessed"}, {"evidence_gate": false, "name": "support_audit", "semantic_result": "not_assessed", "status": "passed"}, {"computation_status": "assessed", "evidence_gate": false, "falsification_outcome": "not_rejected", "name": "time_placebo", "result": {"message": "Placebo (fake adoption 2 periods early, pre-data only): effect=0.9714, p=0.87.", "semantic_result": "not_rejected", "status": "assessed"}, "semantic_result": "not_rejected", "status": "assessed"}]
artifact_ref	source.artifacts.claim_cards

Structured disagreement

Evidence field	Persisted value
reporting_rule	Report the selected primary and each estimator contribution separately; do not average estimator estimates.
status	not_assessed
confirmatory_gate	false
interpretation	Fewer than two finite estimates share the headline causal target. Target or specification differences establish non-comparability but do not explain the numerical gap without a quantified bridge or decomposition.
next_action	Obtain a second exact-target estimate or document why none is available.
contribution_counts	{"complementary": 1, "degenerate": 0, "exact_target": 0, "failed": 0, "incomparable": 1}
concordance	{"confirmatory_agreement": {"analysis_profile": "quick", "basis": "simultaneous exact-target difference intervals against a predeclared margin", "decision_purpose": "exploratory", "equivalence_margin": 10.0, "estimator_pooling": "none", "independent_family_count": 0, "joint_inference_valid": false, "p_value_role": "none", "purpose_eligible": false, "selection_role": "none", "status": "not_assessed"}, "confirmatory_available": false, "confirmatory_status": "not_assessed", "descriptive_available": false, "descriptive_concordance": {"basis": "verified exact-target point estimates in outcome units", "directions": {}, "estimate_count": 0, "estimator_pooling": "none", "maximum_pairwise_distance": null, "numerical_zero_tolerance": 1e-12, "p_value_role": "none", "policy_benchmark": {}, "policy_threshold": null, "practical_materiality": {}, "practical_significance_threshold": 10.0, "selection_role": "none", "status": "not_assessed", "target_relation": null, "target_verified": false}, "descriptive_status": "not_assessed", "estimate_count": 0, "independent_family_count": 0, "statement": "The persisted descriptive concordance and confirmatory agreement tiers are not assessed. The report does not reconstruct either tier from lower-level evidence.", "tier": "not_assessed"}
heterogeneity_attributions	none recorded
heterogeneity_interpretation	Persisted equality tests are diagnostic only; equality non-rejection does not establish homogeneity.
artifact_ref	source.artifacts.json

Degeneracy

Evidence field	Persisted value
realized_category	none
ordinary_finite_zero_is_evidence	true
categories	[{"code": "none", "meaning": "Finite ordinary evidence is available; an exact zero remains evidence."}, {"code": "structural_nonidentification", "meaning": "The declared effect is not identified on the realized support."}, {"code": "scientific_empty_selection", "meaning": "A selection procedure retained no scientifically reportable effect."}, {"code": "point_only_inference", "meaning": "A point estimate exists but valid uncertainty is unavailable."}, {"code": "numerical_failure", "meaning": "Numerical failure prevents a scientifically reportable estimate."}]

Formal sensitivity

Evidence field	Persisted value
policy	optional_declared
result_count	1
conclusions	[{"conclusion": {}, "procedure": "honest_did", "status": "skipped"}]
artifact_ref	source.artifacts.json

Confirmatory requirements

Evidence field	Persisted value
confirmatory_requested	false
overall_eligible	false
overall_disposition	exploratory
blockers	none recorded
requirements	[{"code": "declared_profile_and_purpose", "recorded_evidence": {"analysis_profile": {"name": "quick", "source": "recorded", "statement": "Quick profile: intended for rapid exploratory screening. Successful computation does not make this profile confirmatory."}, "planned_purpose": "exploratory"}, "requirement": "Confirmatory use requires a confirmatory purpose and operating profile."}, {"code": "procedure_certificates", "recorded_evidence": {"artifact_ref": "source.artifacts.claim_cards", "certificate_count": 2}, "requirement": "Every relied-on procedure must retain its persisted certificate and any required parity result."}, {"code": "assumptions_and_waivers", "recorded_evidence": {"assumption_count": 8, "missing_assumptions": []}, "requirement": "Confirmatory use cannot rely on an exploratory waiver or an unresolved identifying assumption."}, {"code": "source_bound_diagnostics", "recorded_evidence": {"artifact_ref": "source.artifacts.claim_cards", "diagnostic_count": 6}, "requirement": "Diagnostics must retain their source and canonical rejected, not_rejected, or not_assessed result."}, {"code": "exact_target_agreement", "recorded_evidence": {"artifact_ref": "source.artifacts.json", "confirmatory_agreement_status": "not_assessed", "diagnosis_status": "not_assessed"}, "requirement": "Any confirmatory agreement claim requires the persisted exact-target, joint-inference assessment; estimator estimates are never averaged."}, {"code": "formal_sensitivity", "recorded_evidence": {"artifact_ref": "source.artifacts.json", "policy": "optional_declared", "result_count": 1}, "requirement": "The declared sensitivity policy must be satisfied by persisted formal sensitivity results and their recorded source."}]

Persisted evidence provenance

Evidence field	Persisted value
run_id	run_29478d8d9c2b4059b02a
plan_id	plan_eb1d28fda7743a349f3b
plan_hash	eb1d28fda7743a349f3bb5b831a97422189ad15320d4f7ca4b361c3430c294a5
result_hash	fe08fa1b7f118b2722ab3b9c19402f2e4c935daf4453c3874dd92429ffa3b572
report_semantics_hash	26601b06bdd4965c724631926b708b565468c16c10d5d0ee1178e147b8b271ed
full_evidence_artifact_ref	source.artifacts.json
claim_cards_artifact_id	claim_cards_848e8fdf90aae64a9fbf
claim_cards_content_hash	848e8fdf90aae64a9fbff5da8b1624aae2e3e6f871b0dc0baacc9329d3c3fb0f
report_mode	full
rendered_from_persisted_results	true
estimator_calls	0
analysis_engine_provenance	{"implementation": "causal-didacto", "provenance_role": "analysis_engine", "release_version": "0.3.0rc18", "source": "persisted_run.engine", "source_revision": null, "source_revision_recorded": false}
renderer_runtime_provenance	{"immutable_release_environment_complete": false, "implementation": "causal-didacto evidence renderer", "provenance_role": "renderer_runtime", "release_version": "0.3.0rc18", "release_version_source": "installed_distribution", "source": "installed_distribution", "source_revision": null, "source_revision_source": "not_recorded"}
artifacts	{"claim_cards": "claim_cards-ee51982e9866714b42df954a6919f0ea14d5f1f8cd683fcdaabdf39fcb1f44c3.json", "html": "report-full-dcdc9f4ab88f3586d8753b049313da064a8f17b30ed5736376bc61fdceec12f3.html", "json": "evidence-40326c601de285491e58d027953b5cdb812bb19eb8a019b8676e180e981c72aa.json", "markdown": "report-full-ef3d2ce5509eab2d2142e9b76bf30b9a6f6bda327903db5fad230117d0cf1562.md"}
artifact_hashes	{"claim_cards": "ee51982e9866714b42df954a6919f0ea14d5f1f8cd683fcdaabdf39fcb1f44c3", "html": "dcdc9f4ab88f3586d8753b049313da064a8f17b30ed5736376bc61fdceec12f3", "json": "40326c601de285491e58d027953b5cdb812bb19eb8a019b8676e180e981c72aa", "markdown": "ef3d2ce5509eab2d2142e9b76bf30b9a6f6bda327903db5fad230117d0cf1562"}