Dex repoint-flip plan — the campaign's final phase

Scoped 2026-07-07. The build track is done: every shipped paper's data now lives in a dex (WSPR was the last, ab03e55). This plan scopes the flip — Mike's contained final step where each paper stops pulling its raw source and instead reads its data through a dex slot, so every datum traces to a registered source ([[feedback_provenance_every_datum]], [[feedback_organize_not_process]]).

Per Mike's sequencing this is ONE contained phase at the end, done paper by paper, not a running hazard during the build.

The load-bearing finding (changes the "easy plumbing" assumption)

docs/paper-source-inventory.md called repoint "near-pure plumbing." That was wrong in one important way: there is no dex read-path today. The storehouse layer (event_storehouse.py) is write-only by design (write_dossier, rebuild_index_from_disk — no load, no iter, no get_slot). Every shipped paper reads its data straight from Postgres observations, an external API, or a local cache. Nothing in src/ reads a dex kind's data back out for a consumer.

provenance.py resolves a source's metadata (resolve_for_kind → name/url/access) but not its rows. So resolving "where did this come from" works; "give me the data" does not exist yet.

Consequence: Phase 0 of the flip is building the dex reader. It is the first, load-bearing task, and it unblocks every paper.

What "flip done" means for one paper

A paper is flipped when:

Its data-acquisition step reads through a dex slot, not from the raw source directly.
Re-running the analysis reproduces the published numbers (the dex holds the same measured data, so results are preserved by construction — see the two read modes below).
Provenance resolves through the dex slot's registered source (resolve_for_kind).

The reproduction guarantee rests on a clean framing that the WSPR "carry-both" design already enables:

The dex slot is the provenance anchor; a paper reads one of its two surfaces, but always through the slot:

Working surface — the dex's organized/aggregated data (e.g. the WSPR census month×band series, the event-window cell×15min×band SNR grid, a Locationdex roster). A paper whose grain IS the aggregate reads this directly. Identical numbers because it's the same organized measurement.
Archive pointer — the slot's recorded pointer to the underlying rows + the exact filter (e.g. partition_path + partition_filter on a WSPR event window; a source_id/metric predicate on a PG-backed kind). A paper that needs the raw grain reads the raw via the pointer the slot records, so it gets bit-identical rows while still tracing through the dex.

This resolves the only real risk (a flip silently changing a shipped paper's numbers): a paper that can't reproduce from the working surface reads the archive pointer instead. No paper is forced onto an aggregate that loses its grain.

Phase 0 — the dex reader (blocks everything) — BUILT 2026-07-07

src/terrapulse/monitor/dex_reader.py — the consumption counterpart to event_storehouse. Verified against the real dexes (both storage patterns) and covered by 7 hermetic tests (tests/test_monitor/test_dex_reader.py, tmp-fixture dexes, no /mnt/ursa dependency). API:

locate(kind) — finds a kind across all storehouses, detects spine vs dossier storage.
read_slots(kind, **filters) → (frame, provenance) — the slot catalog (spine rows, or one flat row per dossier). For the event-per-slot kinds (eq/tor/neo/entry) this catalog IS the working surface. Equality + list-membership filters.
read_aggregate(kind, slot_id) → (frame, provenance) — a slot's finer working surface: the sidecar aggregate parquet it points to (aggregate_path, e.g. wspr_event_window) or an inline series (daily_band_series, e.g. wspr_spot_census).
read_archive(kind, slot_id) → (frame, provenance) — resolve the slot's file-partition pointer to the raw time+band slice (WSPR); the caller applies its own finer spatial predicate.
read_source_rows(kind, source=, select=, where=) → (frame, provenance) — the PG-source archive pointer (Option A, added after the tornado-alley pilot). Reads a kind's underlying observations rows THROUGH the dex's kind→source registry: the paper never hard-codes a source_id, the registry resolves it (resolve_source_ids maps the kind's registered slug → datasources.source_id). The SQL projection is the paper's own, so the pull is bit-identical while the datum now traces through the dex.
provenance_for(kind) — the kind's source block(s).

Pilot finding (tornado-alley-east) — dexes have drifted from their live sources

The first "clean" pilot proved the reader works but surfaced a campaign-level truth: a dex built weeks ago is not a mirror of the paper's live source. The tor dex (73,926 slots, built 2026-06-12, NCEI-enriched) vs the paper's spc_tornado_history source (73,458, 1950–2025): ~72,541 shared, but the dex holds ~467 extra 2026 tornadoes (NCEI, beyond the SPC archive), ~2,014 dossiers with ef_scale=None (F→EF coding gaps), and ~409 coordinate-rounding drifts. So reading the working surface would NOT reproduce the published 71,832 / EF2+ 13,400 / EF1+ 38,565. This is why Option A (archive-pointer / source-through-registry) is the right flip for every PG-backed paper — it preserves published numbers exactly. The working surface stays available; the paper just reads its source rows through the slot.

Original spec (satisfied):

Load a kind's spine-parquet slots (predicate pushdown on the spine: by id, type, time, papers) AND iterate file-per-slot dossiers — the two storage patterns.
Return a slot's working surface (spine row / dossier fields, or the sidecar aggregate parquet it points to) as a polars frame.
Resolve a slot's archive pointer to the underlying rows: for a PG-backed kind, run the recorded source_id/metric predicate against observations; for a file-backed kind (WSPR), read the recorded partition_path under the recorded partition_filter.
Attach the provenance block via resolve_for_kind, so every returned frame carries its source.

Deliberately small: it wraps read patterns that already exist ad hoc (the dossier_path + json.loads in tor_stormevents_enrich.py; direct pl.read_parquet on spines) into one consumer API. Ships with its own tests; self-checks nothing on disk (read-only).

The 19 papers, bucketed by flip difficulty

Bucket A — clean PG-source flips (Option A archive-pointer / source-through-registry)

Flip = the extract calls read_source_rows(kind, source=…, select=<paper's projection>) instead of hard-coding a source_id; the registry supplies it, numbers preserved bit-exact.

tornado-alley-east → tor (spc_tornado_history). DONE 2026-07-07 — repointed scripts/extract.py onto read_source_rows; output parquet byte-identical to the published one (73,458 raw → 71,832 clean, EF2+ 13,400, EF1+ 38,565, 13,174 convective days). Provenance resolves to spc_tornado_history through the dex.
fireball-surge-2026 → neo/entry. DEFERRED 2026-07-08 (Mike, option 3) — can't do a bit-exact Option-A flip; needs its own freeze-vs-re-run decision. The published paper (357 events, 1998–2026) pulled metric='neo_fireball' from source nasa_neo (1113d243). That source has since been gutted: rerunning the paper's own extract.py today yields 3 events, not 357 (the live neo_fireball metric now holds only a ~3-week 2026 window). The full bolide catalog migrated to the dedicated cneos_fireballs source (f4d141de, metric cneos_bolide) — which is exactly what the entry dex registers — and is now richer: 1,064 events back to 1988 (971 in the paper's 1998+ span), ~3× the published 357. So no live source reproduces 357; the choice is (1) freeze the committed 357-row snapshot as an archived dex artifact, or (2) re-source + re-run on the full cneos_fireballs catalog (a paper re-open, every number changes). Held for a separate decision; the committed snapshot is restored/intact.

Bucket B — seismic, grain-gated (need a backfill cleared first)

All six read usgs_earthquake. Grain is the whole story ([[project_checkpoint_20260707]] cascadia finding):

lunar-syzygy-seismicity → eq (usgs_earthquake). DONE 2026-07-08 — repointed scripts/extract.py onto read_source_rows("eq", source="usgs_earthquake", …). This flip needed a new reader capability: the pre-reg dedup runs in SQL (DISTINCT ON keep-largest-magnitude), so read_source_rows gained an order_by= param to carry the tiebreak. The published 7,477 deduped events reproduce bit-identical (0 removed; every shared row equal). LIVE-FEED FINDING: unlike static SPC tornado data, usgs_earthquake is a perpetual feed — a live re-pull now yields 7,479 (the same 7,477 plus 2 new real M6+ events on 2026-07-03, the 6 days of ingestion since the paper's extract). The published snapshot is preserved (committed parquet restored to 7,477, unchanged); picking up the 2 new events would be a deliberate paper refresh, a separate editorial act, not a plumbing side effect.
sanriku-m66-may2026 → eq (single M6.6 event; spine should hold it).
cascadia-swarm-apr2026 → eq_mag micro-tiers. Blocked by EQ-2 (9 of 23 swarm events never ingested by the live feed).
reno-earthquake-swarm → eq_mag (reuses cascadia workspace; same EQ-2 gate).
kermadec-doublet-apr2026 → eq + cross-network. Blocked by EQ-3 (EMSC/GFZ/ISC not associated).
earthquake-lights-104-years → eq + nuforc. Blocked by EQ-4 (paper spans 104 yr; eq starts 1973).

Bucket C — WSPR (6 papers, external→dex; the campaign's biggest single win)

Today these hit /mnt/ursa raw directly or the external db1.wspr.live ClickHouse. Flip moves them onto the two WSPR dexes and removes the live-external dependency at analysis time.

wspr-solar-cycle-bug (wspr-solar-cycle-modulation) → wspr_spot_census working surface + sunspot_number. DONE 2026-07-08 — the campaign's first WORKING-SURFACE flip. New scripts/extract.py reads both halves through a dex: WSPR month×band rolls up the census's day×band working surface (via read_aggregate; n_spots = Σ spot_count, month mean_snr = spot-count-weighted mean of daily means = the exact raw monthly mean), and the provisional tail month (2026-04, no day-aggregate yet) reads its raw-partition archive pointer. The SSN half reads sunspot_number/silso_sunspots through the registry (read_source_rows gained a group_by= param for the AVG(value) GROUP BY month pull). Reproduction: 213 of 220 months bit-identical; the published per-band correlations reproduce to |Δr| ≤ 0.003 (every band's sign + magnitude holds). DRIFT FINDING (new failure mode, worse than lunar's append-only growth): /mnt/ursa WSPR raw was backfilled in place in 7 months since publish (n_spots up to +1.16 M; mean_snr up to 0.20 dB), and SILSO revised some SSN values — so the working surface no longer bit-reproduces those 7 published months (the raw-at-publish-time is gone). The committed snapshot (caches + results.json) is preserved unchanged; adopting the revised catalog would move r by ≤0.003 and is a deliberate refresh, not a plumbing side effect. This is the first flip where the conclusion, not the exact numbers, is what carries.
BANDS band-code note: the census/daily/ band field and the paper's raw band share the same encoding (3=80m … 28=10m); the census's extra low bands (-1/0/1 LF-MF) fall outside the paper's BANDS filter, so the roll-up matches the paper's 8-band scope and 220-month coverage exactly.
wspr-tornado-precursor / v2 / v3 / v4 → wspr_event_window archive pointer (these need per-spot grain; the slot records the raw partition + carve filter, so they read bit-identical raw through the slot). The cell×15min×band aggregate is a bonus working surface, not the flip path.
aircraft + precursor are also blocked by WS-4 (their 2026 windows are in the un-backfilled raw tail; 4 slots pending). Gate on WS-3→WS-4.

Bucket D — solar/space-weather cluster (grain = continuous series)

bz-dst-lead-time → geomagnetic_dst + solar_wind. FLIP DONE 2026-07-08 — scripts/extract.py now reads Dst through geomagnetic_dst/dst_index and Bz+speed through solar_wind/dscovr_solar_wind via read_source_rows (group_by/order_by carry the hourly AVG/MIN aggregation). Gotcha banked: dscovr_solar_wind is a MULTI-metric source (bz, speed, density, temp…), so the metric= predicate is mandatory on top of the registry source_id — without it the AVG mixes all solar-wind metrics and the correlation collapses to r=0.04 (source_id ≠ metric, the fireball lesson). The donki_storm_kp chart overlay has no dex kind (nasa_donki) and never enters a test, so it stays a documented direct PG read. ADOPTION + PROSE PARKED for the writing fiesta (Mike 2026-07-08): re-running the flipped extract adopts a much larger window (dst_index backfilled to 1963; DSCOVR now 2025-10→2026-06) → N 1,359 → 4,343 (~2 mo → ~8.5 mo). Headline HOLDS and strengthens (Bz→Dst peak r=0.42 @ 5 h, Granger survives all lags, strict-null fails, Burton R²=0.098), BUT the secondary stratification claim ("harder driving → faster response") no longer cleanly holds (strong strata ~4 h vs moderate ~2 h) and the old "45-hour wrong turn" dissolves (now 143 h, powered). That's a real rewrite, not a number-swap, so per the campaign scope (flip now, rewrite in the fiesta) the published snapshot (data + article + paper.tex) is restored intact; the fiesta re-runs the flipped extract and rewrites the stratification section + abstract.
starlink-thermosphere-storm-pilot → sunspot_number + solar_xray_flux + solar_wind (possible SW-2 deep backfill for pre-2025 wind/X-ray).
solar-watch-april-2026 → geophysical Yeardex kinds (compound landing article; verify each layer).

Bucket E — the one paper that can't fully flip

skinwalker-ranch-seven-layers — a ~28-source kitchen-sink null baseline. Its measured layers flip onto their dexes (NOAA/USGS/CNEOS/Kp/etc.), but three sources are model output, bright-line OUT and undexable: SPC convective outlook, open_meteo, era5_cloud. At the flip they stay documented held-out context (their current status) or are dropped; never dexed. Structurally cannot be 100% dex-sourced. Do last, lowest value, flag don't force.

Recommended order

Phase 0 — build the reader (blocks all; ~the real first task).
Pilot: tornado-alley-east (Bucket A, cleanest, grain matches, exercises the reader end-to-end on a PG-backed spine kind).
fireball-surge-2026 (Bucket A, second, exercises dedup reconciliation).
lunar-syzygy-seismicity (Bucket B, no backfill; exercises the notable-spine read).
wspr-solar-cycle-bug (Bucket C, the working-surface read; first WSPR win).
Clear the gated backfills (EQ-2/3/4, WS-4, SW-2/3) as each blocked paper comes up — see docs/dex-backfill-backlog.md. Flip each gated paper once its backfill lands.
skinwalker last.

Blockers cross-referenced to `docs/dex-backfill-backlog.md`

EQ-2 (cascadia/reno) · EQ-3 (kermadec) · EQ-4 (earthquake-lights) · WS-4 (aircraft/precursor 2026 windows) · SW-2 (starlink pre-2025 wind/X-ray) · SW-3 (bz-dst Kyoto live edge). A paper in a gated bucket does not flip until its row clears.

One decision for Mike before Phase 0

Confirm the two-surface flip definition (a paper reads a dex slot's working surface when its grain is the aggregate, else the slot's archive pointer for bit-identical raw — always through the slot). This is what makes a shipped paper's numbers safe. If instead the intent is that every paper must read the working surface (accepting documented number deltas where the aggregate differs from raw), that is a different, larger effort and changes several buckets. The two-surface framing is recommended.