Listening for events…
← All Docs

Dex Coverage Map

The finish line for "dex everything we already hold." Mike's rule (2026-06-16): organize every phenomenon already ingested into the platform into a clean Eventdex/Yeardex kind before adding new data sources. New sources are easy to add and pile up forever; the value is getting the data we already hold into per-phenomenon slots papers can draw from.

Built to the canonical model: docs/dex-data-model.md (one slot = one event, one year, or one place; per-phenomenon parallel lists; measured reality only). See also docs/event-spine-framework.md, docs/yeardex-framework.md, docs/locationdex-framework.md, and memory feedback_organize_not_process, feedback_measured_reality_only.

Three families now: Eventdex (slot = event), Yeardex (slot = year), Locationdex (slot = a fixed place / station; new 2026-06-19). Locationdex gives the orphaned fixed-station networks (neutron monitors, tide gauges, magnetic observatories, streamgauges) a home — they were filed as "not dex material" only because the place axis was missing.

This is a burn-down checklist, not a vibe. Update it as kinds land.

Snapshot (2026-06-16)

  • 11 Eventdex kinds dexed: entry 25,536 · eq 13,842 · fema 5,191 · gst 1,147 · gw 431 · neutrino 348 · tc 13,545 · tor 73,634 · uhecr 109 · vol 11,089. (Plus the cosmic_coincidences cross-match read-across, which is downstream, not a kind.)
  • 2 Yeardex kinds dexed: landuse 16 year-slots · dairy 28 year-slots.
  • 2 Locationdex kinds dexed (new family 2026-06-19): neutron_monitor 64 station-slots (NMDB, 22 countries; 4 streaming flux, 60 registry-only) · tide_gauge 301 station-slots (NOAA CO-OPS, 34 states; 10 streaming water-level, 291 registry-only).
  • Platform holds 367 active datasources with observations. Most are the AutoSense research tail (dataset-*, 5xxx-*, hourly-precip-*, datasets-*-cdls), which is mostly statistical/research — year-shaped at best, much of it not discrete events. The genuine remaining dex work is bounded: ~8 event kinds + a Yeardex batch, below.

Done

  • eq — earthquakes (usgs_earthquake; emsc/isc/gfz_geofon are additional catalogs that could later merge into the same slots, cited).
  • tor — tornadoes (spc_tornado_history, enriched with stormevents-csvfiles NCEI narratives).
  • vol — volcanic eruptions (smithsonian_gvp; usgs_volcanoes monitoring adjacent).
  • entry — atmospheric entries / fireballs (gmn_meteors, cneos_fireballs, meteorite_falls, sat:).
  • gw — gravitational waves (gwosc_events).
  • neutrino — high-energy neutrinos (icecube_neutrinos, IceCat-1).
  • uhecr — ultra-high-energy cosmic rays (auger_uhecr).
  • gst — geomagnetic storms (Dst-derived).
  • tc — tropical cyclones (ibtracs_hurricanes, nhc_storms).
  • fema — US disaster declarations (fema_disasters).
  • landuse (Yeardex) — US major land uses (usda_landuse, 1945–2017).
  • dairy (Yeardex) — US dairy supply (usda_dairy, 1998–2025).
  • forest_cover (Yeardex) — global forest area % of land (worldbank_forest, 221 countries, 1990–2023). COMPLETE 2026-06-19; deep World Bank API pull, not the 15-country live slice.
  • energy_use (Yeardex) — global energy use per capita (worldbank_energy, kg oil eq, 1990–2024). COMPLETE 2026-06-19; population (worldbank_population) joined as per-country context, not its own kind (Mike's ruling).
  • flood — DECLARED dangerous floods (nws_alerts flood Warnings + fema_disasters Flood declarations). COMPLETE 2026-06-19. Mike's ruling: the first v1 (deriving "high water" from USGS discharge by day-of-year P98) was scope creep — a creek over its 98th percentile is usually normal seasonal flow, and the percentile machine manufactured 540,000× "floods" on dry washes. v2 catalogs floods the declaring agencies judged a threat to life/property: NWS Flood + Flash Flood Warnings (~10,700, 2021+, located; advisories/watches excluded) + FEMA Flood DR/EM declarations (~919, 1953+, area-based). Each slot cites its agency; pure organize of already-ingested records, no spine pull. Sweep + NWS↔FEMA correspondence deferred-v2.

Done — Locationdex (slot = a fixed place / station)

  • neutron_monitor (Locationdex) — NMDB cosmic-ray neutron monitors (nmdb_stations registry spine + nmdb_cosmic_rays flux series, cited). COMPLETE 2026-06-19, first Locationdex kind. Full 64-station deep-pull roster (vs the 6 the live fetcher hard-codes), 22 countries, 100% with coords; each slot carries place + altitude + cutoff rigidity + measured-flux summary. Mike's ruling: pilot the family on a clean fixed-station network, not safecast (which is ~half roving drive-surveys, so a sensor is not a place). docs/scope-neutron-monitor-locationdex.md, scripts/reload_nmdb_stations.py, src/terrapulse/monitor/neutron_monitor_locationdex.py. Spatial sweep deferred-v2.
  • tide_gauge (Locationdex) — NOAA CO-OPS water-level stations (noaa_tide_stations registry spine + noaa_tides water-level series, cited). COMPLETE 2026-06-19, second Locationdex kind. Full 301-station deep-pull roster (vs the 10 the live fetcher hard-codes), 34 states/territories, 100% with coords; each slot carries place + station character (tidal vs Great-Lakes) + measured water-level summary. Measured reality: only the water_level product; CO-OPS predictions (harmonic-model output) held out. docs/scope-tide-gauge-locationdex.md, scripts/reload_noaa_tide_stations.py, src/terrapulse/monitor/tide_gauge_locationdex.py. Spatial sweep deferred-v2.

Remaining — place-shaped, measured, ingested, NOT yet dexed (Locationdex batch)

Fixed-station networks already ingested, each a clean Locationdex candidate (slot = one station). Each gets its own scope freeze; full-roster deep pull, not the thin live slice.

  • magnetic_observatory — INTERMAGNET geomagnetic observatories (intermagnet, ~150 global).
  • streamgauge — USGS NWIS streamgauges (usgs_water, ~13,000+ US, some to 1900).
  • safecast (later, binned-mobile special case) — radiation. ~half fixed posts, ~half roving drive-surveys; needs a geographic-cell treatment, not a station roster. Deferred per Mike.

Remaining — event-shaped, measured, ingested, NOT yet dexed

Ordered roughly largest/most-distinct phenomenon first. Each gets its own scope freeze (Brick A) before building; Mike rules the load-bearing categorization calls.

  • lightning — SPINE DONE 2026-06-17 (54f4879), 13th kind, 133,165 cluster slots. blitzortung_lightning (24.8M) + goes19_glm_flashes (17.7M) + goes18_glm_flashes (10.0M) = 52.5M detections. Slot = lightning cluster (Mike's ruling, cell→multicell→MCS scale-agnostic), flash-counted (consensus: ground 10 km/1 s, GLM 16.5 km/330 ms upstream), ≥30-flash floor, natural single-link (Option A, chaining accepted + disclosed). Rogue lone strikes preserved (sidecar). Scope docs/scope-lightning-eventdex.md. Remaining: Brick C live sweep + peak_utc fill.
  • severe_wx → SPLIT into 15 phenomenon kinds, SPINE DONE 2026-06-17. Mike's ruling: not one phenomenon, split it (each its own Eventdex) and leave existing dexes untouched. NCEI Storm Events details (1950–2026, 2.02M rows) minus the excluded types (tornado→tor, tropical/hurricane→tc, volcanic ash→vol, aurora dropped) → 15 kinds, 1,932,478 slots: wind 733,957 · hail 420,025 · winter 286,127 · flood 185,532 · drought 82,518 · heat 56,077 · cold 53,663 · heavy_rain 34,038 · fog 18,300 · lightning_report 18,173 · funnel 16,179 · coastal 15,167 · wildfire 9,532 · dust 2,321 · avalanche 869. Slot = NCEI EVENT_ID; zone events kept with null coords. Per-kind spine parquet (<kind>/<kind>_spine.parquet) not file-per-slot, to avoid bloating the shared index past ~770 MB. Scope docs/scope-severe-wx-eventdex.md, scripts/build_severe_wx_spine.py. Remaining: Brick D (per-slot dossiers once the index is made per-kind) + Brick C live sweep (spc_reports).
  • flare + cme — SPLIT into two kinds, COMPLETE 2026-06-18. Mike's ruling: a solar flare (EM X-ray burst, classed A–X) and a CME (plasma cloud, km/s + width + direction) are distinct phenomena → two kinds, not one. Deep DONKI catalog backfill 2010→present (the staged nasa_donki slice was a thin, ~88–110×-duplicated 3-day rolling tail): flare 3,268 slots (A:1 B:45 C:551 M:2,517 X:154; headline X9.3 2017-09-06) · cme 9,768 slots (597 Earth-directed / 9,171 not). Both catalog-only (heliographic origin, no terrestrial sweep), file-per-slot dossiers. Slot = DONKI activityID. CME WSA-ENLIL Earth-arrival forecast held out of the measured value under flagged model_context. DONKI linked flare↔CME pairs = deferred-v2 cross-match read-across. Scope docs/scope-flare-cme-eventdex.md, scripts/reload_donki_spine.py, src/terrapulse/monitor/flare_cme_sweep.py.
  • neo — near-Earth-object close approaches (nasa_neo 280k, jpl_sbdb).
  • transient — astronomical transients / alerts (fink_transients).
  • launch — orbital rocket launches (launch_library).
  • nuforc — UFO sighting reports (nuforc 80k). Already used in the earthquake-lights paper; no kind of its own yet.
  • alert — NWS alerts (nws_alerts 819k). Its own kind (currently only borrowed to enrich tor).

Remaining — year-shaped, NOT yet dexed (Yeardex batch)

  • ghg_inventory — US GHG inventory (ghg-inv, EPA GHG dataset-* family, 1990–latest).
  • power_emissions — electric-power emissions by state (egrid-*, dataset-annual-u-s-electric-power-industry-...).
  • edgar / carbon — global emissions grids (edgar, carbon-emissions) — verify measured-vs-modeled before dexing.
  • energy_stats — SEDS / RECS energy consumption (dataset-state-energy-data-system-seds, dataset-residential-energy-consumption-survey-recs-*).
  • usda_ag — USDA agricultural 5xxx tables beyond dairy/landuse (subject-scoped per Mike; several are already year-shaped statistical series).

Future-source adds (logged, not yet ingested)

New sources to add after the existing platform is fully dexed (Mike's ordering). Logged so the need isn't forgotten:

  • Rogue-lightning certification — GLM L2 flash-level energy/area product, or a superbolt network (WWLLN/ENTLN). The current lightning kind flags spatial rogue candidates only; certifying superbolts (by energy) or megaflashes (by extent) needs per-flash energy/extent these feeds lack. See docs/scope-lightning-eventdex.md → Rogue preservation.

Held OUT — measured-reality bright line (NOT dexed, stay live as platform sources)

Per feedback_measured_reality_only: model/forecast output is OUT of the dex even when the source is live. Do not dex:

  • cems-glofas, cems-glofas-ra-streamflow-analysis, copernicus EFAS (flood forecast).
  • open_meteo, open_meteo_marine, open_meteo_aqi, open_meteo_flood (weather/flood forecast).
  • era5_cloud and any reanalysis/objective-analysis grid.

These remain ingested and queryable as platform sources; they are simply not eligible for an Eventdex/Yeardex kind.

Not dex material (for reference)

  • Continuous monitoring time series feed papers directly as series and are not event lists. NOTE (2026-06-19): the ones that come from a fixed-station network (nmdb_cosmic_rays, noaa_tides, intermagnet, usgs_water, igra_soundings) are no longer "not dex material" — they are Locationdex material (slot = the station), see the Locationdex batch above; the raw series stays put and the slot summarizes + cites it. Genuinely placeless series (dscovr_solar_wind, silso_sunspots) and roving radiation (safecast, until its binned-mobile treatment) stay series. (Some, e.g. goes_xray, double as a flare-event source — dex the events, leave the raw series.)
  • The bulk AutoSense research tail (dataset-* EPA/DOE/USGS research datasets, datasets-*-cdls cropland rasters, hourly-precip-3240-*) is the curated research backlog (project_curated_sources): sort year-shaped ones into the Yeardex batch as they become relevant, leave the rest as the backlog they are. Nothing uploaded is wasted; not all of it is a dex kind.

Rule

When unsure how a phenomenon should be categorized (its own kind? merge into an existing slot? event vs year?), ask Mike — he holds the mental model. Build every kind to docs/dex-data-model.md.

Live Feed