Scope — Geophysical-index Yeardex (paper-data standardization campaign)
Frozen 2026-07-07. Part of [[project_paper_dex_repoint_campaign]]. Shape ruled by Mike 2026-07-07: year-bucket Yeardex, one slot per channel × year, the fine series carried inside the slot. Not a new dex family: this is the existing Yeardex (dairy / landuse / world-bank) run at the resolution a continuous scalar needs. A Yeardex slot has always held a bundle of a period's numbers; here the bundle is a year's worth of a continuous geophysical index.
What this organizes
Four continuous measured geophysical index series that several solar papers draw from but
that no dex currently holds. Today we dex space weather only as discrete events (gst storms,
flare, cme) or as station rosters (neutron_monitor, magnetic_observatory). None of those
keeps the running curve. gst, the closest, peak-picks storm events out of the Dst series and
throws the curve away. This Yeardex keeps the curve.
All four are measured reality (instrument readings / observed counts), never model output:
| Kind | Source (slug) | What it measures | Native cadence | Ingested coverage |
|---|---|---|---|---|
sunspot_number |
silso_sunspots |
daily total sunspot number (global) | daily | 1818–2026 (~209 yr) |
geomagnetic_dst |
dst_index |
Dst geomagnetic disturbance (nT) | hourly | 1963–2026 (~64 yr) |
solar_wind |
dscovr_solar_wind |
solar-wind speed (km/s) + IMF Bz (nT) | minute | 2025-10–2026-06 (~9 mo) |
solar_xray_flux |
goes_xray |
GOES X-ray flux (W/m²) | minute | 2026-03–2026-07 (~3.6 mo) |
kp_index |
gfz_kp |
Kp planetary geomagnetic index (+ ap) | 3-hourly | 1932–2026 (~94 yr) |
kp_index added 2026-07-07 (clears SW-1). Kp was not previously ingested; fetched from the
GFZ Niemegk definitive file (reload_gfz_kp_spine.py, Mike-approved URL) into observations as
metrics kp_index + ap_index (each UT day expanded to eight 3-hourly readings). Measured index
(ground magnetometers), not a model.
Each channel is its own kind (per the organize-not-process rule: one phenomenon = one
category), sitting in the Yeardex family beside dairy/landuse. Sibling to the gst event dex,
not a replacement for it.
Slot model (year-bucket, fine series inside)
One slot = one channel × one calendar year, event_id = str(year), stored file-per-slot JSON in
data/year_storehouse/<kind>/<year>.json via the shared event_storehouse.write_dossier +
rebuild_index_from_disk machinery (same as dairy). Each slot carries:
annual_summary— the year's roll-up (n readings, min/max/mean, and channel-specific peaks: peak sunspot, minimum Dst, peak wind speed / most-negative Bz, peak X-ray flux). Cheap top-level read.daily_series— the inline fine series: one row per UTC day{date, n, min, max, mean}. ≤ 366 rows/slot. Forsunspot_number(native daily) this is full fidelity.series_parquet— for channels finer than daily (geomagnetic_dsthourly,solar_wind+solar_xray_fluxminute), the full native-resolution deduped series is written to a side filedata/year_storehouse/<kind>/<year>_series.parquet(theentry-dex dossier + side-parquet pattern; side parquets stay out of the storehouse JSON index).sunspot_numberhas none.- Standard Yeardex keys:
kind, event_id, subtype, year, origin_utc, start_utc, last_fix_utc, provisional, name, sources, swept=False, cross_match="n/a-yeardex", radius_km=None, sensors_swept=[], total_sensor_hits=0, sensor_summary={}, generated_utc.
provisional=True for the current (incomplete) year.
Dedup is load-bearing
Every raw feed is heavily over-duplicated by rolling-window re-polling (goes_xray: 86M raw rows
for 3.6 months; silso: 3.0M rows for a daily series; dscovr: 32M rows for 9 months). The build
dedups to one reading per timestamp (GROUP BY timestamp_utc, take the value) before any
daily aggregation. Failing to dedup would corrupt every mean.
Provenance
Four kinds registered in KIND_SOURCES (all four source slugs already in KNOWN_SOURCES):
sunspot_number→[silso_sunspots], geomagnetic_dst→[dst_index],
solar_wind→[dscovr_solar_wind], solar_xray_flux→[goes_xray].
Validation
Self-validates at build end via dex_validator.validate_kind_or_raise (file-per-slot family),
once per kind, requiring name/year/daily_series and a non-empty daily series per slot.
Deferred (backfill-backlog)
- SW-1 Kp is not ingested (the plan listed it, but
gstis built from Dst and skinwalker pulled Kp straight from GFZ at paper-time). A Kp channel is a fetch, not an organize; add anoaa_kp/ GFZ Kp feed first, then akp_indexsibling kind. - SW-2 Deep backfill of the two minute channels:
solar_wind(DSCOVR only reaches ~2016; ingested window is ~9 mo) andsolar_xray_flux(GOES record runs decades; ingested ~3.6 mo). - SW-3 Merge
kyoto_dst_realtimeas the live edge ofgeomagnetic_dst(used alongsidedst_indexingst); v1 usesdst_indexonly. - SW-4 Pre-1963 Dst / pre-1818 sunspot historical reconstructions if a paper needs deeper time.
Not in scope
No repoint of any paper here (the flip is the campaign's last step). No new fetching. No model products. Kp, being unfetched, is out of v1.