INVENTORY-202608 — every processing item, measured

Written 2026-08-17 for Brad, to size the headless process manager and its worker pool against real numbers instead of guesses.

Companion to docs/TERRA-STACK.md (the system as it runs). Where that doc describes the topology, this one measures the work.

Method and window

Three measurement sources, all direct:

What	Source	Window
Scheduler job runtimes	`journalctl -u terrapulse.service`, pairing APScheduler `Running job` → `executed successfully`	19.5 h, the current process only
Dropped executions	same journal, `was missed` / `maximum number of running instances`	same 19.5 h
Per-source fetch cost	`loader_runs.duration_ms` in Postgres	14 days, and 24 h where noted
Memory	`/proc/<pid>/status` VmRSS + systemd `MemoryCurrent`	instantaneous

Why 19.5 hours. terrapulse.service was restarted 2026-08-16 20:11:10 EDT and has not restarted since (NRestarts=0). A wider journal window spans that boundary and mis-pairs job start/finish lines across two processes, so every runtime figure below is scoped to the current process. Interval jobs in APScheduler first fire one full interval after start, so daily and weekly jobs legitimately show 0–1 runs in this window — that is not evidence they are broken.

A limitation stated up front: APScheduler runs every job inside the single API process, so per-job memory cannot be measured — only the process total. Where this document attributes memory to a specific job it says so and shows the reasoning. Everything else is a direct read.

0. Context — the machine and the stack this runs on

Self-contained summary so this document can be read alone. Full detail, including the complete filesystem inventory, is in docs/TERRA-STACK.md.

Host

Single box, impera, multi-tenant and Brad's — nginx serves ~33 sites from it, of which TerraPulse is one. AMD Ryzen 5 5600X, 6 cores / 12 threads. 62 GB RAM; 25 GB used, 34 GB buff/cache. Swap 31 GB total, 31 GB used — fully consumed.

Storage — only two devices are internal

Everything else, 18 filesystems, hangs off USB, and most are spinning disks. This constrains the worker design more than CPU does.

Mount	Device	Bus	Type	Size	Free	Seq read
`/`	`nvme1n1p3`	internal NVMe	SSD	459 G	27 G (95%)	1855 MB/s
`/var/lib/pgdata`	`nvme1n1p2`	internal NVMe	SSD	456 G	132 G (70%)	same device
`/mnt/storage`	`nvme0n1`	internal NVMe	SSD	469 G	4.8 G (99%)	1321 MB/s
`/mnt/working`	`sdd1`	USB 5 Gb/s	SSD¹	1.8 T	210 G (88%)	331 MB/s
`/mnt/ursa`	`sdl1`	USB 5 Gb/s	spinning	7.3 T	3.7 T (47%)	221 MB/s
`/mnt/backup-ursa`	`sdh1`	USB 5 Gb/s	spinning	4.6 T	1000 G (78%)	126 MB/s
`/mnt/grateful`	`sda`	USB 480 Mb/s	SSD	916 G	231 G (74%)	—
`/mnt/tm`	`sdm1`	USB 480 Mb/s	SSD	1.8 T	1.4 T (22%)	—

¹ reports rotational but is a Crucial X6 portable SSD; the USB bridge misreports it.

Twelve further USB filesystems (/mnt/models, /mnt/shared, /mnt/stability, /mnt/nom01, /mnt/nom02, /mnt/green, /mnt/marzano, /mnt/blue, /mnt/backup, /mnt/passport, …) are not TerraPulse's. ext4 throughout except /mnt/passport (hfsplus) and a small tank zfs pool. Inode pressure is nowhere near binding — worst is 7%.

Two SSDs are strangled on USB 2.0 (480 Mb/s ≈ 40 MB/s): /mnt/grateful and /mnt/tm. Neither is TerraPulse's, but moving them to a 5 Gb/s port is free performance.

Where TerraPulse's data actually sits

Data	Path	Device	Size
Postgres 16 (:5433)	`/var/lib/pgdata/16-main`	internal NVMe	301 G
DuckDB staging	`data/duckdb` → `/mnt/ursa/…`	USB spinning	17 G
Event storehouse	`data/event_storehouse` → `/mnt/ursa/…`	USB spinning	8.1 G, 1,288,493 files
WSPR raw	`/mnt/ursa/data/terrapulse/wspr`	USB spinning	201 G
Other storehouses	`data/*_storehouse` (real dirs)	root NVMe	~1.5 G
Caches + rosters	`data/*`	root NVMe	11 G

Postgres is NOT on root, and it is not what fills it — root's 410 G is 314 G of /home across all tenants (TerraPulse is 20 G of that). There is also a second cluster on this box: PG 15 on :5432, 154 G on /mnt/ursa. TerraPulse does not use it. Check which cluster before anyone tunes "the database".

The binding storage fact is latency, not capacity. Both USB volumes have terabytes free. But the hot derived data — the 1.29 M-file storehouse and the DuckDB staging tree — sits on the slowest medium in the box, and §2 shows exactly what that costs.

Runtime topology

              nginx :443 (letsencrypt, terrapulse.info)
                    │
   ┌────────────────┼──────────────┬──────────────┐
 location /     /api/, /docs     /admin      /ws, /api/v1/ws
   ▼                ▼               ▼              ▼
terrapulse_web   terrapulse_api  (same proc)   (same proc)
127.0.0.1:4321   127.0.0.1:8111   FastHTML     websocket
Astro 6 SSR      FastAPI/uvicorn  at /admin
node v24.0.1     python 3.13
                 SINGLE WORKER — no --workers
                    │
                    ├── APScheduler IN-PROCESS: 29 named jobs + 51 fetchers
                    └── SQLAlchemy async → PostgreSQL 16 + PostGIS :5433

Independent listeners (never idle, not scheduled):
  terrapulse-glm · terrapulse-blitzortung · terrapulse-pulse

systemd + nginx only. No Docker, no CI/CD — pushing to main deploys nothing. Only the backup units are versioned in infra/systemd/; the API, web and three listener units exist solely in /etc/systemd/system/ and carry non-obvious environment (pyenv/nvm PATHs, TMPDIR=/mnt/ursa/…, and the real port). Versioning them is a prerequisite for any migration.

Port trap: config.py and .env.example declare API_PORT=8000 and nothing reads it. The API actually runs on 8111, set in the systemd unit. Start it on 8000 and the site renders empty with no error.

Scale of the data

495 registered datasources (370 active), ~586.7 M observations (reltuples), 300 GB database of which observations is 251 GB, ~202 metrics. Dexes on disk: 41 event kinds, 14 location, 4 celestial, 4 platform, 1 directory.

COUNT(*) on observations takes ~30 minutes — already worked around in the stats cache via reltuples. Any new worker that counts rows needs the same care.

What the UI actually consumes

Despite 13 API route modules, the site uses nine endpoints plus a websocket plus a static passthrough for lab workspace assets: /health, /alerts, /alerts/count, /events/today, /events/today/count, /datacards/graph/data, /client-log, /lab/workspaces/{slug}, /lab/workspaces/{slug}/{paper|www|data|scripts}/{file}.

There is no HTTP surface for the dex layer at all. 40 /data/* pages describe dexes in prose; none reads one. docs/dex-web-display.md designs it and it was never built. A new UI stack without that API renders the same nine endpoints more prettily.

1. Headline: the scheduler is saturated and silently dropping work

In 19.5 hours the process dropped 396 scheduled executions:

197 logged was missed (the tick came due while the loop was busy)
199 logged skipped: maximum number of running instances (the previous run of that same job had not finished)

That is roughly 20 dropped executions per hour, continuously, and none of it surfaces anywhere a human looks — it is WARNING-level journal noise inside a service whose status stays active.

What is being dropped:

Job	missed	skipped	Scheduled/19.5 h	Actually completed
`run_fetch` (all 51 fetchers)	161	42	—	4,639
`_autosense_sweep`	0	26	39	12
`_run_tor_sweep`	0	22	39	16
`_run_entry_sweep`	0	22	39	16
`_run_vol_sweep`	0	21	39	17
`_run_storm_sweep`	0	21	39	17
`_run_gst_sweep`	0	21	39	17
`_run_eq_sweep`	0	21	39	17
`_run_fema_sweep`	2	3	19	14
`loader_liveness_check`	16	0	234	216
`_run_safecast_refresh`	1	0	1	0
`_run_satellite_decay_refresh`	1	0	—	0
`_run_gw_sweep` / `_run_neutrino_sweep` / `_run_uhecr_sweep`	3 each	0	39	35 each

Two of those rows matter more than the rest:

The six 30-minute dex sweeps complete fewer than half their scheduled ticks.
loader_liveness_check was missed 16 times. That is the job that emits loader_stale alerts. The staleness alarm is itself being starved by the condition it exists to detect.
_run_safecast_refresh did not run at all in its one scheduled slot.

The platform is not losing data — fetchers are idempotent and the next tick catches up. It is losing timeliness and predictability, invisibly.

2. Root cause: six jobs each rebuild the same 1.29-million-file index

This is the single finding that explains most of the numbers above.

data/event_storehouse is a symlink to /mnt/ursa/data/terrapulse/event_storehouse — 8.1 GB, 1,288,493 files, on a USB-attached spinning disk (221 MB/s sequential, far worse on small random reads). Its index is a single 264 MB storehouse_index.json, and by design it is rebuilt from disk on read, never updated incrementally — that is what makes it crash-safe.

Every one of the six 30-minute sweeps ends by calling event_storehouse.rebuild_index_from_disk(base_dir=STOREHOUSE_DIR) against the shared directory covering all 41 kinds:

eq_sweep.py:237     gst_sweep.py:236    vol_sweep.py:403
storm_sweep.py:354  entry_sweep.py:373  fema_sweep.py:201  flare_cme_sweep.py:180

So six times every thirty minutes, the process walks 1.29 M files off a spinning USB disk and parses a 264 MB JSON document — to publish, typically, a few dozen refreshed dossiers (tor_sweep: refreshed 46, eq_sweep: refreshed 11, storm_sweep: refreshed 3).

The measured consequence:

Sweep	Completed	Median	p95	Max	Task-hours / 19.5 h
`_run_entry_sweep`	16	3326 s (55 m)	5099 s	5099 s	15.25
`_run_tor_sweep`	16	3268 s	5119 s	5119 s	15.17
`_run_gst_sweep`	17	3204 s	5115 s	5115 s	15.15
`_run_storm_sweep`	17	3204 s	5123 s	5123 s	15.14
`_run_vol_sweep`	17	3219 s	5103 s	5103 s	15.13
`_run_eq_sweep`	17	3212 s	5107 s	5107 s	15.12
`_run_fema_sweep`	14	1638 s	5114 s	5114 s	8.31

Those six medians fall inside a 122-second band, and the p95s inside a 24-second band. Jobs doing genuinely different work — tornadoes, volcanoes, geomagnetic storms — do not independently converge on the same runtime. They are all waiting on the same resource. Each is scheduled every 1800 s and takes ~3200 s, so each systematically overruns its own interval by ~1.8×, which is exactly why max_instances=1 skips their next tick.

Inference, flagged as such: the sweeps dispatch via loop.run_in_executor(None, ...), which uses Python's default thread pool (16 threads on this 12-thread box). Six concurrent rebuilds each holding a parsed 264 MB JSON is plausibly several GB of the process's 13 GB RSS. I measured the RSS and the index size directly; the causal link between them is reasoning, not measurement, and is worth confirming with a heap sample before anyone sizes a worker pool on it.

Design implication for the process manager. This is not a "make it concurrent" problem — it is already concurrent, and that is the problem. The index rebuild is a single shared serialized resource. It wants to be one worker, run once per cycle after all sweeps have written, not six workers each redoing it. Getting that one thing right removes most of the load in this document.

3. Named scheduler jobs — the worker inventory

29 named jobs. Runtime measured over 19.5 h; "safe concurrent?" is read from the code, not assumed.

Tier A — heavy, must move to workers (7 jobs, ~99 task-hours / 19.5 h)

Job	Cadence	Median	p95	What it does	Safe concurrent?
`autosense_sweep`	1800 s	1537 s	47194 s (13.1 h)	25 stalest AutoSense sources per tick, sequentially	`max_instances=2`; see §5
`entry_sweep`	1800 s	3326 s	5099 s	multi-source entry dossiers + shared index rebuild	No — shared index
`tor_sweep`	1800 s	3268 s	5119 s	tornado dossiers + shared index rebuild	No — shared index
`gst_sweep`	1800 s	3204 s	5115 s	geomagnetic storm dossiers + shared index rebuild	No — shared index
`storm_sweep`	1800 s	3204 s	5123 s	TC dossiers + shared index rebuild	No — shared index
`vol_sweep`	1800 s	3219 s	5103 s	eruption dossiers + shared index rebuild	No — shared index
`eq_sweep`	1800 s	3212 s	5107 s	quake dossiers + shared index rebuild	No — shared index
`fema_sweep`	3600 s	1638 s	5114 s	disaster dossiers + shared index rebuild	No — shared index

All eight write to the same storehouse tree and rebuild the same index. They are one worker lane, not eight. The refactor: sweeps become index-free producers and a single follow-on job rebuilds the index once.

Correction, 2026-08-17. An earlier revision of this document said the rebuild_index=False parameter "already exists on every one of them". It did not. It existed only on the backfill_and_store() variants; the sweep_and_store() functions the scheduler actually calls hardcoded the rebuild. Implemented in 793dc37: the parameter was added to all eight sweep_and_store() signatures (defaulting to True, so scripts and backfills calling them directly are unchanged), the scheduler now passes False, and a new storehouse_index_rebuild job publishes once per cycle behind a dirty flag that starts True so the first cycle after any restart always rebuilds. Pinned by tests/test_monitor/test_sweep_index_rebuild_contract.py.

Tier B — moderate, good worker candidates (5 jobs)

Job	Cadence	Median	p95	Notes
`stats_cache_refresh`	3600 s	306 s	399 s	already uses `reltuples` to dodge a 30-min `COUNT(*)`; still 5 min
`timeline_refresh`	3600 s	234 s	586 s	writes a render cache the site reads
`graph_cache_refresh`	3600 s	128 s	239 s	knowledge-graph cache
`geoindex_refresh`	21600 s	371 s	446 s	3 runs in window
`safecast_refresh`	daily 03:45	585 s (prior run)	—	missed its slot in this window

These are pure cache producers — read Postgres, write a file the site renders from. Ideal first workers: no shared mutable state beyond their own output file, and a failure degrades a page rather than corrupting data.

Tier C — cheap, leave in-process or make them trivial workers (16 jobs)

Every one of these has a median under 100 seconds, and most under 10:

eq_mag_refresh (9 s), geophysical_index_refresh (7 s), flare_cme_sweep (4 s), gfz_kp_reload (97 s), gw_sweep (2 s), uhecr_sweep (1 s), neutrino_sweep (1 s), forbush_detector (1 s), neo_refresh (38 s), satcat_refresh (11 s), satellite_decay_refresh (weekly, 0 runs in window), cosmic_messenger (daily, 0 runs), fire_sweep (daily, 0 runs — 1350 s in the prior process), igra_weekly_catchup (weekly, 4693 s in the prior process), loader_prune_job (<1 s), loader_liveness_check (18 s).

Two caveats inside Tier C:

loader_liveness_check must not be a normal worker. It is the alarm. It needs to run in a lane that cannot be starved by the work it is watching — today it is starved (16 misses), which is the worst possible failure mode for a monitor.
fire_sweep and igra_weekly_catchup are Tier-A-sized when they do run (1350 s and 4693 s in the previous process). They are only in Tier C because their cadence kept them out of this window.

4. Fetchers — 51 bespoke + ~440 AutoSense

Aggregate over the last 24 hours:

6,029 fetch runs
206,818 seconds of fetch time = 57.4 fetch-hours in a 24-hour day

That is 2.4× oversubscription on fetching alone, before any sweep work. It is only possible because fetches are async and overlap; it also means the loop is never idle. Median run_fetch is 8 s and p95 is 61 s, so the cost is dominated by a small number of very slow sources, not by the 60-second tick jobs.

The slowest sources (14-day window, by max duration)

Source	Runs	Avg	Max	Rows/run	Path
`castnet-outgoing-data`	26	59,565 s (16.5 h)	86,877 s (24.1 h)	9,497	AutoSense
`igra_soundings`	15	1,343 s	4,694 s	40,470	JOBS
`silso_sunspots`	13	1,993 s	2,794 s	76,183	JOBS
`metis-mix-2030`	26	1,726 s	2,606 s	40,191	AutoSense
`nasa_firms`	13	943 s	2,247 s	34,921	JOBS
`celestrak`	51	597 s	1,609 s	12,982	JOBS
`edgar`	30	991 s	1,321 s	4,613	AutoSense
`goes_xray`	3,493	143 s	926 s	7,189	JOBS
`fink_transients`	311	529 s	568 s	0	AutoSense (see §5)

Zero errors across every source in the 14-day window. Whatever else is wrong, the fetch layer is not failing — it is just slow and oversubscribed.

5. Two AutoSense pathologies worth fixing before the re-architecture

5a. One source can occupy an AutoSense slot for 24 hours

_autosense_sweep takes the 25 least-recently-fetched sources and awaits each sequentially. castnet-outgoing-data — a directory-listing crawl of gaftp.epa.gov/castnet/CASTNET_Outgoing/data/ — averages 16.5 hours and peaked at 24.1 hours per run.

With max_instances=2, one long source pins half the AutoSense capacity for a day. Measured effect: 12 sweeps completed out of 39 scheduled, 26 skipped, and _autosense_sweep accumulated 31.09 task-hours inside a 19.5-hour window — i.e. both slots occupied roughly 80% of the time.

The rotation still eventually covers everything (0 sources stale >7 days, 0 never fetched), but 293 of 370 active sources were fetched in the last 24 h, so 77 did not get a turn. That is the round-robin being dragged, not broken.

For the worker design: per-source work must be independently schedulable with its own timeout and its own failure domain. A 25-source sequential batch inside one task is precisely the shape that cannot be scaled or bounded.

5b. `fink_transients` is a zombie — parked but still fetching

fink_transients was parked on 2026-08-16 (5674c94) by commenting out its JOBS entry, because api.fink-portal.org has been unreachable since 2026-05-09.

It is still being fetched. 311 runs in 14 days, ~529 s each (a timeout), 0 rows inserted. That is roughly 45 hours of wall-clock spent timing out in two weeks.

The mechanism is a genuine trap in the current design:

KNOWN_SLUGS = {job_id for _, _, job_id in JOBS}
# _autosense_sweep selects:  WHERE active = true AND slug NOT IN (KNOWN_SLUGS)

Commenting out a JOBS entry does not stop a source — it promotes it to AutoSense, because the slug leaves KNOWN_SLUGS. The row is still active = true, so the generic fetcher picks it up. emsc, deactivated the correct way with active = false, has 0 runs and is genuinely stopped.

Fix is one row: UPDATE datasources SET active = false WHERE slug = 'fink_transients';. Not applied — it is a production state change and belongs to Mike's source-management lane. Flagged here and raised in chat.

For the re-architecture: "stop this source" should be one CLI verb with one meaning, not two mechanisms that disagree. This is a good first candidate for the management CLI Brad wants.

6. Offline processing — 78 scripts that are already worker-shaped

scripts/ holds 108 files, of which 78 are backfills, dex builders or reloads. These are the batch jobs that currently run by hand in a terminal, and they are the most natural residents of a worker pool — long, restartable, idempotent, and today entirely unmanaged.

24 backfill_* — historical loads (backfill_igra_history.py, backfill_firms_history.py, backfill_eq_dossiers.py, backfill_geo.py, backfill_tag_ids.py, …). Hours to days. The IGRA period-of-record backfill runs country-by-country and one country took 3.6 M rows.
~40 build_* — dex builders, one per kind (build_drought_category_yearlocationdex.py, build_aircraft_movingplatformdex.py, build_glm_event_windows.py, …).
~14 reload_* / rebuild_* / load_* — spine reloads that double as the live edge for sources with no BaseFetcher.

None of these has a queue, a status surface, a retry, or a record that it ran. Every one is a candidate for "submit to the process manager, watch it from the CLI".

Known blocker for whoever wires these up: three tests/test_monitor/ modules plus several scripts import pyarrow and requests. Those are now declared in pyproject.toml (1e216f2), but the scripts also assume a live Postgres on 5433 and a populated data/ tree — they are not hermetic and will need explicit environment contracts to run under a manager.

6b. The backup pipeline — the first thing that should become a CLI verb

Brad's target has backups and backend management running through a CLI. This is what exists today, and it is already the clearest worked example of why.

terrapulse-backup.timer fires nightly at 03:30 local (Persistent=true, 5-min jitter) → scripts/backup_postgres.sh, 380 lines of bash, four stages:

Stage	Target	Measured 2026-08-17
1. `pg_dump -Fc`	`/mnt/backup-ursa/backups/postgres/`	17 GB in 2,056 s (34 min)
2. Claude memory tar	`…/claude-memory/`	417 KB
3. Storehouse rsync snapshot (hardlinked deltas)	`…/storm-storehouse/`	1,288,545 files, 4.1 GB copied, ~92 min
4. Off-box tarball + GitHub release asset	tag `eventdex-backup`	weekly (see below)

Today's run "failed" while every stage succeeded. It hit TimeoutStartSec and systemd SIGTERM'd it:

03:33:41  start
04:07:57  dump succeeded: 17GB in 2056s
04:07:59  memory snapshot: 417KB
05:39:22  storehouse snapshot: 1,288,545 files, 4.1GB copied
07:33:41  systemd: start operation timed out. Terminating. (SIGTERM)

The kill landed during stage 4 — taring 1.29 M small files off the USB spinning disk takes over two hours — and left a 136 MB .tar.gz that gzip -t rejects. systemctl status reports failed, which is true of the unit and false of the backup.

The 2026-08-16 storehouse fix is confirmed working. Storehouse snapshots had been silently self-deleting every night since 2026-08-02 (a snapshot inherited a backdated mtime from rsync -a and the same run's retention prune deleted it). That fix was fixture-proven only; this run is its first live proof — storehouse-2026-08-17-0333 survived on disk at 8.0 GB. Postgres dumps are unbroken back through 2026-08-10 at 16–17 GB each.

Fixed and shipped in 4c16f6f after that run: the off-box tarball now rolls weekly (driven by the age of the newest good tarball, so a missed week retries the next night rather than waiting seven more), and both tarballs are written .partial → gzip -t → mv, so a kill can never publish an archive that looks valid until someone restores it. TimeoutStartSec raised 4 h → 6 h as headroom. Verified by tests/test_backup_storehouse_rotation.sh: 18 checks, all pass.

Why this belongs in the CLI lane. Every failure mode above is a reporting failure, not a backup failure: work that succeeded reported as failed; a corrupt artifact that looked valid; three months of silently missing snapshots that only surfaced when someone went looking. A backup that runs correctly and cannot tell you so is one incident away from a backup that runs incorrectly and also cannot tell you so. The CLI needs backup run, backup verify and backup status as first-class verbs with real exit codes — and the unit reduced to a thin caller.

Restore verification exists (scripts/verify_backup_restore.sh, monthly) and is the natural backup verify.

7. Memory

Process	RSS	systemd `MemoryCurrent`	Threads
`terrapulse.service` (API + scheduler)	13,054 MB	17 G	91
`terrapulse-glm`	189 MB	196 M	29
`terrapulse-pulse`	126 MB	124 M	26
`terrapulse-blitzortung`	91 MB	93 M	13
`terrapulse-web`	61 MB	48 M	11

The API process alone accounts for ~17 GB. Host RAM is 62 GB with swap at 31/31 GB — fully consumed. The four other TerraPulse services together use under 470 MB; the entire memory story of this platform is one process.

Two consequences for the worker pool:

There is no headroom to add workers on top of the current process. Work has to move out, not be added alongside. Splitting the scheduler out is memory-neutral at worst and probably a large win, because the 264 MB index parses would leave the API's address space.
Size the pool by memory, not by CPU. The box has 12 threads but is already in swap. A pool of N workers each capable of holding a parsed 264 MB index is a swap-thrash generator, and it would take Brad's other ~32 nginx tenants down with it.

8. Recommended worker taxonomy

Four lanes, derived from what the measurements actually show:

Lane 1 — the storehouse lane (serialized, 1 worker). All eight dex sweeps plus the index rebuild. They contend on one directory tree on one spinning disk; running them in parallel is what created this problem. Sweeps run with rebuild_index=False, then one index rebuild follows — shipped in 793dc37, ahead of the manager, since it needed no new infrastructure. Expected saving: 7 of 8 rebuilds per cycle.

Lane 2 — the fetch lane (wide, per-source, bounded). One task per source, not a 25-source batch. Per-source timeout — a hard ceiling well under castnet's 24 hours — plus per-source concurrency limits. This is where AutoSense's ~440 sources and the 51 bespoke fetchers both belong.

Lane 3 — the cache lane (parallel-safe, low risk). stats_cache, timeline, graph_cache, geoindex. Each writes one file, no shared mutable state, and a failure degrades a page instead of corrupting data. Start here — it is the cheapest lane to build and validates the manager end-to-end without risking the dex layer.

Lane 4 — the batch lane (submitted, not scheduled). The 78 scripts/ jobs plus the backup pipeline (§6b). Submitted from the CLI, long-running, restartable, status-visible. This is the lane that does not exist at all today, and the one Brad's CLI is really about: backup run / backup verify / backup status, backfill <source> --from <date>, dex build <kind>, each with a real exit code and a record that it ran.

Outside all four lanes: the watchdog. loader_liveness_check must run where saturation cannot starve it. It is missing 16 ticks per 19.5 h today, which means the platform's staleness alarm currently fails in the exact circumstance it was built to report.

Sequencing

~~Fix the eight-fold index rebuild.~~ DONE — 793dc37. All eight sweep_and_store() functions take rebuild_index (default True); the scheduler passes False and a new storehouse_index_rebuild job publishes once per cycle. No new infrastructure. Pinned by tests/test_monitor/test_sweep_index_rebuild_contract.py.
~~Stop fink_transients~~ DONE 2026-08-17 — active = false applied; the source now has the same disposition as emsc. Recovers ~45 h per fortnight of pure timeout.
Cap AutoSense per-source runtime, so no single source can hold a slot for a day. Not done — castnet-outgoing-data can still run 24 h.
Then split the scheduler into its own unit, and build the manager around the four lanes.

Re-measure before using this document for capacity planning. Steps 1 and 2 have already materially changed what it measures: the figures above describe the system as it ran on 2026-08-17 before those two changes, and are kept as the baseline they were taken as. Re-run the same journal pairing after a full day on the new code to get the post-fix numbers.

9. Provenance

Claim class	Command
Job runtimes, missed/skipped counts	`journalctl -u terrapulse.service --since "2026-08-16 20:11:10"`, paired in Python
Service restart boundary	`systemctl show terrapulse.service -p ActiveEnterTimestamp -p NRestarts`
Per-source fetch cost	`loader_runs` aggregate over 14 d and 24 h
Source states	`datasources` (`active`, `last_fetched_at`, `url`)
Index rebuild call sites	`grep -rn rebuild_index_from_disk src/terrapulse/monitor/`
Index size, file count	`ls -la storehouse_index.json`, `find … -type f \| wc -l`
Memory	`/proc/<pid>/status` VmRSS, `systemctl show -p MemoryCurrent`
Script inventory	`ls scripts/`
Host / drives	`lsblk -o …,ROTA,MODEL,TRAN`, `findmnt`, `lsusb -t`, `df -hT`, `df -i`, `free -g`
Drive throughput	`hdparm -t --direct` (read-only, ~3 s per device)
Postgres location + size	`pg_lsclusters`, `du -sh /var/lib/pgdata`, `pg_database_size`
Backup timeline	`journalctl -u terrapulse-backup.service`, `ls` of `/mnt/backup-ursa/backups/*`
Backup fix verification	`bash tests/test_backup_storehouse_rotation.sh` (18 checks)
UI API contract	`grep` for `/api/v1` paths across `web/src`

Everything above is a direct read except the attribution of RSS to concurrent index parses in §2, which is explicitly labelled as inference.