TERRA-STACK — the running system, as measured
Written 2026-08-17 for Brad, ahead of a UI-stack swap and a scale-out pass. Every number and path below was read off the live box on 2026-08-17, not copied from an older doc. Where something is inferred rather than measured it says so.
Companion docs:
CLAUDE.md(working agreements + prime directive),docs/dex-data-model.md(the dex layer),docs/ARCHITECTURE.md(stale — layer 1 only, predates the dex layer).
0. The short answer to "are we processing feeds nightly?"
No. Nothing about the feed pipeline is nightly, and it hasn't been for a long time. Ingestion is continuous. The scheduler runs 51 bespoke fetchers on per-source intervals ranging from 60 seconds to weekly, plus a rolling AutoSense sweep every 30 minutes covering the other ~440 sources, plus two persistent socket listeners that never sleep.
What is nightly is a short list of maintenance and heavy-rebuild jobs that happen to sit in the 03:00–04:00 window:
| Job | When | What |
|---|---|---|
loader_prune |
03:15 UTC | drop loader_runs rows older than 30 days |
gfz_kp_reload |
03:20 UTC | re-pull GFZ definitive Kp (nowcast → definitive revisions) |
safecast_refresh |
03:45 UTC | full device reclassify + dex rebuild (~9 min) |
terrapulse-backup |
03:30 local | Postgres dump + memory + storehouse snapshot |
terrapulse-alert-log |
20:06 local | yesterday's NWS alert-log PDF |
The distinction matters for scale-out planning: there is no nightly batch window to widen. Load is a continuous floor with a spike at ~03:30 local.
1. Host
Single box, impera. This box is Brad's and is multi-tenant — nginx serves
~33 sites from it, of which TerraPulse is one. Nothing below should be read as a
claim on the whole machine.
| Resource | State (2026-08-17) |
|---|---|
| CPU | AMD Ryzen 5 5600X, 6 cores / 12 threads |
| RAM | 62 GB total; 25 GB used, 34 GB buff/cache, 35 GB available |
| Swap | 31 GB total, 31 GB used — fully consumed |
Storage — the whole picture
Only two devices are internal NVMe. Everything else — 18 filesystems — hangs off USB, and most of those are spinning disks. This shapes the re-architecture more than any other single fact about the box.
| Mount | Device | Bus / link | Type | Size | Free | Seq read |
|---|---|---|---|---|---|---|
/ |
nvme1n1p3 |
internal NVMe | SSD | 459 G | 27 G (95%) | 1855 MB/s |
/var/lib/pgdata |
nvme1n1p2 |
internal NVMe | SSD | 456 G | 132 G (70%) | same device |
/mnt/storage |
nvme0n1 |
internal NVMe | SSD | 469 G | 4.8 G (99%) | 1321 MB/s |
/mnt/working |
sdd1 |
USB 5 Gb/s | SSD¹ | 1.8 T | 210 G (88%) | 331 MB/s |
/mnt/ursa |
sdl1 |
USB 5 Gb/s | spinning | 7.3 T | 3.7 T (47%) | 221 MB/s |
/mnt/backup-ursa |
sdh1 |
USB 5 Gb/s | spinning | 4.6 T | 1000 G (78%) | 126 MB/s |
/mnt/models |
sdb1 |
USB 5 Gb/s | SSD | 916 G | 663 G (24%) | — |
/mnt/shared |
sde |
USB 5 Gb/s | SSD | 916 G | 802 G (8%) | — |
/mnt/stability |
sdf |
USB 5 Gb/s | SSD | 916 G | 401 G (54%) | — |
/mnt/nom01 |
sdk |
USB 5 Gb/s | spinning | 3.6 T | 2.2 T (37%) | — |
/mnt/nom02 |
sdp |
USB 5 Gb/s | spinning | 3.6 T | 2.8 T (20%) | — |
/mnt/green |
sdj |
USB 5 Gb/s | spinning | 1.8 T | 917 G (48%) | — |
/mnt/marzano |
sdr |
USB 5 Gb/s | spinning | 1.8 T | 1.2 T (30%) | — |
/mnt/blue |
sdo1 |
USB 5 Gb/s | spinning | 1.8 T | 20 G (99%) | — |
/mnt/backup |
sdn1 |
USB 5 Gb/s | spinning | 896 G | 508 G (41%) | — |
/mnt/grateful |
sda |
USB 480 Mb/s | SSD | 916 G | 231 G (74%) | — |
/mnt/tm |
sdm1 |
USB 480 Mb/s | SSD | 1.8 T | 1.4 T (22%) | — |
/mnt/passport |
sds2 |
USB | spinning (hfsplus) | 1.9 T | 926 G (51%) | — |
¹ reports rotational but is a CT2000X6SSD9 — a Crucial X6 portable SSD; the USB bridge misreports the flag.
Two drives are on USB 2.0 (480 Mb/s ≈ 40 MB/s real). /mnt/grateful and
/mnt/tm are SSDs strangled by a USB 2.0 link — 40× slower than the NVMe they
could be. If either holds anything latency-sensitive, moving it to a 5 Gb/s port
is free performance. Neither is TerraPulse's.
Filesystems are ext4 everywhere except /mnt/passport (hfsplus) and a small
tank zfs pool. Inode pressure is nowhere close to binding — the worst is
/mnt/backup-ursa at 7%.
Where TerraPulse actually lives
| Data | Path | Device | Size |
|---|---|---|---|
| Postgres 16 (:5433) | /var/lib/pgdata/16-main |
internal NVMe | 301 G |
| DuckDB staging | data/duckdb → /mnt/ursa/... |
USB spinning | 17 G |
| Event storehouse | data/event_storehouse → /mnt/ursa/... |
USB spinning | 8.1 G, 1,288,493 files |
| WSPR raw | /mnt/ursa/data/terrapulse/wspr |
USB spinning | 201 G |
| Other storehouses | data/*_storehouse (real dirs) |
root NVMe | ~1.5 G |
| Caches + rosters | data/* |
root NVMe | 11 G total |
| Repo (excl. data) | /home/bisenbek/projects/terrapulse |
root NVMe | 20 G |
There is a second Postgres cluster on this box: PG 15 on :5432, data
on /mnt/ursa/data/postgresql, 154 G. TerraPulse does not use it — TerraPulse is
PG 16 on :5433. Worth knowing before anyone tunes "the database".
What this means for scale-out
Correcting a claim in the first draft of this doc: Postgres is not on the root filesystem. It has its own NVMe partition with 132 G free at 70%. The database is not the thing filling root.
Root's 410 G is 314 G of /home (all tenants' project trees; TerraPulse's is
20 G of it), plus 54 G /usr and 26 G /opt. Root at 95% is still worth fixing,
but it is a home-directory cleanup problem, not a database-growth problem, and it
does not block the re-architecture.
The genuinely binding storage facts are:
/mnt/storageis at 99% with 4.8 GB free on an internal NVMe. That is the most urgent number on the box. Not TerraPulse's, but it is internal flash that the re-architecture would otherwise be a natural home for.- The hot derived data is on the slowest sensible medium. DuckDB staging and the 1.29 M-file event storehouse both sit on a USB-attached spinning disk at 126–221 MB/s sequential, and small-file random I/O on that link is far worse than the sequential figure suggests. This is precisely why taring the storehouse takes over two hours (§7) and why the index rebuild costs ~13 min. Moving the storehouse to NVMe is the single largest available performance win for the dex layer.
- Swap is fully consumed (31/31 GB). Adding worker processes lands here.
Both /mnt/ursa (3.7 T free) and /mnt/backup-ursa (1000 G free) have plenty of
capacity. Capacity is not the constraint on the spinning tier; latency is.
2. Process topology
Everything is systemd + nginx. No Docker, no CI/CD, no orchestrator. Pushing
to main deploys nothing.
nginx :443 (letsencrypt, terrapulse.info)
│
┌───────────────────┼────────────────────┬──────────────┐
│ │ │ │
location / /api/, /docs, /admin /ws,
│ /redoc, /openapi.json │ /api/v1/ws
▼ ▼ ▼ ▼
terrapulse_web terrapulse_api (same API proc) (same API proc)
127.0.0.1:4321 127.0.0.1:8111 FastHTML websocket
Astro 6 SSR FastAPI/uvicorn mounted proxy_read
node v24.0.1 python 3.13 at /admin _timeout 86400
│ │
│ ├── APScheduler (in-process, 29 named jobs + 51 fetchers)
│ └── SQLAlchemy async → PostgreSQL 16 + PostGIS :5433
│
└── fetches from PUBLIC_API_URL=http://127.0.0.1:8111
Independent listener services (not scheduled, never idle):
terrapulse-glm.service GOES-19 GLM lightning
terrapulse-blitzortung.service Blitzortung lightning
terrapulse-pulse.service pulse event streamer
Units
| Unit | State today | Notes |
|---|---|---|
terrapulse.service |
active | uvicorn, single process, no --workers |
terrapulse-web.service |
active | node dist/server/entry.mjs, standalone adapter |
terrapulse-glm.service |
active | listener |
terrapulse-blitzortung.service |
active | listener |
terrapulse-pulse.service |
active | listener |
terrapulse-backup.service |
failed (timeout) | see §7 — the work succeeded |
terrapulse-alert-log.service |
inactive (timer-driven) | fires 20:06 local |
Only the backup units are versioned in infra/systemd/. The other five live in
/etc/systemd/system/ and are not in the repo — that is a real gap for
scale-out, because the API and web units carry non-obvious environment
(TMPDIR=/mnt/ursa/data/terrapulse/tmp, the pyenv/nvm PATHs, the port).
Ports — the one that bites
| Thing | Port | Set where |
|---|---|---|
| PostgreSQL | 5433 (not 5432) | cluster config |
| FastAPI | 8111 | the systemd unit's ExecStart |
| Astro SSR | 4321 | unit env PORT + astro.config.mjs |
config.py and .env.example both declare API_PORT=8000. Nothing reads
it. Start the API on 8000 and the site renders empty with no error, because
the web app hardcodes PUBLIC_API_URL default http://127.0.0.1:8111. Either
wire the setting up or delete it — flagged as a known divergence in CLAUDE.md.
3. Data layer
| Item | Measured 2026-08-17 |
|---|---|
| Database size | 300 GB |
observations table (incl. indexes) |
251 GB |
observations row estimate |
~586.7 M (reltuples) |
| Registered datasources | 495 |
| Active datasources | 370 |
Postgres 16 + PostGIS is the source of truth. Per-source DuckDB files stage raw
data append-only; every normalized row carries duckdb_source_ref +
duckdb_row_id back to its staged origin. DuckDB is local and unreplicated —
only Postgres is backed up.
Geography is denormalized string columns on observations (continent,
country, country_code, state, city), populated at ingest from an
offline Natural Earth lookup. There is no geo_hierarchy table and no geo
foreign keys; that was ratified as the design on 2026-07-26 on read-cost grounds
at this row count. Do not wire network geocoding into the ingest path.
Counting is a scale-out hazard already handled: COUNT(*) on observations
takes ~30 minutes, so the homepage stats cache uses pg_class.reltuples and
scopes its distinct-metric count to a 30-day window.
4. Ingestion — three paths
| Path | Count | Where |
|---|---|---|
| Bespoke fetcher | 51 active (2 parked) | ingestion/fetchers/*.py + JOBS in scheduler.py:79 |
| AutoSense | ~440 | a row added via /admin; generic sniffing fetcher |
| Persistent listener | 3 | scripts/glm_listener.py, Blitzortung, pulse_streamer.py |
Most datasources are admin rows, not code. The datasources table is
production state that does not exist in git — the repo seeds ~45, production runs
495. "Add a data source" usually means adding a row, not writing a class.
Cadence distribution of the 51 bespoke fetchers:
- ≤5 min: 8 (USGS quakes 60s, NWS alerts 60s, GOES X-ray, DSCOVR solar wind, …)
- 5–60 min: 21
- 1–6 h: 7
- 6–24 h: 13
- weekly: 2
Parked, deliberately: fink_transients (host unreachable since 2026-05-09) and
emsc (non-commercial licence; USGS ComCat covers the same catalog).
On top of those, 29 named scheduler jobs handle dex sweeps, cache refreshes and maintenance — mostly 30-minute intervals, with the CelestialObjectDex rebuilds staggered weekly on Monday 08:00/08:15/08:30 UTC so they can't overlap.
The scale-out fact that matters here: APScheduler runs in-process inside the API. Every sweep, every rebuild, every fetch shares an event loop with the thing serving terrapulse.info, on a single uvicorn worker. That is the first structural thing to change if you want to scale the API independently — see §8.
5. The dex layer
Downstream of Postgres. Sweeps read normalized observations and file them into dexes — per-phenomenon indexed lists where one slot holds exactly one thing. Seven families; on disk today:
| Storehouse | Kinds |
|---|---|
data/event_storehouse/ |
41 (eq, tor, tc, gw, flare, cme, fema, hail, wildfire, …) |
data/location_storehouse/ |
14 (streamgauge, tide_gauge, magnetic_observatory, …) |
data/celestial_storehouse/ |
4 (neo, satcat, satellite_decay, …) |
data/platform_storehouse/ |
4 (aircraft, mobile_radiosonde, …) |
data/directory_storehouse/ |
1 (regulated_facility) |
Storage is one directory of slot records per kind plus an index rebuilt from
disk on read — crash-safe by construction, but the index rebuild is the known
cost centre (~101 MB / ~297k entries / ~13 min at last measure). Large kinds use
a spine-parquet mode instead of file-per-slot; the clean cutoff between the two
is still an open call, documented in docs/celestialdex-framework.md.
Two rules that constrain any feature work, both non-negotiable:
- Measured data only — no models, forecasts, projections, reanalysis, or interpolated estimates reach a dex. This is the platform's public claim. Directorydex is the one designed exception (registry facts, so measured dexes have something to resolve against).
- Scope docs are frozen before backfill and not tuned afterward. 52 of them
under
docs/scope-*.md. Changing a frozen threshold is a new, dated, documented decision — never an edit in passing.
6. The web/UI stack — the part you're replacing
As it stands
| Layer | Version / detail |
|---|---|
| Framework | Astro 6.0.5, output: 'server' (SSR) |
| Adapter | @astrojs/node 10.0.2, mode: 'standalone' |
| Runtime | Node v24.0.1 (nvm), engines pin >=22.12.0 |
| Styling | Tailwind 4.2.1 via @tailwindcss/vite + web/src/styles/global.css |
| Content | @astrojs/mdx 5.0.1; marked 17.0.4 for runtime markdown |
| Build output | web/dist, 2.9 MB |
| Pages | 77 (40 of them /data/*.mdx dex-descriptive pages) |
| Components | 10, all .astro — no React/Vue/Svelte anywhere |
| Layouts | 1 — Base.astro, 303 lines |
There is no client-side framework, no state library, no component library, and
no design tokens beyond Tailwind's defaults plus one hardcoded chrome colour
(#152942, "Neural Network Navy", single palette site-wide since 2026-06-08).
47 .astro files carry Tailwind classes; 6 still have inline <style> blocks.
The site's oversized, high-contrast styling is intentional — it is Mike's accessibility call, not an accident of defaults. A new UI stack should carry that forward rather than compress for density.
The API contract a new UI must satisfy
This is the useful part for a swap. Despite 13 route modules on the API side, the web app consumes a narrow surface:
GET /api/v1/health
GET /api/v1/alerts GET /api/v1/alerts/count
GET /api/v1/events/today GET /api/v1/events/today/count
GET /api/v1/datacards/graph/data
POST /api/v1/client-log
GET /api/v1/lab/workspaces/{slug}
GET /api/v1/lab/workspaces/{slug}/{paper|www|data|scripts}/{file}
WS /api/v1/ws (and /ws)
Everything else — /observations, /datasources, /tags, /geo,
/space_weather, /traffic — is API-only surface the site does not currently
touch. A UI rewrite is therefore a thin-contract job: nine endpoints plus a
websocket plus a static-file passthrough for lab workspace assets.
The gap worth knowing before you design: there is no HTTP surface for the
dex layer at all. 40 /data/* pages describe dexes in prose; none of them read
one. docs/dex-web-display.md designs the roster-cards / slot-detail-page
pattern and it was never built. If the new UI is meant to surface the dexes —
which is where most of the platform's actual content now lives — that API
needs building first, and it is the single highest-leverage thing on this list.
Deploy
scripts/deploy_web.sh builds web/dist and restarts the unit. Manual, gated on
Mike's approval because it touches the public site. A failed build leaves
production untouched. The Astro unit runs an ExecStartPost smoke test that
curls /, /lab, /status, /garden.
Measured now: web root responds 200 in 54 ms, API health {"status":"ok","db":true},
/docs 200.
7. Backups — and today's "failure", which wasn't one
Nightly terrapulse-backup.timer at 03:30 local (Persistent=true,
5-min jitter) → /mnt/backup-ursa/backups/, 14-day retention, three tiers:
pg_dump -Fc→postgres/— 17 GB, took 34 min today, symlinkedlatest.dump- Claude memory tar →
claude-memory/— 417 KB - Storehouse rsync snapshot (hardlinked deltas) →
storm-storehouse/ - Weekly off-box tarball + GitHub release asset (
eventdex-backuptag)
Today's run: every tier succeeded, then systemd killed the script. Timeline from the journal:
03:33:41 start
04:07:57 dump succeeded: 17GB in 2056s
04:07:59 memory snapshot: 417KB
05:39:22 storehouse snapshot: 1,288,545 files, 4.1GB copied
07:33:41 systemd: start operation timed out. Terminating. (SIGTERM)
It hit TimeoutStartSec (4h at the time) while taring 1.29 M small files off the
USB disk, leaving a 136 MB .tar.gz that gzip -t rejected.
The thing I was actually waiting to confirm is confirmed:
storehouse-2026-08-17-0333 survived, 8.0 GB on disk. Storehouse backups had
been silently self-deleting every night since 2026-08-02 — the fix shipped
2026-08-16 in 695880b was fixture-proven only, and this is its first live
proof. It works. Postgres dumps are unbroken back through 2026-08-10 at 16–17 GB
each.
Uncommitted work sits staged in the tree right now (scripts/backup_postgres.sh,
tests/test_backup_storehouse_rotation.sh, +97/-7) responding to this morning's
kill: the off-box tarball rolls weekly instead of nightly and is driven by the
age of the newest good tarball rather than a fixed weekday, and both tarballs are
written .partial → gzip -t → mv so a kill can never publish a
plausible-looking corrupt archive. The unit's TimeoutStartSec has been raised
4h → 6h as belt-and-braces. Staged, not committed — someone should land it.
Restore verification: scripts/verify_backup_restore.sh, monthly.
8. Scale-out — where the real ceilings are
Ordered by how soon they bind, based on what's measured above.
1. I/O latency on the dex layer, not capacity. The event storehouse
(1.29 M files) and the DuckDB staging tree both live on a USB-attached spinning
disk. Postgres is fine — it has its own NVMe with 132 G free. But every dex sweep,
index rebuild and backup tar pays USB-HDD random-I/O cost. Moving
/mnt/ursa/data/terrapulse/{event_storehouse,duckdb} onto internal NVMe is the
highest-leverage infrastructure change available, and it needs ~25 GB of flash
that does not exist free today (/mnt/storage has 4.8 G, root has 27 G).
Freeing space is the prerequisite. See §1.
2. Swap fully consumed (31/31 GB). The box is already reaching past physical memory with 34 GB in buff/cache. Adding a second uvicorn worker, a worker pool, or a heavier UI runtime lands directly on this. This is the constraint the process manager has to be designed against — a worker pool that assumes free RAM will push the box into swap thrash and take the other 32 tenants with it.
3. The scheduler shares a process with the API. One uvicorn worker, no
--workers, with APScheduler and every dex sweep inside it. The consequences:
the API cannot be scaled horizontally without either running duplicate schedulers
or splitting the scheduler into its own unit; a long sweep competes with request
serving; and a restart to pick up a scheduler change also drops the API (~3 min,
the import is slow). Splitting the scheduler into its own systemd unit is the
single cleanest structural win available and it is mostly a packaging change,
not a rewrite.
4. The event-loop discipline is load-bearing and must not be relaxed. ruff
ASYNC100/210/220/230 are selected specifically to catch sync I/O inside async
functions, and main.py sets loop.slow_callback_duration = 0.2 so a blocking
callback logs its frame. This came out of a real production bug class (#213–#217).
With scheduler and API in one process, one blocking call stalls the public site.
Don't disable these; push sync work off the loop with asyncio.to_thread.
5. Five of seven systemd units are not in the repo. Rebuilding this box today
means reconstructing the API and web units from memory, including the pyenv/nvm
paths, the TMPDIR override and the real port. Versioning them into
infra/systemd/ is cheap and should happen before any migration.
6. The storehouse index rebuild. ~13 min, race-prone, rebuilt from disk on read. Fine at current scale; it is the thing that breaks first if kind count or slot count grows sharply.
7. schedule_cron does not schedule anything. It is the sole input to
liveness alerting via croniter — editing it changes when you get paged, not
when a fetch runs. A source with no schedule_cron is silently skipped by the
liveness check entirely. Do not "fix" this by deleting the field; AutoSense
sources have no JOBS entry, so it is the only place their cadence expectation
can live.
Multi-tenancy note. ~33 nginx sites share this box. Any TerraPulse resource decision — a second worker, a Postgres tuning change, a new service — is a decision about the other 32 tenants too, and belongs to Brad rather than to TerraPulse work.
9. Suggested order of work
Given the three goals — iterate on UI/services, swap the UI stack, scale out:
- Land the staged backup fix. It's written and tested; leaving it uncommitted is the only thing between here and a fully green backup.
- Version the five missing systemd units into
infra/systemd/. Cheap, removes the largest single-point-of-knowledge risk. - Build the dex HTTP surface. It's the prerequisite for a UI that shows what
the platform actually contains, and
docs/dex-web-display.mdalready designs it. Without it, a new UI stack renders the same nine endpoints more prettily. - Split the scheduler out of the API process. Unblocks scaling the API and decouples "restart to change a sweep" from "drop the public site".
- Then swap the UI stack, against the thin contract in §6 — carrying the deliberate large/high-contrast styling forward.
- Disk and memory headroom — Brad's lane, but items 3–5 all assume it holds.
10. Provenance of this document
| Claim class | How it was obtained |
|---|---|
| Host resources | df -hT, df -i, free -g, nproc, /proc/cpuinfo |
| Drive type / topology | lsblk -o …,ROTA,MODEL,TRAN, findmnt, lsusb -t, /sys/block/*/queue/rotational |
| Drive throughput | hdparm -t --direct (read-only, ~3 s per device) |
| Postgres location | pg_lsclusters, du -sh /var/lib/pgdata |
| Units and timers | systemctl list-units, systemctl status, cat of unit files |
| Backup timeline | journalctl -u terrapulse-backup.service, ls of backup dirs |
| DB figures | psql against localhost:5433 (pg_database_size, reltuples) |
| Ingestion counts | parsed from scheduler.py JOBS + add_job ids |
| Dex kind counts | ls of data/*_storehouse/ |
| Web stack | web/package.json, astro.config.mjs, find/grep over web/src |
| API contract | grep for /api/v1 paths across web/src |
| Liveness | curl of :8111/api/v1/health, :4321/, :8111/docs |
Figures marked as estimates (reltuples, storehouse index size) are exactly
that. Everything else is a direct read.