Finding connections across the dexes — a simple, elegant funnel
A starting-point note. We'll return to this and refine it.
The problem, honestly
The reason a growing data store like this turns into a rat's nest isn't mysterious — it's arithmetic. Every dex you add grows the number of possible pairings faster than it grows the number of dexes.
- 10 dexes → 45 possible pairs
- 20 dexes → 190 pairs
- 50 dexes → 1,200+ pairs
And that's only pairs, before you get to three-way coincidences. If the "algorithm" is check everything against everything, you've built a conspiracy corkboard with red string between every pin. It will be bursting with connections and almost all of them will be junk.
So elegance here is not a clever formula. It's a funnel that throws most pairs out cheaply and only ever puts a handful on a human's desk.
The funnel — three gates
Gate 1 — Do they even share an axis? Two dexes can connect only if they line up on something — usually the same dates, or the same places, or both. A meteorite-falls dex and a Kp-index dex share time. A stream-gauge dex and a fireball dex mostly share nothing. This gate alone kills most of the noise for free — it's just "can I lay these two graphs on the same timeline (or the same map)."
Gate 2 — Lined up, do they actually move together? Now the cheap statistics: do the highs tend to land on the highs. A quick yes / almost / no. Most survivors of Gate 1 die here.
Gate 3 — Is there a believable reason? The one a computer can't do and a human can. A number that co-moves with no story behind it is usually two things quietly driven by the same season — or pure coincidence. This gate is taste, and it's the whole reason the wall stays clean: only a human decides which survivors are worth a paper.
The elegance is that Gates 1 and 2 are dumb and automatic, and the human only ever looks at what limps through to Gate 3.
The AI/human collaboration — two ways to run the funnel
Genuinely different. Worth deciding first, because it flavors everything.
A. Human leads with a hunch, AI tests it. You say "I bet geomagnetic storms and UFO reports move together" — a reason first — and the algorithm runs it through the gates and reports back, loudly including when the answer is no. Fewer swings, each one meaningful. Nulls are real findings.
B. AI surfaces the coincidences, human judges them. The algorithm quietly runs Gates 1–2 across all the pairs and hands over a short list of "these five actually co-move." The human decides which have a believable story. More surprising leads, but more junk to sift — and more discipline needed so a pretty coincidence doesn't sneak into a paper.
Good shops end up doing both, but you start with one.
Recommended first dive
Start with A. Pick one pair of dexes you're curious about, walk it end to end through all three gates together, and watch the funnel actually work on something you care about. One honest pass teaches more than a dashboard of 1,200 correlations.
Guardrails against the rat's nest (for later)
- The funnel is narrow at the top by design — most pairs never reach a human.
- A co-movement is a lead, never a conclusion. Gate 3 (mechanism) is mandatory before anything becomes a paper.
- Beware common-cause: two series both driven by season/solar-cycle/time-of-day will co-move without being connected.
- Prefer the dumbest thing that could work; add complexity only when the dumb thing fails.