Tracking Market Zeitgeist
A market-intelligence pipeline being re-architected in place. The deterministic cycle runs about 48 times a day in production today; beside it sits the next-gen layer — a ReAct orchestrator over five replaceable Experts (Collection, Enrichment, Novelty, Evaluation, Signal) plus a replayable reasoning-trace store — built, tested, and deliberately still gated off, because the seam ships before it takes over. An env-gated A2A server exposes the results to the rest of the ecosystem, and has carried thousands of successful protocol calls to Trader Joe across three live capabilities.
Share this case study →The hard part: re-architecting a live production pipeline in place — every expert ships first as a byte-identical wrapper, proven against a consistency test, before any orchestration logic is allowed to change what actually runs.
ReAct orchestrator over the five replaceable Experts, gated behind a single environment flag. The production daemon has never set it, so each of the ~48 cycles a day still runs the deterministic pipeline these Experts wrap — which is exactly the migration order this rearchitecture committed to.
Python, Postgres, Dagu (daily scoring DAG), Grafana
The production cycle today is already stage-structured: main.py's cycle driver runs schema setup, ticker resolution, scrape-and-validate, persist, enrichment, three self-healing backfills, evaluation, and signal export in sequence, with every stage writing a result artifact into the pipeline_runs table. That structure is the foundation the re-architecture builds on, not something it discards — stages become Experts, and the cycle driver becomes an orchestrator. Today, though, that structure lives inside one 1,047-line god-module with no stable seam: stage wrappers take concrete functions as parameters, not an interface an alternative implementation can satisfy. Every Expert's first landing is a pure refactor of that same proven function — byte-identical behavior, verified before any orchestration logic is switched on — so a system already running in production tonight evolves without ever going dark.
Collection, Enrichment, Novelty, Evaluation, and Signal each implement a minimal Expert protocol and wrap the pipeline's existing, proven logic — Collection owns scrape, validate, and persist; Enrichment owns L1 sentiment plus L2 deep-read plus backfills; Novelty owns fingerprint dedup and scoring; Evaluation owns T+1/T+5 accuracy against real market outcomes; Signal owns the export contract Trader Joe already reads directly from Postgres. Each is independently unit-testable and independently replaceable — a future Finnhub or RSS collector becomes a second collection Expert implementation, not a scraper rewrite. They deliberately stay one flat level: no Expert spawns a sub-expert, the same orchestrator-plus-expert shape used across every other system in this portfolio.
The orchestrator runs a bounded thought-action-observation loop whose default policy is the current fixed stage order — ReAct is the mechanism here, not a license for nondeterminism, and a consistency test asserts identical database writes with the orchestrator on or off. Every step writes into a new reasoning-trace table: which expert ran, what it considered, what it did, a redacted observation payload, and a correlation ID shared with the A2A layer, so a cross-system call and the internal reasoning that produced it trace back to one key. A replay CLI reconstructs any cycle end to end from that table — real audit capability where, today, only counts and durations survive a run, never the reasoning behind a label. None of this is running in production yet: the orchestrator sits behind a single environment flag the production daemon has never set, so today's ~48 cycles a day still take the deterministic path these Experts wrap, and the trace table is still empty. That ordering is deliberate — the seam ships and gets proven before it takes over.
An env-gated peer server exposes TMZ's signal contract to both Trader Joe and Family CFO, and its accuracy contract to Trader Joe, serialized through a privacy whitelist stricter than the underlying export table — source URLs and raw payloads never leave the process. It runs dual-channel on purpose: the existing Postgres direct-read keeps working exactly as it does today, and the HTTP surface is an additive, independently toggled layer with zero behavioral change until a peer actually switches its read. TMZ is built to be a consumer as well as a provider: the reverse path that reads Trader Joe's own watchlist back is implemented and contract-frozen on this side, and stays dark until Trader Joe enables that capability on its own. Until it does, scanning still runs from the configured ticker list — the same additive, zero-behavioral-change rule the provider direction follows.
Every prompt TMZ's LLM calls use today is a hardcoded string constant inside the enrichment module; the re-architecture externalizes them to a versioned prompt store keyed by language and name, with an env-overridable root and an inline fallback so a mispackaged image degrades instead of crashing. The same discipline applies to everything else an operator might want to tune — LLM provider and model per agent, the score-weighting formula, novelty and evaluation thresholds — moved from code constants into one YAML config file under env-over-yaml-over-default precedence, so a canary deployment can point at an alternate file and a recalibration becomes a reviewable one-line diff instead of a rebuild. It also retires a real, repeat-offending failure mode: new files silently failing to deploy because the Dockerfile and the CI workflow each hand-enumerate their own file list — the package restructuring collapses both into one directory copy.
Python, Postgres, OpenRouter (DeepSeek V4 Flash), Dagu-scheduled batch cycles, yfinance ingestion, self-hosted Gitea Actions CI/CD.