Tracking Market Zeitgeist
A market-intelligence pipeline being re-architected in place: a ReAct orchestrator schedules five replaceable Experts — Collection, Enrichment, Novelty, Evaluation, Signal — over the same deterministic batch pipeline already running in production, and records every decision as a replayable reasoning trace. An env-gated A2A server then exposes the results to the rest of the ecosystem — and, in the household's first reverse data flow, TMZ reads Trader Joe's own watchlist back, so its scanning converges on what the portfolio actually holds instead of a static ticker list.
Share this case study →The hard part: re-architecting a live production pipeline in place — every expert ships first as a byte-identical wrapper, proven against a consistency test, before any orchestration logic is allowed to change what actually runs.
ReAct orchestrator scheduling five replaceable Experts over the same deterministic batch pipeline already running in production
Python, Postgres, Dagu (batch trigger), Grafana
The production cycle today is already stage-structured: main.py's cycle driver runs schema setup, ticker resolution, scrape-and-validate, persist, enrichment, three self-healing backfills, evaluation, and signal export in sequence, with every stage writing a result artifact into the pipeline_runs table. That structure is the foundation the re-architecture builds on, not something it discards — stages become Experts, and the cycle driver becomes an orchestrator. Today, though, that structure lives inside one 1,047-line god-module with no stable seam: stage wrappers take concrete functions as parameters, not an interface an alternative implementation can satisfy. Every Expert's first landing is a pure refactor of that same proven function — byte-identical behavior, verified before any orchestration logic is switched on — so a system already running in production tonight evolves without ever going dark.
Collection, Enrichment, Novelty, Evaluation, and Signal each implement a minimal Expert protocol and wrap the pipeline's existing, proven logic — Collection owns scrape, validate, and persist; Enrichment owns L1 sentiment plus L2 deep-read plus backfills; Novelty owns fingerprint dedup and scoring; Evaluation owns T+1/T+5 accuracy against real market outcomes; Signal owns the export contract Trader Joe already reads directly from Postgres. Each is independently unit-testable and independently replaceable — a future Finnhub or RSS collector becomes a second collection Expert implementation, not a scraper rewrite. They deliberately stay one flat level: no Expert spawns a sub-expert, the same orchestrator-plus-expert shape used across every other system in this portfolio.
The orchestrator runs a bounded thought-action-observation loop whose default policy is the current fixed stage order — ReAct is the mechanism here, not a license for nondeterminism, and a consistency test asserts identical database writes with the orchestrator on or off. Every step writes into a new reasoning-trace table: which expert ran, what it considered, what it did, a redacted observation payload, and a correlation ID shared with the A2A layer, so a cross-system call and the internal reasoning that produced it trace back to one key. A replay CLI reconstructs any cycle end to end from that table — real audit capability where, today, only counts and durations survive a run, never the reasoning behind a label.
An env-gated peer server exposes TMZ's signal contract to both Trader Joe and Family CFO, and its accuracy contract to Trader Joe, serialized through a privacy whitelist stricter than the underlying export table — source URLs and raw payloads never leave the process. It runs dual-channel on purpose: the existing Postgres direct-read keeps working exactly as it does today, and the HTTP surface is an additive, independently toggled layer with zero behavioral change until a peer actually switches its read. TMZ is a consumer as well as a provider — it reads Trader Joe's own watchlist back, the ecosystem's first reverse data flow, so its scanning narrows to what the household actually holds instead of a static ticker list.
Every prompt TMZ's LLM calls use today is a hardcoded string constant inside the enrichment module; the re-architecture externalizes them to a versioned prompt store keyed by language and name, with an env-overridable root and an inline fallback so a mispackaged image degrades instead of crashing. The same discipline applies to everything else an operator might want to tune — LLM provider and model per agent, the score-weighting formula, novelty and evaluation thresholds — moved from code constants into one YAML config file under env-over-yaml-over-default precedence, so a canary deployment can point at an alternate file and a recalibration becomes a reviewable one-line diff instead of a rebuild. It also retires a real, repeat-offending failure mode: new files silently failing to deploy because the Dockerfile and the CI workflow each hand-enumerate their own file list — the package restructuring collapses both into one directory copy.
Python, Postgres, OpenRouter (DeepSeek V4 Flash), Dagu-scheduled batch cycles, yfinance ingestion, self-hosted Gitea Actions CI/CD.