Case Study: An AI Portfolio Manager That Has to Show Its Work

Why we built a trading agent we never intend to fund

Trader is an internal R&D project: an AI portfolio manager that runs on a schedule, reads the market, decides, executes against paper and simulated accounts, and writes a plain-English decision journal explaining every move. No real money touches it, and that's by design — the execution layer is hard-coded to paper endpoints.

We built it because trading is the most honest eval environment we've found for agentic AI. There's numeric ground truth (P&L), an unforgiving benchmark (buy-and-hold), a firehose of noisy tempting data, and a strong pull toward the classic agent failure mode: doing something just to look busy. If a pattern survives here, it survives in a client's back office.

The product is not the trading. The product is the journal — a dated record of what the agent saw, what it decided, why, and how that stacked up against doing nothing. Every design decision below serves that.

Code computes, the model reasons

The first rule we enforced: the model never does arithmetic over price series. LLMs know technical-analysis vocabulary fluently and compute it terribly — ask one to derive RSI from 90 days of candles and you'll get a confident, wrong number. So every indicator is computed in code by a standard library, handed to the model as a finished snapshot, and the system prompt says explicitly: trust these numbers, never recompute them.

This division — deterministic code for anything with a right answer, the model for judgment over the results — is the single most transferable pattern in the project. It's the same split we use for document extraction, pricing engines, and scheduling agents. If a subtask has a verifiable correct output, it should never be generated by a language model.

One agent, six venues

The agent runs against six venues behind a pair of narrow interfaces: a Market (data) and a Broker (execution). One venue is a real paper brokerage account for stocks and ETFs. The other five are simulated brokers we wrote ourselves — a crypto-exchange ledger priced from live public order-book data, and four on-chain DEX venues (Solana, Ethereum, Base, BNB Chain) priced from live pool data.

The simulated brokers model friction honestly: exchange fees, slippage, gas, and — on the DEX venues — constant-product price impact against actual pool liquidity, with orders refused outright above 3% impact. A simulator that fills you at mid-price with no fees will make any strategy look brilliant. Most of the "our agent beats the market" demos floating around fail exactly this test.

The venue abstraction also meant the agent itself never changed as we added chains. Tools are registered only when a venue's Market provides them, so the same agent loop runs a blue-chip equity mandate at 9:30 and a small-cap DEX mandate at midnight, each seeing only the tools that make sense there.

Guardrails live in code, not prompts

The mandate — budget, risk tolerance, universe, exclusions — is enforced where the model can't argue with it. The budget cap is checked inside the order function, which refuses buys past the limit. The equity venue refuses crypto orders in code. On-chain buys require a token-safety check (contract analysis, liquidity locks, holder data from public scanners) before the order path opens at all.

The prompt also says all of this, because prompts shape behavior. But prompts are a preference, not a control. Anything that must not happen gets a hard check in the execution path. We build client agents the same way: the LLM proposes, deterministic code disposes.

One more prompt-level rule that pays off everywhere: doing nothing is a valid run outcome, stated explicitly. Without it, an agent invoked on a schedule invents work. With it, a meaningful share of runs end with a journal entry explaining why the right trade today was no trade.

The feedback loops are the interesting part

A one-shot agent is a demo. This one closes three loops:

Reflection. On its own schedule, a stronger model reviews the live record — equity curve versus the benchmark, drawdowns, fills, the journal itself — and proposes lessons. A lesson isn't trusted immediately: it must be re-confirmed by a later reflection before it becomes active, and two misses retire it. Only active lessons are injected into future runs. That lifecycle — propose, confirm, act, retire — is our answer to the failure mode where an agent "learns" a spurious pattern from one lucky week and compounds it.

Human hunches. The operator can drop free-text intuition into the system — "I think small caps are stretched" — scoped to one venue or all, and every subsequent journal entry must say whether it acted on each open hunch. This is the lightest-weight human-in-the-loop design we've shipped: no approval gates that stall the loop, but no silently ignored guidance either.

The scorecard. Weekly, per venue: P&L, return versus benchmark, deployment, trade count, which lessons changed, and what the run cost in model spend. Cost sits next to performance on purpose. An agent whose reasoning bill outruns its edge is a losing agent regardless of its win rate.

Replay, not vibes

Weeks of paper trading in an uptrend prove nothing — everything works in an uptrend. So the comfort test is a replay engine: run the full agent, same tools and prompts, over historical windows like the 2020 crash, the 2022 rate shock, and the FTX collapse, with dates redacted and event knowledge withheld, then compare the equity curve to buy-and-hold.

We'll be straight about the known leak: a frontier model can often infer the era from price levels alone — Bitcoin at $20,000 narrows the calendar considerably — so replay results are a stress test, not a clean backtest, and we treat them that way. Documenting the limitation is the point. An eval whose weaknesses you can't state is an eval you can't trust.

What it costs to run

Model routing is tiered: a mid-tier model handles every scheduled trading run, the frontier model is reserved for the low-volume, judgment-heavy work (reflections, scorecards, research summaries), and the small model summarizes filings and token research into cached briefs. Nearly all market data comes from free public sources — Treasury yields, BLS releases, SEC EDGAR, options chains, on-chain analytics — with web search rationed to a few queries per run for dealbreaker checks only. The whole system runs for less per month than a single seat of most SaaS tools.

What transfers to client work

Trader will never manage real money, and we're not in the trading-software business. We ran this experiment because every hard question in production agents shows up here with a scoreboard attached, and the answers generalize:

• Give the model computed facts, not raw data, and forbid it from redoing the computation.
• Put every hard constraint in the execution path, not the prompt.
• Make "no action" an explicit, first-class outcome.
• Let the agent learn, but make learning earn its way in through confirmation — and make it reversible.
• Measure cost next to outcomes, in the same report.
• Stress-test against history and state your eval's weaknesses out loud.

If you're building an agent that acts on real systems — inventory, pricing, scheduling, claims — these are the load-bearing decisions. The model is the easy part.