Public eval methodology

"Does an agent actually work on Crank" should be an answerable, checkable question -- not a claim. This page documents how we measure it: a reference agent, run in paper mode, over a fixed window, against point-in-time market data, with a full cost model and a strategy-consistency score alongside the raw outcome. The harness is a script in the repo (scripts/eval/run_reference_eval.py); anyone can run it and get the same numbers.

Measured outcomes, not promises

Every number on this page is a measured outcome over one fixed eval window, on one reference strategy, in paper mode. It is not a return, a profit projection, or an expectation of future performance (see Fee schedule and the platform's language rules). Simulated behaviour over a single historical window does not indicate what any agent will do going forward.

What "reference agent" means here

The harness runs a minimal, fully-specified implementation of the same loop shape the Crank portfolio loop skill teaches for portfolio_loop: observe -> assess -> plan -> simulate -> execute -> verify -> journal. For this eval it is an SMA-crossover long/flat strategy on a single asset -- deliberately simple, so every step in the report is auditable by hand. It is not a "best" strategy; it is a reference implementation that exercises the full loop and the full cost model.

Eval discipline (why the harness is built this way)

Three findings from 2025-26 live-trading-agent eval literature (LiveTradeBench, PortBench, CLQT) shaped the harness:

1. Point-in-time data only -- no look-ahead. At step i the agent only ever sees OHLCV bars [0..i]. The eval loop in run_reference_eval.run_reference_eval is structured so a later bar is never indexed before its own step -- there is no code path that can leak a future price into a decision.
2. Strategy-consistency scoring alongside outcome. A good-looking outcome that came from the agent ignoring its own stated plan is not evidence the loop works. Every step records both the planned intent (from assess/plan) and the executed action (after any guardrail veto -- the reference run uses a position-flip cooldown guardrail), and strategies.backtest.score_strategy_consistency reports the match rate. A mismatch is not a failure of the harness; it is the guardrail working and the harness being honest about it.
3. Full cost modeling. Every paper fill is charged the real cost stack an agent would pay live: the platform's technology service fee (never "trading commission" -- see Fee schedule), a slippage haircut, and a Solana priority-fee estimate. strategies.backtest.compute_trade_cost computes this per fill; nothing here nets fees out of the reported outcome.

Running it

# Reproducible, offline, no API key -- runs against the bundled frozen fixture.
python3 scripts/eval/run_reference_eval.py

# Replay against a freshly recorded pull instead of the frozen fixture
# (needs Django settings configured + BIRDEYE_API_KEY).
python3 scripts/eval/run_reference_eval.py --data-source live \
    --window-start 2026-06-01T00:00:00Z --window-end 2026-06-06T00:00:00Z

# Write the full JSON report (per-step planned/executed log, trade-by-trade
# cost breakdown, equity curve) alongside the printed summary.
python3 scripts/eval/run_reference_eval.py --out /tmp/eval_report.json

The default (fixture) mode is the reproducibility guarantee: same inputs, same seeded reference data, same deterministic strategy -> byte-identical output every run, on any machine, with no network access. --live-fees swaps the hand-built LOCKED fee-schedule snapshot for a live read of the admin-updatable FeeShareConfig row, for checking the harness against current governance parameters rather than the schedule at time of writing.

Reference run (fixed window)

Window: SOL/USDC, 1h bars, 2026-06-01T00:00Z .. 2026-06-06T00:00Z (120 bars), reference SMA(8)/SMA(24) crossover, $10,000 starting paper capital, 4-bar flip cooldown guardrail, scripts/eval/run_reference_eval.py default (fixture) mode:

Metric	Measured value
Bars evaluated	120
Trades executed	4
Starting capital	$10,000.00
Ending equity	$9,728.2524
Measured outcome over eval window	-$271.7476 (-2.7175%)
Total cost paid (fee + slippage + priority)	$354.8881
Strategy-consistency score	1.0 (120/120 steps, 0 mismatches)

This is one run, on one fixed window, on one simple reference strategy. It is published so the harness itself -- point-in-time discipline, cost accounting, consistency scoring -- is checkable, not as a claim about what any live agent will earn. Re-run the command above to reproduce these exact numbers, or point --data-source live at a different window to measure a different one.

Recall Network competition assessment

Recall Network runs verifiable paper-trading arenas with a public leaderboard -- infrastructure that overlaps with, rather than competes against, this eval harness (per the harness spec's MB#18289 research: complementary, not competitive). This section is a recommendation for Andrew, not an entry -- entering any Crank-branded agent on an external platform is a public representation of the brand and stays a deliberate, Andrew-gated decision, not an auto-entry from this harness.

Recommendation: go, at low priority, once the reference harness above has a second (non-toy) reference strategy.

Reasoning:

• Complementary, not competitive. Recall's leaderboard is a third-party verifiable venue; it does not replace this page's self-published, reproducible-from-a-script methodology. Entering does not require retiring or changing anything documented above.
• Low direct cost. The reference agent + cost model already built for this ticket (strategies.backtest, scripts/eval/run_reference_eval.py) is most of the integration surface -- the incremental work is an adapter from Recall's competition API/format to the existing paper-mode loop, not a new agent.
• Effort estimate: small-to-medium (1-2 dev-days). Scope: (1) read Recall's competition entry API/rules, (2) adapt the reference loop's execute step to Recall's paper-trading submission format, (3) decide which reference strategy to enter (the SMA-crossover reference above is intentionally minimal -- entering it as-is risks reading as "this is Crank's best effort" rather than "this is the harness's audit fixture"; recommend at least one composite/regime-aware strategy from backend/strategies/composite.py as the entrant instead).
• Decision points for Andrew before entry:
• Which strategy represents Crank publicly on the leaderboard (not the bare SMA-crossover reference -- see above).
• Whether Recall's leaderboard copy/branding constraints are compatible with language rules 5-8 (no "returns"/performance-promise framing) -- needs a one-time review of Recall's public profile/bio fields before entry, since that copy is Recall's UI, not ours.
• Timing relative to this harness's own roadmap -- entering before a second reference strategy exists risks the SMA-crossover audit fixture becoming the public face of "Crank on Recall" by default.

No entry has been made. This is a documented go/no-go for Andrew to action, not a completed integration.