Skip to content

Free tool · Backtest Honesty Check

Is your LLM trading agent's backtest telling the truth?

Six checks before you put real capital behind it: memorisation, costs, look-ahead, market regimes, paper vs live, and sample size. Add your trade log for sharper checks, or just describe the backtest. No exchange account, no keys, no upload.

Nothing you enter leaves your browser.

Your model and backtest
Trades

With no trade log, these three numbers stand in for it.

Your result

2 of 6 checks fail, 2 warn, 0 pass, 2 not run.

  1. 1. Memorisation leak

    fail

    The whole window (2025-01-01 to 2025-06-30) is before the model's cut-off (2025-10-01). The model may have learned what happened next, so good results can be recall rather than skill.

  2. 2. Fee and slippage realism

    fail

    Constant-product estimate of price impact (your typical trade size and pool): 243.9 bps, against 5.0 bps assumed. Your assumption is less than half the estimate. The estimate excludes pool fees, gas and MEV.

  3. 3. Look-ahead fields

    not run

    Upload or paste the backtest trade log to scan it for fields the agent could not have known at decision time.

  4. 4. Regime coverage

    warn

    Without prices this uses window length only, so it can't confirm the regimes. 181 days may cover only one or two regimes.

  5. 5. Paper vs live drift

    not run

    Add a live or paper-trading log to compare its fills with the prices the backtest assumed.

  6. 6. Sample size

    warn

    80 trades. Under 100, one or two lucky trades can carry the result.

P&L under realistic costs

P&L the backtest reports
12,000 USD
Slippage re-priced at (median)
243.9 bps
Extra cost
9,556 USD
Adjusted P&L
2,444 USD

Slippage is re-priced at the larger of your assumption, the constant-product estimate and the live drift, on 400,000 USD traded. Fees stay as you assumed. This re-prices costs only; it is not a forecast.

What this means

  • The symptom is "great backtest, worse live". The cause is usually one of these six: the model already knew the period, costs set too low, a future value in the inputs, one market regime, a paper-to-live gap, or too few trades.
  • An LLM may have effectively memorised what happened in the window it is tested on, so a strong backtest can be recall, not skill.
  • Markets change regime. What worked in one stretch often stops working in the next.
  • LLMs are weak at reading raw numeric price series, so how prices reach the model matters as much as the strategy.
  • The honest test is live: tiny size, kill rules, and logging that compares paper with live trade by trade.

How each check works

  • Memorisation leak: the share of your window on or before the training cut-off you entered. Half or more fails.
  • Fee and slippage realism: price impact ≈ size / (liquidity / 2 + size), a constant-product estimate, compared with your slippage assumption.
  • Look-ahead fields: column names that often hold future values, prices observed after the decision time, and trades outside the window.
  • Regime coverage: annualised realised volatility in each calendar month for your most-traded pair, sorted into three bands.
  • Paper vs live drift: each live fill matched to a backtest trade with the same pair and side within one hour.
  • Sample size: under 30 trades fails, under 100 warns.

What this tool can't see

  • It works only from the dates, costs and logs you gave. It can't see your prompts, data pipeline or what the model saw at each decision.
  • The cut-off is the date you entered. Retrieval, tools or fine-tuning can still give the model later information.
  • Price impact is a constant-product estimate. Concentrated-liquidity pools, order books, gas and MEV behave differently.
  • Volatility bands come from your own trade prices, not a market data feed.
  • Passing every check does not mean a strategy will make money. It means these six weak spots were not found.

Before it trades live

A backtest can't show whether your agent could sell what it buys. Thin pools, sell taxes and honeypots only show up on the way out. Before it trades live: can it get back out of these tokens?

Want a second pair of eyes on one real workflow?

This tool checks six weak spots in the backtest you described from what you typed. It can't see your code, keys, contracts or production traffic. If the result shows gaps, send me one real workflow. In my free 48-hour integration review I go through it end to end and send you a written report: fit, dependencies, permission risks, acceptance tests and a build or don't-build-yet call.

Request the free 48-hour review (opens Cal.com in a new tab)

With the box ticked, the booking link opens Cal.com with a short summary of your result in the notes field, which you can edit or delete. Cal.com's privacy and cookie policies apply there.

Contact: Shilpa Goel, [email protected]

This is an engineering checklist, not a security audit, legal, compliance or investment advice.