Skip to content

"Vibe-Trading: Your Personal Trading Agent"

22.2 KB
Raw Blame
"""Read-only crypto L2 order-book depth tool (OKX / Binance spot, via ccxt).

Built to close a documented gap: ``src/swarm/presets/statistical_arbitrage_desk.yaml``
tells its microstructure analyst "There is no order-book depth or quote feed
here" because that is true for the equity data sources this project has for
free — but it is *not* true for crypto. OKX and Binance both publish full L2
order-book snapshots over public, no-auth REST, and this project already
depends on ``ccxt>=4.0.0`` (see ``pyproject.toml``) and already talks to both
venues for OHLCV (``backtest/loaders/okx.py``, ``backtest/loaders/binance_loader.py``).
This tool reuses that same connectivity — ccxt's unified ``fetch_order_book``
— instead of hand-rolling per-venue REST parsing.

Symbol convention matches the rest of this repo (see CLAUDE.md: "OKX pairs
use ``BTC-USDT`` format, hyphen, uppercase") and ``ccxt_loader._parse_ccxt_symbol``:
the caller passes ``BASE-QUOTE`` (e.g. ``BTC-USDT``), which is upper-cased and
translated to ccxt's ``BASE/QUOTE`` form before the request. Perpetuals
(``-PERP``) are out of scope here — this tool is spot-book depth only.

What ccxt's ``fetch_order_book`` actually returns (read from the installed
``ccxt==4.5.54`` source, not assumed): ``parse_order_book`` always returns
``{"symbol", "bids", "asks", "timestamp", "datetime", "nonce"}`` where
``bids`` is sorted descending by price and ``asks`` ascending — i.e. index 0
on each side is always the best price, by construction of
``Exchange.parse_order_book``. This module still re-sorts defensively before
using either side (cheap, and removes any reliance on that upstream
invariant holding for every exchange/version). ``timestamp`` is venue
end-to-end epoch milliseconds when the venue's payload carries one — for
Binance *spot* depth (``GET /api/v3/depth``) it does not: ccxt's own
``fetch_order_book`` reads a ``"T"`` field that only appears in the
futures/options response shapes (see ``ccxt.binance.fetch_order_book``
docstring examples), so ``timestamp`` comes back ``None`` for Binance spot.
When that happens this tool falls back to the local fetch time and marks
``timestamp_source`` accordingly, rather than silently passing a null-dated
snapshot to the caller (an order book is only valid for seconds — the reader
must be able to tell how fresh it is either way).

The network call is isolated in :func:`_fetch_raw_book` for exactly one
reason: tests monkeypatch that single function and never touch the network,
same pattern as ``tests/test_prediction_market_tool.py`` monkeypatching
``throttled_get_json`` on its own tool module.

Domain decisions worth being explicit about (also covered in the tests):

* **Units.** Every ``amount``/``amount_base`` is a quantity of the pair's
  BASE asset. Every ``*_notional_quote`` / ``notional_quote`` / price is in
  the pair's QUOTE asset — USDT for ``BTC-USDT``, but BTC for ``ETH-BTC``.
  These are never averaged or added together; the ``units`` block on every
  response spells this out so a caller cannot mix them by accident.
* **Mid price** is always ``(best_bid + best_ask) / 2``. If either side of
  the book is empty this tool refuses to compute a mid price (or anything
  downstream of it) rather than guessing — see the empty-book handling below.
* **Crossed book** (``best_bid >= best_ask``) is treated as a data error and
  rejected outright. A negative or zero spread is never returned.
* **Impact-cost sizing.** "A market order for $10,000 notional" is sized
  identically for both directions: ``base_qty_target = notional_quote /
  mid_price``, computed once from the pre-trade mid. The buy side then walks
  the (fully fetched, not just the displayed) ask ladder consuming that many
  base units; the sell side walks the bid ladder the same way. This keeps
  buy and sell directly comparable (same order size) instead of the sell
  side chasing a moving quote-proceeds target, which would make the two
  sides answer different questions.
* **Partial fills are always reported, never disguised as full fills.** If
  the fetched ladder (``_BOOK_FETCH_LIMIT`` levels — see below) cannot
  satisfy ``base_qty_target``, the response carries ``fully_filled: false``,
  the actual ``filled_base_qty`` / ``filled_notional_quote`` (which will be
  less than requested), and an explanatory ``note``. The average price and
  slippage are still computed over whatever *was* filled — a partial number
  is more honest than no number, as long as it is labelled partial.
* **Fetch depth vs. requested depth are different knobs.** ``levels``
  (1..``_MAX_LEVELS``) controls how many rows are echoed back in ``depth``
  and how many are used for the imbalance ratio. Impact cost always walks
  the full ``_BOOK_FETCH_LIMIT``-level ladder pulled from the exchange
  (100 levels — a valid discrete Binance depth tier and well under OKX's
  400-row default-endpoint cap), regardless of ``levels``, since a $10k
  order can easily need more rows than a human wants echoed back.
"""

from __future__ import annotations

import json
import logging
import math
from datetime import datetime, timezone
from typing import Any

from backtest.loaders.ccxt_loader import _CCXT_TIMEOUT_MS, _ccxt_proxy_config
from src.agent.tools import BaseTool

logger = logging.getLogger(__name__)

_EXCHANGES = ("okx", "binance")
_DEFAULT_EXCHANGE = "okx"

_DEFAULT_LEVELS = 10
_MAX_LEVELS = 50

# Rows pulled from the exchange per call. Fixed (not caller-configurable):
# impact-cost needs headroom beyond whatever `levels` the caller wants
# echoed back, and 100 is a valid discrete Binance order-book depth tier
# (5/10/20/50/100/500/1000/5000) while sitting comfortably under OKX's
# default-endpoint 400-row cap — one request works cleanly against both.
_BOOK_FETCH_LIMIT = 100

_DEFAULT_NOTIONAL_QUOTE = 10_000.0

_UNITS = {
    "amount_base": "quantity of the pair's BASE asset (e.g. BTC in BTC-USDT) — never a currency value",
    "price": "quote-asset price of 1 unit of base (e.g. USDT per BTC in BTC-USDT)",
    "notional_quote / *_notional_quote / filled_notional_quote": (
        "value denominated in the pair's QUOTE asset (USDT for BTC-USDT, but "
        "BTC for ETH-BTC — read the quote leg off the symbol, do not assume USD)"
    ),
    "relative_bps / slippage_bps": "basis points (1 bps = 0.01%), relative to mid_price at fetch time",
    "normalized_imbalance": (
        "(bid_notional_quote - ask_notional_quote) / (bid_notional_quote + ask_notional_quote), "
        "in [-1, 1]. Positive = more resting buy-side notional (bid-heavy), negative = ask-heavy"
    ),
}


def _fetch_raw_book(exchange_id: str, ccxt_symbol: str, limit: int) -> dict[str, Any]:
    """Fetch one L2 order-book snapshot via ccxt's public REST API.

    Isolated as its own function so tests can monkeypatch exactly this name
    and never open a socket — see the module docstring.

    Args:
        exchange_id: ``"okx"`` or ``"binance"`` (a ccxt exchange id).
        ccxt_symbol: Unified ccxt symbol, e.g. ``"BTC/USDT"``.
        limit: Number of levels per side requested from the venue.

    Returns:
        A ccxt ``OrderBook`` dict: ``{"symbol", "bids", "asks", "timestamp",
        "datetime", "nonce"}``.

    Raises:
        Exception: Whatever ccxt raises for a network, symbol, or venue error.
    """
    import ccxt

    exchange_cls = getattr(ccxt, exchange_id)
    config: dict[str, Any] = {"enableRateLimit": True, "timeout": _CCXT_TIMEOUT_MS}
    proxies = _ccxt_proxy_config()
    if proxies:
        config["proxies"] = proxies
    exchange = exchange_cls(config)
    return exchange.fetch_order_book(ccxt_symbol, limit)


def _error(message: str) -> str:
    """Build the failure envelope as a JSON string."""
    return json.dumps({"ok": False, "error": message}, ensure_ascii=False)


def _parse_symbol(raw: Any) -> tuple[str, str] | None:
    """Validate + normalize a ``BASE-QUOTE`` symbol into ``(display, ccxt)`` forms.

    Args:
        raw: Caller-supplied symbol, e.g. ``"btc-usdt"``.

    Returns:
        ``(normalized_hyphen_symbol, ccxt_slash_symbol)``, or ``None`` when
        ``raw`` is not a well-formed ``BASE-QUOTE`` string.
    """
    if not isinstance(raw, str):
        return None
    normalized = raw.strip().upper()
    parts = normalized.split("-")
    if len(parts) != 2 or not parts[0] or not parts[1]:
        return None
    if not parts[0].isalnum() or not parts[1].isalnum():
        return None
    return normalized, f"{parts[0]}/{parts[1]}"


def _coerce_int(value: Any) -> int | None:
    """Coerce a parameter to int, accepting the JSON shapes an LLM tends to send."""
    if isinstance(value, bool):
        return None
    if isinstance(value, int):
        return value
    if isinstance(value, float):
        return int(value) if value.is_integer() else None
    if isinstance(value, str):
        try:
            number = float(value.strip())
        except ValueError:
            return None
        return int(number) if number.is_integer() else None
    return None


def _coerce_float(value: Any) -> float | None:
    """Coerce a parameter to a finite float."""
    if isinstance(value, bool):
        return None
    if isinstance(value, (int, float)):
        number = float(value)
    elif isinstance(value, str):
        try:
            number = float(value.strip())
        except ValueError:
            return None
    else:
        return None
    return number if math.isfinite(number) else None


def _clean_levels(raw: Any) -> list[tuple[float, float]]:
    """Normalize a ccxt bids/asks array into sorted-by-caller ``(price, amount)`` pairs.

    Rows with a non-positive or non-finite price/amount are dropped rather
    than propagated into downstream arithmetic (a zero-amount row would
    silently no-op through the impact walk; a negative one would corrupt it).

    Args:
        raw: The ``bids`` or ``asks`` value from a ccxt order-book dict.

    Returns:
        A list of ``(price, amount)`` float pairs, in whatever order ``raw``
        had them (the caller sorts).
    """
    out: list[tuple[float, float]] = []
    if not isinstance(raw, list):
        return out
    for row in raw:
        if not isinstance(row, (list, tuple)) or len(row) < 2:
            continue
        try:
            price = float(row[0])
            amount = float(row[1])
        except (TypeError, ValueError):
            continue
        if not math.isfinite(price) or not math.isfinite(amount):
            continue
        if price <= 0 or amount <= 0:
            continue
        out.append((price, amount))
    return out


def _walk_book(levels: list[tuple[float, float]], base_qty_target: float) -> dict[str, Any]:
    """Consume ``levels`` (best price first) until ``base_qty_target`` base units are filled.

    Args:
        levels: ``(price, amount)`` pairs in walk order (best price first);
            the caller is responsible for that ordering.
        base_qty_target: Base-asset quantity the simulated market order wants
            to fill.

    Returns:
        ``{"filled_base_qty", "filled_notional_quote", "avg_price",
        "levels_used", "fully_filled", "unfilled_base_qty"}``. ``avg_price``
        is ``None`` only when nothing at all could be filled (e.g. the side
        was empty, though callers already reject that case upstream).
    """
    remaining = base_qty_target
    base_filled = 0.0
    quote_filled = 0.0
    levels_used = 0
    for price, amount in levels:
        if remaining <= 0:
            break
        take = min(amount, remaining)
        base_filled += take
        quote_filled += take * price
        remaining -= take
        levels_used += 1
    # Tolerance absorbs float noise from the running subtraction above; it is
    # not a relaxed pass/fail threshold on the caller's requested size.
    fully_filled = remaining <= 1e-9
    return {
        "filled_base_qty": base_filled,
        "filled_notional_quote": quote_filled,
        "avg_price": (quote_filled / base_filled) if base_filled > 0 else None,
        "levels_used": levels_used,
        "fully_filled": fully_filled,
        "unfilled_base_qty": max(remaining, 0.0),
    }


class OrderBookDepthTool(BaseTool):
    """Read-only crypto L2 order-book depth: raw levels, spread, imbalance, impact cost."""

    name = "orderbook_depth"
    repeatable = True
    is_readonly = True
    description = (
        "READ-ONLY crypto L2 order-book snapshot from OKX or Binance spot "
        "public REST (via ccxt) — the depth/quote feed equity sources here "
        "cannot provide. Returns: (1) raw bid/ask levels (price + base-asset "
        "amount, up to 'levels' per side); (2) bid-ask spread, absolute and "
        "relative in bps; (3) depth imbalance within 'levels' — "
        "bid-notional/ask-notional ratio and a normalized (B-A)/(B+A) score "
        "in [-1, 1]; (4) impact cost — for a simulated market order sized at "
        "'notional_quote' (in the pair's quote asset, e.g. USDT for "
        "BTC-USDT), the volume-weighted average fill price and slippage in "
        "bps vs. mid price, computed BOTH for a buy (walking asks) and a "
        "sell (walking bids). A snapshot is only valid for seconds — every "
        "response carries a timestamp and 'exchange'. A quote-side that "
        "can't fill the full requested notional reports 'fully_filled: "
        "false' with the actually-filled amount, never a fabricated full "
        "fill. An empty book, a one-sided-empty book, or a crossed book "
        "(best_bid >= best_ask, a data error) is refused with an explicit "
        "error rather than a guessed number. Spot only, no perpetuals. "
        'Example: {"symbol": "BTC-USDT", "exchange": "okx", "levels": 10, '
        '"notional_quote": 10000}.'
    )
    parameters = {
        "type": "object",
        "properties": {
            "symbol": {
                "type": "string",
                "description": (
                    "BASE-QUOTE pair, hyphenated and uppercase (OKX-style), "
                    "e.g. 'BTC-USDT' or 'ETH-BTC'. Required."
                ),
            },
            "exchange": {
                "type": "string",
                "enum": list(_EXCHANGES),
                "description": "Which venue's public order book to read.",
                "default": _DEFAULT_EXCHANGE,
            },
            "levels": {
                "type": "integer",
                "description": (
                    f"Number of price levels per side to return in 'depth' "
                    f"and to use for the imbalance calculation (1-{_MAX_LEVELS})."
                ),
                "default": _DEFAULT_LEVELS,
            },
            "notional_quote": {
                "type": "number",
                "description": (
                    "Simulated market-order size, in the pair's QUOTE asset "
                    "(USDT for BTC-USDT), used for the impact-cost "
                    "calculation. Must be positive."
                ),
                "default": _DEFAULT_NOTIONAL_QUOTE,
            },
        },
        "required": ["symbol"],
    }

    def execute(self, **kwargs: Any) -> str:
        """Fetch one order-book snapshot and return the depth/spread/imbalance/impact envelope.

        Args:
            **kwargs: ``symbol`` (required), plus optional ``exchange``,
                ``levels``, ``notional_quote``.

        Returns:
            A JSON string. On success: ``{"ok": true, "source": "ccxt",
            "market": "crypto", "units": {...}, "data": {...}}``. On failure
            (bad params, empty/one-sided/crossed book, or a request error):
            ``{"ok": false, "error": str}``.
        """
        parsed = _parse_symbol(kwargs.get("symbol"))
        if parsed is None:
            return _error(
                "symbol is required and must be BASE-QUOTE, hyphenated and "
                "uppercase, e.g. 'BTC-USDT' (got "
                f"{kwargs.get('symbol')!r})"
            )
        display_symbol, ccxt_symbol = parsed

        exchange_id = kwargs.get("exchange", _DEFAULT_EXCHANGE)
        if exchange_id not in _EXCHANGES:
            return _error(f"exchange must be one of {list(_EXCHANGES)}, got {exchange_id!r}")

        levels_raw = kwargs.get("levels", _DEFAULT_LEVELS)
        levels = _coerce_int(levels_raw)
        if levels is None or levels < 1 or levels > _MAX_LEVELS:
            return _error(
                f"levels must be an integer between 1 and {_MAX_LEVELS} "
                f"(got {levels_raw!r})"
            )

        notional_raw = kwargs.get("notional_quote", _DEFAULT_NOTIONAL_QUOTE)
        notional_quote = _coerce_float(notional_raw)
        if notional_quote is None or notional_quote <= 0:
            return _error(f"notional_quote must be a positive number (got {notional_raw!r})")

        try:
            raw_book = _fetch_raw_book(exchange_id, ccxt_symbol, _BOOK_FETCH_LIMIT)
        except Exception as exc:  # noqa: BLE001 - surface as error envelope
            return _error(f"{exchange_id} order book request failed for {display_symbol}: {exc}")

        if not isinstance(raw_book, dict):
            return _error(f"unexpected order book payload shape from {exchange_id}")

        bids = sorted(_clean_levels(raw_book.get("bids")), key=lambda lvl: lvl[0], reverse=True)
        asks = sorted(_clean_levels(raw_book.get("asks")), key=lambda lvl: lvl[0])

        if not bids and not asks:
            return _error(f"order book for {display_symbol} on {exchange_id} is empty (no bids or asks)")
        if not bids:
            return _error(
                f"order book for {display_symbol} on {exchange_id} has no bids — "
                "refusing to compute mid price / spread from a one-sided book"
            )
        if not asks:
            return _error(
                f"order book for {display_symbol} on {exchange_id} has no asks — "
                "refusing to compute mid price / spread from a one-sided book"
            )

        best_bid_price, best_bid_qty = bids[0]
        best_ask_price, best_ask_qty = asks[0]
        if best_bid_price >= best_ask_price:
            return _error(
                f"crossed book for {display_symbol} on {exchange_id}: "
                f"best_bid {best_bid_price} >= best_ask {best_ask_price}. "
                "This is a data error — refusing to compute a spread from it."
            )

        mid_price = (best_bid_price + best_ask_price) / 2.0
        spread_abs = best_ask_price - best_bid_price
        spread_bps = (spread_abs / mid_price) * 10_000.0

        timestamp_ms = raw_book.get("timestamp")
        if isinstance(timestamp_ms, (int, float)) and not isinstance(timestamp_ms, bool):
            timestamp = (
                datetime.fromtimestamp(timestamp_ms / 1000.0, tz=timezone.utc)
                .isoformat()
                .replace("+00:00", "Z")
            )
            timestamp_source = "exchange"
        else:
            # Binance spot depth carries no server timestamp field at all
            # (ccxt's parser only finds one in the futures/options response
            # shapes) — fetch time is the honest fallback, explicitly labelled
            # so a caller never mistakes it for a venue-stamped time.
            timestamp = datetime.now(tz=timezone.utc).isoformat().replace("+00:00", "Z")
            timestamp_source = "local_fetch_time"

        depth_bids = bids[:levels]
        depth_asks = asks[:levels]
        bid_notional = sum(p * a for p, a in depth_bids)
        ask_notional = sum(p * a for p, a in depth_asks)
        total_notional = bid_notional + ask_notional
        normalized_imbalance = (
            (bid_notional - ask_notional) / total_notional if total_notional > 0 else 0.0
        )

        base_qty_target = notional_quote / mid_price
        buy = _walk_book(asks, base_qty_target)
        buy["notional_quote_requested"] = notional_quote
        buy["slippage_bps"] = (
            ((buy["avg_price"] - mid_price) / mid_price) * 10_000.0
            if buy["avg_price"] is not None
            else None
        )
        sell = _walk_book(bids, base_qty_target)
        sell["notional_quote_requested"] = notional_quote
        sell["slippage_bps"] = (
            ((mid_price - sell["avg_price"]) / mid_price) * 10_000.0
            if sell["avg_price"] is not None
            else None
        )
        for side_name, side in (("buy", buy), ("sell", sell)):
            if not side["fully_filled"]:
                side["note"] = (
                    f"could not fill the full {notional_quote} notional within the "
                    f"{len(asks) if side_name == 'buy' else len(bids)} "
                    f"{'ask' if side_name == 'buy' else 'bid'} level(s) fetched "
                    f"(exchange fetch limit={_BOOK_FETCH_LIMIT}); filled_notional_quote "
                    "and avg_price reflect only the partial fill actually achieved. "
                    "This may mean genuinely thin liquidity, or just that the book "
                    "goes deeper than this snapshot's fetched window."
                )

        data = {
            "exchange": exchange_id,
            "symbol": display_symbol,
            "ccxt_symbol": ccxt_symbol,
            "timestamp": timestamp,
            "timestamp_source": timestamp_source,
            "best_bid": {"price": best_bid_price, "amount_base": best_bid_qty},
            "best_ask": {"price": best_ask_price, "amount_base": best_ask_qty},
            "mid_price": mid_price,
            "spread": {"absolute_quote": spread_abs, "relative_bps": spread_bps},
            "depth": {
                "levels": levels,
                "bids": [{"price": p, "amount_base": a} for p, a in depth_bids],
                "asks": [{"price": p, "amount_base": a} for p, a in depth_asks],
            },
            "imbalance": {
                "levels": levels,
                "bid_notional_quote": bid_notional,
                "ask_notional_quote": ask_notional,
                "ratio_bid_over_ask": (bid_notional / ask_notional) if ask_notional > 0 else None,
                "normalized_imbalance": normalized_imbalance,
            },
            "impact_cost": {
                "notional_quote": notional_quote,
                "base_qty_target": base_qty_target,
                "buy": buy,
                "sell": sell,
            },
            "levels_available": {"bids": len(bids), "asks": len(asks)},
        }
        return json.dumps(
            {"ok": True, "source": "ccxt", "market": "crypto", "units": _UNITS, "data": data},
            ensure_ascii=False,
        )