# Benchmarking

Fynd ships with a benchmark tool (`fynd-benchmark`) for load-testing a solver and comparing output quality between two solver instances. Both features live in `tools/benchmark/` and run against live solver instances.

{% hint style="info" %}
**Prerequisite:** You need a running solver before using any benchmark command. See [Quickstart](/get-started/quickstart.md) for setup instructions.
{% endhint %}

## Load Testing

Measures latency (round-trip, solve time, overhead) and throughput for a single solver instance.

```bash
cargo run -p fynd-benchmark --release -- load [OPTIONS]
```

{% hint style="warning" %}
Always build with `--release`. Debug builds produce misleading latency numbers.
{% endhint %}

### Options

| Flag              | Default                 | Description                              |
| ----------------- | ----------------------- | ---------------------------------------- |
| `--solver-url`    | `http://localhost:3000` | Solver URL to benchmark against          |
| `-n`              | `1`                     | Number of requests to send               |
| `-m`              | `sequential`            | Parallelization mode                     |
| `--requests-file` | *(none)*                | Path to JSON file with request templates |
| `--output-file`   | *(none)*                | Output file for JSON results             |

### Parallelization Modes

Control how requests are dispatched with the `-m` flag:

* **`sequential`** — Send one request at a time, wait for each response before sending the next. Good for measuring single-request latency.
* **`fixed:N`** — Maintain exactly N concurrent in-flight requests (e.g., `fixed:5`). Good for simulating sustained load.
* **`rate:Nms`** — Fire a new request every N milliseconds regardless of pending responses (e.g., `rate:100`). Good for testing behavior under a fixed request rate.

### Examples

```bash
# Measure single-request latency (10 sequential requests)
cargo run -p fynd-benchmark --release -- load -n 10

# Simulate 10 concurrent users sending 100 total requests
cargo run -p fynd-benchmark --release -- load -m fixed:10 -n 100

# Fire a request every 50ms using custom request templates
cargo run -p fynd-benchmark --release -- load \
  -m rate:50 -n 100 \
  --requests-file tools/benchmark/requests_set.json

# Export results to JSON for further analysis
cargo run -p fynd-benchmark --release -- load \
  -m fixed:10 -n 1000 \
  --output-file results.json
```

### Output

The tool prints real-time progress, summary statistics (min, max, mean, median, p95, p99, stddev), and ASCII histograms of timing distributions. Pass `--output-file` to export the full results as JSON.

***

## Comparing Two Solvers

Sends identical quote requests to two running solver instances and compares output quality: amount out (in bps), gas estimates, and route selection.

```bash
cargo run -p fynd-benchmark --release -- compare [OPTIONS]
```

### Setup

You need two Fynd instances running simultaneously — typically from different git branches. Since both share the same binary target directory and metrics port, use **git worktrees** to avoid conflicts.

#### 1. Create a worktree for the baseline

```bash
# From the main repo
git worktree add ../fynd-baseline main
```

#### 2. Start solver A (baseline) in the worktree

```bash
cd ../fynd-baseline
RUST_LOG=info cargo run --release -- serve \
  --protocols uniswap_v2,uniswap_v3 \
  --http-port 3000 \
  --tycho-url <TYCHO_URL> \
  --tycho-api-key <API_KEY>
```

#### 3. Start solver B (your branch) in the original repo

```bash
cd /path/to/fynd
RUST_LOG=info cargo run --release -- serve \
  --protocols uniswap_v2,uniswap_v3 \
  --http-port 3001 \
  --tycho-url <TYCHO_URL> \
  --tycho-api-key <API_KEY>
```

#### 4. Wait for both solvers to be healthy

```bash
curl http://localhost:3000/v1/health
curl http://localhost:3001/v1/health
```

Both should return `{"healthy": true, ...}` before running the comparison.

#### 5. Run the comparison

```bash
cargo run -p fynd-benchmark --release -- compare \
  --url-a http://localhost:3000 \
  --url-b http://localhost:3001 \
  --label-a main \
  --label-b my-branch \
  -n 100
```

### Options

| Flag              | Default                   | Description                            |
| ----------------- | ------------------------- | -------------------------------------- |
| `--url-a`         | `http://localhost:3000`   | Solver A (baseline) URL                |
| `--url-b`         | `http://localhost:3001`   | Solver B (candidate) URL               |
| `--label-a`       | `main`                    | Label for solver A in output           |
| `--label-b`       | `branch`                  | Label for solver B in output           |
| `-n`              | `500`                     | Number of requests to send             |
| `--requests-file` | *(none)*                  | Path to JSON file with custom requests |
| `--output`        | `comparison_results.json` | Path for full results JSON             |
| `--timeout-ms`    | `15000`                   | Per-request timeout in milliseconds    |
| `--seed`          | `42`                      | Random seed for reproducibility        |

### Net-of-Gas Comparison

The compare tool uses the server-computed `amount_out_net_gas` field for net-of-gas output comparison. This value represents the output amount minus gas cost denominated in the output token, calculated by the solver. It works for all token pairs.

### Output

Prints a summary table to stdout showing win/loss counts and bps differences (both gross and net-of-gas). Writes detailed per-request results to the output JSON file. Positive bps diffs mean solver B returned more output than solver A.

***

## CPU Scaling

Measures how solver throughput (req/s) scales with worker thread count. The tool builds a solver in-process for each worker count, runs a load test, shuts down, and repeats.

```bash
cargo run -p fynd-benchmark --release -- scale [OPTIONS]
```

{% hint style="warning" %}
Requires a `worker_pools.toml` with exactly **one** pool defined.
{% endhint %}

### Options

| Flag                    | Default             | Description                                       |
| ----------------------- | ------------------- | ------------------------------------------------- |
| `--base-config`         | `worker_pools.toml` | Single-pool TOML config                           |
| `--worker-counts`       | *(required)*        | Comma-separated worker counts (e.g. `1,2,4,8,16`) |
| `--protocols`           | *(required)*        | Comma-separated protocols for solver              |
| `--tycho-url`           | `localhost:4242`    | Tycho WebSocket URL                               |
| `--tycho-api-key`       | *(none)*            | Tycho API key                                     |
| `--disable-tls`         | `false`             | Disable TLS for Tycho connection                  |
| `--rpc-url`             | *(none)*            | Node RPC URL                                      |
| `--chain`               | `ethereum`          | Chain name                                        |
| `--http-port`           | `3000`              | Solver HTTP port                                  |
| `-n`                    | `100`               | Requests per iteration                            |
| `-m`                    | `fixed:8`           | Parallelization mode                              |
| `--requests-file`       | *(none)*            | Custom request templates                          |
| `--warmup-secs`         | `30`                | Seconds to wait after health before benchmarking  |
| `--health-timeout-secs` | `300`               | Max seconds to wait for solver health             |
| `--output-file`         | *(none)*            | JSON output file                                  |

### Example

```bash
cargo run -p fynd-benchmark --release -- scale \
  --base-config single_pool.toml \
  --worker-counts 1,2,4,8,16 \
  --protocols uniswap_v2,uniswap_v3 \
  --tycho-url wss://tycho.example.com \
  --tycho-api-key $TYCHO_API_KEY \
  -n 200 \
  -m fixed:8 \
  --output-file scale_results.json
```

### Output

The tool prints a summary table showing throughput, latency, and per-worker efficiency at each worker count:

```
=== CPU Scaling Results ===

Pool: most_liquid_2_hops_fast (algorithm: most_liquid)
Requests per run: 200, Mode: fixed:8

 Workers | Throughput (req/s) | Median RT (ms) | P99 RT (ms) | RPS/Worker
---------+--------------------+----------------+-------------+-----------
       1 |               8.50 |             95 |          142 |      8.50
       2 |              16.20 |             50 |           98 |      8.10
       4 |              30.10 |             28 |           65 |      7.53
       8 |              48.90 |             18 |           52 |      6.11
      16 |              62.30 |             15 |           48 |      3.89
```

Pass `--output-file` to export the full results as JSON for further analysis.

***

## Request Data

By default, the load test uses a single WETH→USDC swap and the compare tool samples from a built-in set of 50 real aggregator trades. Both commands accept `--requests-file` to supply custom requests.

### Downloading More Trades

For broader coverage, download a larger set of real aggregator trades (10k):

```bash
cargo run -p fynd-benchmark --release -- download-trades
```

Then use it with either command via `--requests-file aggregator_trades_10k.json`.

### Custom Request Format

The file should be a JSON array of quote request bodies:

```json
[
  {
    "orders": [{
      "token_in": "0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2",
      "token_out": "0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48",
      "amount": "1000000000000000000000",
      "side": "sell",
      "sender": "0x0000000000000000000000000000000000000001"
    }]
  }
]
```

See `tools/benchmark/requests_set.json` in the repository for a complete example.


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.fynd.xyz/guides/benchmarking.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.