Benchmarking

Measure solver performance and compare output quality between branches.

Fynd ships with a benchmark tool (fynd-benchmark) for load-testing a solver and comparing output quality between two solver instances. Both features live in tools/benchmark/ and run against live solver instances.

Prerequisite: You need a running solver before using any benchmark command. See Quickstart for setup instructions.

Load Testing

Measures latency (round-trip, solve time, overhead) and throughput for a single solver instance.

cargo run -p fynd-benchmark --release -- load [OPTIONS]

Always build with --release. Debug builds produce misleading latency numbers.

Options

Flag	Default	Description
--solver-url	http://localhost:3000	Solver URL to benchmark against
-n	1	Number of requests to send
-m	sequential	Parallelization mode
--requests-file	(none)	Path to JSON file with request templates
--output-file	(none)	Output file for JSON results

Parallelization Modes

Control how requests are dispatched with the -m flag:

• sequential — Send one request at a time, wait for each response before sending the next. Good for measuring single-request latency.

• fixed:N — Maintain exactly N concurrent in-flight requests (e.g., fixed:5). Good for simulating sustained load.

• rate:Nms — Fire a new request every N milliseconds regardless of pending responses (e.g., rate:100). Good for testing behavior under a fixed request rate.

Examples

# Measure single-request latency (10 sequential requests)
cargo run -p fynd-benchmark --release -- load -n 10

# Simulate 10 concurrent users sending 100 total requests
cargo run -p fynd-benchmark --release -- load -m fixed:10 -n 100

# Fire a request every 50ms using custom request templates
cargo run -p fynd-benchmark --release -- load \
  -m rate:50 -n 100 \
  --requests-file tools/benchmark/requests_set.json

# Export results to JSON for further analysis
cargo run -p fynd-benchmark --release -- load \
  -m fixed:10 -n 1000 \
  --output-file results.json

Output

The tool prints real-time progress, summary statistics (min, max, mean, median, p95, p99, stddev), and ASCII histograms of timing distributions. Pass --output-file to export the full results as JSON.

---

Comparing Two Solvers

Sends identical quote requests to two running solver instances and compares output quality: amount out (in bps), gas estimates, and route selection.

cargo run -p fynd-benchmark --release -- compare [OPTIONS]

Setup

You need two Fynd instances running simultaneously — typically from different git branches. Since both share the same binary target directory and metrics port, use git worktrees to avoid conflicts.

1. Create a worktree for the baseline

# From the main repo
git worktree add ../fynd-baseline main

2. Start solver A (baseline) in the worktree

cd ../fynd-baseline
RUST_LOG=info cargo run --release -- serve \
  --protocols uniswap_v2,uniswap_v3 \
  --http-port 3000 \
  --tycho-url <TYCHO_URL> \
  --tycho-api-key <API_KEY>

3. Start solver B (your branch) in the original repo

cd /path/to/fynd
RUST_LOG=info cargo run --release -- serve \
  --protocols uniswap_v2,uniswap_v3 \
  --http-port 3001 \
  --tycho-url <TYCHO_URL> \
  --tycho-api-key <API_KEY>

4. Wait for both solvers to be healthy

curl http://localhost:3000/v1/health
curl http://localhost:3001/v1/health

Both should return {"healthy": true, ...} before running the comparison.

5. Run the comparison

cargo run -p fynd-benchmark --release -- compare \
  --url-a http://localhost:3000 \
  --url-b http://localhost:3001 \
  --label-a main \
  --label-b my-branch \
  -n 100

Options

Flag	Default	Description
--url-a	http://localhost:3000	Solver A (baseline) URL
--url-b	http://localhost:3001	Solver B (candidate) URL
--label-a	main	Label for solver A in output
--label-b	branch	Label for solver B in output
-n	500	Number of requests to send
--requests-file	(none)	Path to JSON file with custom requests
--output	comparison_results.json	Path for full results JSON
--timeout-ms	15000	Per-request timeout in milliseconds
--seed	42	Random seed for reproducibility

Net-of-Gas Comparison

The compare tool uses the server-computed amount_out_net_gas field for net-of-gas output comparison. This value represents the output amount minus gas cost denominated in the output token, calculated by the solver. It works for all token pairs.

Output

Prints a summary table to stdout showing win/loss counts and bps differences (both gross and net-of-gas). Writes detailed per-request results to the output JSON file. Positive bps diffs mean solver B returned more output than solver A.

---

CPU Scaling

Measures how solver throughput (req/s) scales with worker thread count. The tool builds a solver in-process for each worker count, runs a load test, shuts down, and repeats.

cargo run -p fynd-benchmark --release -- scale [OPTIONS]

Requires a worker_pools.toml with exactly one pool defined.

Options

Flag	Default	Description
--base-config	worker_pools.toml	Single-pool TOML config
--worker-counts	(required)	Comma-separated worker counts (e.g. 1,2,4,8,16)
--protocols	(required)	Comma-separated protocols for solver
--tycho-url	localhost:4242	Tycho WebSocket URL
--tycho-api-key	(none)	Tycho API key
--disable-tls	false	Disable TLS for Tycho connection
--rpc-url	(none)	Node RPC URL
--chain	ethereum	Chain name
--http-port	3000	Solver HTTP port
-n	100	Requests per iteration
-m	fixed:8	Parallelization mode
--requests-file	(none)	Custom request templates
--warmup-secs	30	Seconds to wait after health before benchmarking
--health-timeout-secs	300	Max seconds to wait for solver health
--output-file	(none)	JSON output file

Example

cargo run -p fynd-benchmark --release -- scale \
  --base-config single_pool.toml \
  --worker-counts 1,2,4,8,16 \
  --protocols uniswap_v2,uniswap_v3 \
  --tycho-url wss://tycho.example.com \
  --tycho-api-key $TYCHO_API_KEY \
  -n 200 \
  -m fixed:8 \
  --output-file scale_results.json

Output

The tool prints a summary table showing throughput, latency, and per-worker efficiency at each worker count:

=== CPU Scaling Results ===

Pool: most_liquid_2_hops_fast (algorithm: most_liquid)
Requests per run: 200, Mode: fixed:8

 Workers | Throughput (req/s) | Median RT (ms) | P99 RT (ms) | RPS/Worker
---------+--------------------+----------------+-------------+-----------
       1 |               8.50 |             95 |          142 |      8.50
       2 |              16.20 |             50 |           98 |      8.10
       4 |              30.10 |             28 |           65 |      7.53
       8 |              48.90 |             18 |           52 |      6.11
      16 |              62.30 |             15 |           48 |      3.89

Pass --output-file to export the full results as JSON for further analysis.

---

Request Data

By default, the load test uses a single WETH→USDC swap and the compare tool samples from a built-in set of 50 real aggregator trades. Both commands accept --requests-file to supply custom requests.

Downloading More Trades

For broader coverage, download a larger set of real aggregator trades (10k):

cargo run -p fynd-benchmark --release -- download-trades

Then use it with either command via --requests-file aggregator_trades_10k.json.

Custom Request Format

The file should be a JSON array of quote request bodies:

[
  {
    "orders": [{
      "token_in": "0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2",
      "token_out": "0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48",
      "amount": "1000000000000000000000",
      "side": "sell",
      "sender": "0x0000000000000000000000000000000000000001"
    }]
  }
]

See tools/benchmark/requests_set.json in the repository for a complete example.