Benchmarking

Measure solver performance and compare output quality between branches.

Fynd ships with a benchmark tool (fynd-benchmark) for load-testing a solver and comparing output quality between two solver instances. Both features live in tools/benchmark/ and run against live solver instances.

Prerequisite: You need a running solver before using any benchmark command. See Quickstart for setup instructions.

Load Testing

Measures latency (round-trip, solve time, overhead) and throughput for a single solver instance.

cargo run -p fynd-benchmark --release -- load [OPTIONS]

Always build with --release. Debug builds produce misleading latency numbers.

Options

Flag

Default

Description

--solver-url

http://localhost:3000

Solver URL to benchmark against

-n

1

Number of requests to send

-m

sequential

Parallelization mode

--requests-file

(none)

Path to JSON file with request templates

--output-file

(none)

Output file for JSON results

Parallelization Modes

Control how requests are dispatched with the -m flag:

sequential — Send one request at a time, wait for each response before sending the next. Good for measuring single-request latency.
fixed:N — Maintain exactly N concurrent in-flight requests (e.g., fixed:5). Good for simulating sustained load.
rate:Nms — Fire a new request every N milliseconds regardless of pending responses (e.g., rate:100). Good for testing behavior under a fixed request rate.

Examples

# Measure single-request latency (10 sequential requests)
cargo run -p fynd-benchmark --release -- load -n 10

# Simulate 10 concurrent users sending 100 total requests
cargo run -p fynd-benchmark --release -- load -m fixed:10 -n 100

# Fire a request every 50ms using custom request templates
cargo run -p fynd-benchmark --release -- load \
  -m rate:50 -n 100 \
  --requests-file tools/benchmark/requests_set.json

# Export results to JSON for further analysis
cargo run -p fynd-benchmark --release -- load \
  -m fixed:10 -n 1000 \
  --output-file results.json

Output

The tool prints real-time progress, summary statistics (min, max, mean, median, p95, p99, stddev), and ASCII histograms of timing distributions. Pass --output-file to export the full results as JSON.

Comparing Two Solvers

Sends identical quote requests to two running solver instances and compares output quality: amount out (in bps), gas estimates, and route selection.

cargo run -p fynd-benchmark --release -- compare [OPTIONS]

Setup

You need two Fynd instances running simultaneously — typically from different git branches. Since both share the same binary target directory and metrics port, use git worktrees to avoid conflicts.

1. Create a worktree for the baseline

# From the main repo
git worktree add ../fynd-baseline main

2. Start solver A (baseline) in the worktree

cd ../fynd-baseline
RUST_LOG=info cargo run --release -- serve \
  --protocols uniswap_v2,uniswap_v3 \
  --http-port 3000 \
  --tycho-url <TYCHO_URL> \
  --tycho-api-key <API_KEY>

3. Start solver B (your branch) in the original repo

cd /path/to/fynd
RUST_LOG=info cargo run --release -- serve \
  --protocols uniswap_v2,uniswap_v3 \
  --http-port 3001 \
  --tycho-url <TYCHO_URL> \
  --tycho-api-key <API_KEY>

4. Wait for both solvers to be healthy

curl http://localhost:3000/v1/health
curl http://localhost:3001/v1/health

Both should return {"healthy": true, ...} before running the comparison.

5. Run the comparison

cargo run -p fynd-benchmark --release -- compare \
  --url-a http://localhost:3000 \
  --url-b http://localhost:3001 \
  --label-a main \
  --label-b my-branch \
  -n 100

Options

Flag

Default

Description

--url-a

http://localhost:3000

Solver A (baseline) URL

--url-b

http://localhost:3001

Solver B (candidate) URL

--label-a

main

Label for solver A in output

--label-b

branch

Label for solver B in output

-n

100

Number of requests to send

--requests-file

(none)

Path to JSON file with custom requests

--output

comparison_results.json

Path for full results JSON

--timeout-ms

15000

Per-request timeout in milliseconds

--seed

42

Random seed for reproducibility

Output

Prints a summary table to stdout showing win/loss counts and bps differences. Writes detailed per-request results to the output JSON file. Positive bps diffs mean solver B returned more output than solver A.

CPU Scaling

Measures how solver throughput (req/s) scales with worker thread count. The tool builds a solver in-process for each worker count, runs a load test, shuts down, and repeats.

cargo run -p fynd-benchmark --release -- scale [OPTIONS]

Requires a worker_pools.toml with exactly one pool defined.

Options

Flag

Default

Description

--base-config

worker_pools.toml

Single-pool TOML config

--worker-counts

(required)

Comma-separated worker counts (e.g. 1,2,4,8,16)

--protocols

(required)

Comma-separated protocols for solver

--tycho-url

localhost:4242

Tycho WebSocket URL

--tycho-api-key

(none)

Tycho API key

--disable-tls

false

Disable TLS for Tycho connection

--rpc-url

(none)

Node RPC URL

--chain

ethereum

Chain name

--http-port

3000

Solver HTTP port

-n

100

Requests per iteration

-m

fixed:8

Parallelization mode

--requests-file

(none)

Custom request templates

--warmup-secs

30

Seconds to wait after health before benchmarking

--health-timeout-secs

300

Max seconds to wait for solver health

--output-file

(none)

JSON output file

Example

cargo run -p fynd-benchmark --release -- scale \
  --base-config single_pool.toml \
  --worker-counts 1,2,4,8,16 \
  --protocols uniswap_v2,uniswap_v3 \
  --tycho-url wss://tycho.example.com \
  --tycho-api-key $TYCHO_API_KEY \
  -n 200 \
  -m fixed:8 \
  --output-file scale_results.json

Output

The tool prints a summary table showing throughput, latency, and per-worker efficiency at each worker count:

=== CPU Scaling Results ===

Pool: most_liquid_2_hops_fast (algorithm: most_liquid)
Requests per run: 200, Mode: fixed:8

 Workers | Throughput (req/s) | Median RT (ms) | P99 RT (ms) | RPS/Worker
---------+--------------------+----------------+-------------+-----------
       1 |               8.50 |             95 |          142 |      8.50
       2 |              16.20 |             50 |           98 |      8.10
       4 |              30.10 |             28 |           65 |      7.53
       8 |              48.90 |             18 |           52 |      6.11
      16 |              62.30 |             15 |           48 |      3.89

Pass --output-file to export the full results as JSON for further analysis.

Custom Request Templates

By default, the load test uses a single WETH→USDC swap and the compare tool generates random requests from a built-in set of token pairs. Both commands accept --requests-file to supply custom requests.

The file should be a JSON array of quote request bodies:

[
  {
    "orders": [{
      "token_in": "0xC02aaA39b223FE8D0A0e5C4F27eAD9083C756Cc2",
      "token_out": "0xA0b86991c6218b36c1d19D4a2e9Eb0cE3606eB48",
      "amount": "1000000000000000000000",
      "side": "sell",
      "sender": "0x0000000000000000000000000000000000000001"
    }]
  }
]

See tools/benchmark/requests_set.json in the repository for a complete example.

PreviousExecuting the solutions NextBenchmark Results

Last updated 1 day ago

hashtagLoad Testing

hashtagOptions

hashtagParallelization Modes

hashtagExamples

hashtagOutput

hashtagComparing Two Solvers

hashtagSetup

hashtag1. Create a worktree for the baseline

hashtag2. Start solver A (baseline) in the worktree

hashtag3. Start solver B (your branch) in the original repo

hashtag4. Wait for both solvers to be healthy

hashtag5. Run the comparison

hashtagOptions

hashtagOutput

hashtagCPU Scaling

hashtagOptions

hashtagExample

hashtagOutput

hashtagCustom Request Templates

Load Testing

Options

Parallelization Modes

Examples

Output

Comparing Two Solvers

Setup

1. Create a worktree for the baseline

2. Start solver A (baseline) in the worktree

3. Start solver B (your branch) in the original repo

4. Wait for both solvers to be healthy

5. Run the comparison

Options

Output

CPU Scaling

Options

Example

Output

Custom Request Templates