Benchmarking reth live sync with reth-bench

The binary contained in this directory, reth-bench, is a tool that can be used to benchmark the performance of the reth live sync. reth-bench is a general tool, and can be used for benchmarking node performance, as long as the node supports the engine API.

A recap on node synchronization

Reth uses two primary methods for synchronizing the chain:

• Historical sync, which is used to synchronize the chain from genesis to a known finalized block. This involves re-executing the entire chain history.
• Live sync, which is used to synchronize the chain from a finalized block to the current head. This involves processing new blocks as they are produced.

Benchmarking historical sync for reth is fairly easy, because historical sync is a long-running, deterministic process. Reth specifically contains the --debug.tip argument, which allows for running the historical sync pipeline to a specific block. However, reth's historical sync applies optimizations that are not always possible when syncing new blocks.

Live sync, on the other hand, is a more complex process that is harder to benchmark. It is also more sensitive to network conditions of the CL. In order to benchmark live sync, we need to simulate a CL in a controlled manner, so reth can use the same code paths it would when syncing new blocks.

The reth-bench tool

The reth-bench tool is designed to benchmark performance of reth live sync. It can also be used for debugging client spec implementations, as it replays historical blocks by mocking a CL client. Performance is measured by latency and gas used in a block, as well as the computed gas used per second. As long as the data is representative of real-world load, or closer to worst-case load test, the gas per second gives a rough sense of how much throughput the node would be able to handle.

Prerequisites

If you will be collecting CPU profiles, make sure reth is compiled with the profiling profile. Otherwise, running make maxperf at the root of the repo should be sufficient for collecting accurate performance metrics.

Command Usage

reth-bench contains different commands to benchmark different patterns of engine API calls. The reth-bench new-payload-fcu command is the most representative of ethereum mainnet live sync, alternating between sending engine_newPayload calls and engine_forkchoiceUpdated calls.

Below is an overview of how to execute a benchmark:

1. Setup: Make sure reth is running in the background with the proper configuration. This setup involves ensuring the node is at the correct state, setting up profiling tools, and possibly more depending on the purpose of the benchmark's.

2. Run the Benchmark:

   reth-bench new-payload-fcu --rpc-url http://<rpc-url>:8545 --from <start_block> --to <end_block> --jwtsecret <jwt_file_path>

   Replace <rpc-url>, <start_block>, <end_block>, and <jwt_file_path> with the appropriate values for your testing environment. Note that this assumes that the benchmark node's engine API is running on http://127.0.0.1:8545, which is set as a default value in reth-bench. To configure this value, use the --engine-rpc-url flag.

3. Observe Outputs: Upon running the command, reth-bench will output benchmark results, showing processing speeds and gas usage, which are crucial for analyzing the node's performance.

   Example output:

   2024-05-30T00:45:20.806691Z  INFO Running benchmark using data from RPC URL: http://<rpc-url>:8545
   // ... logs per block
   2024-05-30T00:45:34.203172Z  INFO Total Ggas/s: 0.15 total_duration=5.085704882s total_gas_used=741620668.0
   

4. Stop and Review: Once the benchmark completes, terminate the reth process and review the logs and performance metrics collected, if any.

5. Repeat.

Additional Considerations

• RPC Configuration: The RPC endpoints should be accessible and configured correctly, specifically the RPC endpoint must support eth_getBlockByNumber and support fetching full transactions. The benchmark will make one RPC query per block as fast as possible, so ensure the RPC endpoint does not rate limit or block requests after a certain volume.
• Reproducibility: Ensure that the node is at the same state before attempting to retry a benchmark. The new-payload-fcu command specifically will commit to the database, so the node must be rolled back using reth stage unwind to reproducibly retry benchmarks.
• Profiling tools: If you are collecting CPU profiles, tools like samply and perf can be useful for analyzing node performance.
• Benchmark Data: reth-bench additionally contains a --benchmark.output flag, which will output gas used benchmarks across the benchmark range in CSV format. This may be useful for further data analysis.
• Platform Information: To ensure accurate and reproducible benchmarking, document the platform details, including hardware specifications, OS version, and any other relevant information before publishing any benchmarks.