docs: reth design decisions

Co-authored-by: Georgios Konstantopoulos <[email protected]>

docs/design/README.md

@@ -1,5 +1,12 @@
 ## Design
 
+Docs under this page contain some context on how we've iterated on the Reth design (still WIP, please contribute!):
+
+- [Database](./database.md)
+- Networking
+    - [P2P](./p2p.md)
+    - [Headers Downloaders](./headers-downloaders.md)
+
 ### Observability
 
-- [Metrics](./metrics.md): Guidelines on metrics and traces.
+- [Metrics](./metrics.md): Guidelines on metrics and traces.

docs/design/database.md

@@ -0,0 +1,22 @@
+# Database
+
+## Abstractions
+
+* We created a [Database trait abstraction](https://github.com/foundry-rs/reth/blob/0d9b9a392d4196793736522f3fc2ac804991b45d/crates/interfaces/src/db/mod.rs) using Rust Stable GATs which frees us from being bound to a single database implementation. We currently use MDBX, but are exploring [redb](https://github.com/cberner/redb) as an alternative.
+* We then iterated on [`StageDB`](https://github.com/foundry-rs/reth/blob/0d9b9a392d4196793736522f3fc2ac804991b45d/crates/stages/src/db.rs#L14-L19) as a non-leaky abstraction with helpers for strictly-typed and unit-tested higher-level database abstractions.
+
+## Codecs
+
+* We want Reth's serialized format to be able to trade off read/write speed for size, depending on who the user is.
+* To achieve that, we created the [Encode/Decode/Compress/Decompress trais](https://github.com/foundry-rs/reth/blob/0d9b9a392d4196793736522f3fc2ac804991b45d/crates/interfaces/src/db/table.rs#L9-L36) to make the (de)serialization of database `Table::Key` and `Table::Values` generic.
+    * This allows for [out-of-the-box benchmarking](https://github.com/foundry-rs/reth/blob/0d9b9a392d4196793736522f3fc2ac804991b45d/crates/db/benches/encoding_iai.rs#L5) (using [Criterion](https://github.com/bheisler/criterion.rs) and [Iai](https://github.com/bheisler/iai))
+    * It also enables [out-of-the-box fuzzing](https://github.com/foundry-rs/reth/blob/0d9b9a392d4196793736522f3fc2ac804991b45d/crates/interfaces/src/db/codecs/fuzz/mod.rs) using [trailofbits/test-fuzz](https://github.com/trailofbits/test-fuzz).
+* We implemented that trait for the following encoding formats:
+    * [Ethereum-specific Compact Encoding](https://github.com/foundry-rs/reth/blob/0d9b9a392d4196793736522f3fc2ac804991b45d/crates/codecs/derive/src/compact/mod.rs): A lot of Ethereum datatypes have unnecessary zeros when serialized, or optional (e.g. on empty hashes) which would be nice not to pay in storage costs. 
+        * [Erigon](https://github.com/ledgerwatch/erigon/blob/12ee33a492f5d240458822d052820d9998653a63/docs/programmers_guide/db_walkthrough.MD) achieves that by having a `bitfield` set on  Table "PlainState which adds a bitfield to Accounts.
+        * Akula expanded it for other tables and datatypes manually. It also saved some more space by storing the length of certain types (U256, u64) using the modular_bitfield crate, which compacts this information.
+        * We generalized it for all types, by writing a derive macro that autogenerates code for implementing the trait. It, also generates the interfaces required for fuzzing using ToB/test-fuzz:
+    * [Scale Encoding](https://github.com/paritytech/parity-scale-codec)
+    * [Postcard Encoding](https://github.com/jamesmunns/postcard)
+    * Passthrough (called `no_codec` in the codebase)
+* We made implementation of these traits easy via a derive macro called [`main_codec`](https://github.com/foundry-rs/reth/blob/0d9b9a392d4196793736522f3fc2ac804991b45d/crates/codecs/derive/src/lib.rs#L15) that delegates to one of Compact (default), Scale, Postcard or Passthrough encoding. This is [derived on every struct we need](https://github.com/search?q=repo%3Afoundry-rs%2Freth%20%22%23%5Bmain_codec%5D%22&type=code), and lets us experiment with different encoding formats without having to modify the entire codebase each time.

docs/design/headers-downloader.md

@@ -0,0 +1,11 @@
+# Headers Downloader And Stage
+
+> Explanation of how the Reth Headers Downloader and Stage works
+
+* We started off with sketching out a generic header downloader interface, so that we can support multiple downloading strategies implementing the interface. See [`reth#58`](https://github.com/foundry-rs/reth/pull/58) and [`reth#118`](https://github.com/foundry-rs/reth/pull/118).
+* First, we implemented the reverse linear download. It received the current chain tip and local head as arguments and requested blocks in batches starting from the tip, and retried on request failure. See [`reth#58`](https://github.com/foundry-rs/reth/pull/58) and [`reth#119`](https://github.com/foundry-rs/reth/pull/119).
+* The first complete implementation of the headers stage was introduced in [`reth#126`](https://github.com/foundry-rs/reth/pull/126). The stage looked up the local head & queried the consensus for the chain tip and queried the downloader passing them as arguments. After the download finished, the stage would proceed to insert headers in the ascending order by appending the entries to the corresponding tables.
+* The original downloader was refactored in [`reth#249`](https://github.com/foundry-rs/reth/pull/249) to return a `Future` which would resolve when either the download is completed or the error occurred during polling. This future kept a pointer to a current request at any time, allowing to retry the request in case of failure. The insert logic of the headers stage remained unchanged.
+    * NOTE: Up to this point the headers stage awaited full range of blocks (from local head to tip) to be downloaded before proceeding to insert.
+* [`reth#296`](https://github.com/foundry-rs/reth/pull/296) introduced the `Stream` implementation of the download as well as the commit threshold for the headers stage. The `Stream` implementation yields headers as soon as they are received and validated. It dispatches the request for the next header batch until the head is reached. The headers stage now has a configurable commit threshold which allows configure the insert batch size. With this change, the headers stage no longer waits for the download to be complete, but rather collects the headers from the stream up to the commit threshold parameter. After collecting, the stage proceeds to insert the batch. The process is repeated until the stream is drained. At this point, we populated all tables except for HeadersTD since it has to be computed in a linear ascending order. The stage starts walking the populated headers table and computes & inserts new total difficulty values.
+* This header implementation is unique because it is implemented as a Stream, it yields headers as soon as they become available (contrary to waiting for download to complete) and it keeps only one header in buffer (required to form the next header request) .

docs/design/metrics.md

@@ -58,4 +58,4 @@ How the metrics are exposed to the end-user is determined by the CLI.
 [metrics.KeyName]: https://docs.rs/metrics/latest/metrics/struct.KeyName.html
 [metrics.Label]: https://docs.rs/metrics/latest/metrics/struct.Label.html
 [prom_base_units]: https://prometheus.io/docs/practices/naming/#base-units
-[metrics_util.PrefixLayer]: https://docs.rs/metrics-util/latest/metrics_util/layers/struct.PrefixLayer.html
+[metrics_util.PrefixLayer]: https://docs.rs/metrics-util/latest/metrics_util/layers/struct.PrefixLayer.html

docs/design/p2p.md

@@ -0,0 +1,7 @@
+# P2P
+
+> Explanation of the [Reth P2P Stack](../../crates/net/p2p) design process
+
+* Our initial design exploration started in [#64](https://github.com/foundry-rs/reth/issues/64), which focused on layering dependent subprotocols as generic async streams, then using those streams to construct higher level network abstractions.
+* Following the above design, we then implemented `P2PStream` and `EthStream`, corresponding to the `p2p` and `eth` subprotocol respectively.
+* The wire protocol used to decode messages in `EthStream` came from ethp2p, making it easy to get the full stack to work.