Network restart followup (massalabs#3783) - Final state on disk

* Optim: If we need to interpolate a lot of cycles, only build the last ones

* TEST RocksDB checkpoints

* Added logs for ledger backups and recovery (check hash)

* Moved the LedgerBatch to final_state, step 1

This does not build, I have to add the logic of apply_change to the given ledger_batch

* Pass around a handle to the RocksDB DB instead of creating multiple ones

* Resolve deadlock in bootstrap test

* Changed AyncPool methods, bootstrap fails

* Fixed bug in bootstrap test.

* Resolved serializing issue with AsyncMessage in DB

* fixed doc test

* Big commit, see details inside desc

- Created massa-db package for easier access to rocks_db in final_state
- Put the AsyncPool in rocks_db
- Re-designed the speculative async_pool to avoid querying rocks_db too much

/!\ Tests fail! Need to investigate
But now the execution tests are not in a deadlock state.

* Added mini cache to async_pool, and used SetUpdateOrDelete

* Update mini cache + doctest passes

* Fix after merge

* Boilerplate for new rocks_db in final_state

* RocksDB for executed_ops and denunciations

* Fmt + Clippy fix

* Updated Pool message iterating

* Handle the async_pool iteration differently

Iteration per message > Iteration per key

* Fixed after merge

* Cleaned some things + docs

* Recompute all caches on init of final_state

* Additionnal cleaning (e.g. unused function)

* Added unit test for rocksdb final_state

* Add bool to reset the final_state on init

* Fix test

* Added back the test_take_batch for async_pool

* fmt fix

* Fix merge

* Big commit, see description

Put Final_state in a single column family.

TODO: Handle bootstrap as a single step and prefixes (e.g. LedgerPrefix !)

* clippy + fmt

* Unified bootstrap of final_state

* Put pos_state in rocks_db

* Fixed prefix_iteration problem

* another pass at iterator problems

* Added last_slot_before_downtime in final_state

* Fixed iteration error in pool cache reconstruction

* Fixed deferred_credits and production_stats behaviour

Still bootstrap test ignore

* fmt after merge

* Cleaned before working on bootstrap changes

* Wrapped the rocksDB structure

Needs several edits to existing code (e.g. bootstrap, write_batch)

* Almost everything up to date

Boostrap test still fails, have to debug with more logs!

* Fixed some of the bootstrap bugs

/!\ Some Deadlocks are a problem
/!\ Several TODOs in the code (commented ou asserts...)

* Started implementing Monotree

* Spring cleaning (DBBatch type)

* Fix cargo.lock after merge

* Fixed deadlock in pos_state

* clippy fixes

* Fixed pos_state change fail in bootstrap test

Also removed println!

* Fixed monotree

But Hash computations do not sync right on write_batch

* Added checks to bootstraped db (is_db_valid())

* Fix after merge w/ testnet23

* changed monotree dep

* Implemented XOR, test passes with it

* Removed old todo comment

* Added network restart documentation in massa-final-state crate lib.rs

* tmp

Signed-off-by: Litchi Pi <[email protected]>

* fixup

Signed-off-by: Litchi Pi <[email protected]>

* assert it's not the initial bytes

Signed-off-by: Litchi Pi <[email protected]>

* Removed create_initial_cycle from new_derived_from_snapshot

* Various fixes

- Fix cargo.lock after massa-sc-runtime update
- Don't create_initial_cycle if restarting from snapshot
- Cargo fixes

* fmt fix

* Cleaned error handling

* Cleaned unused tests, added comments

* Fixed consensus mock in bootstrap test

* Fixed after merge

* Cleanup warnings + monotree, clear the current_hashmap in db

* Fix bootstrap_test after last commit

* Additional cleaning following self review

Still need to review:
- Bootstrap server
- Final_state and massa_db
- Read and writes to the DB in all sub-structures of final_state

* Added a TODO comment

* Updated ledger_db to use MassaDB helper functions directly

* Added comments

* Fix tests after merge

* Finished self review, added TODO_PR comments for everything to fix before merge.

* Fixed bad behaviour of bootstrap streaming of stream_batch

* Added doc_comments for massa_db

Also only put the cur_change_id in the db to avoid desync

* Fixed bad final_state_test

* Import lsmtree instead of new

* removed final_state_hash

* Removed slot in final_state

* Removed initial_ledger_hash

* recompute caches no longer duplicated.

* Changed pos apply change documentation

* Cleaning of un-necessary recomputes

* Fix slot in bootstrap

* Fixed weird behaviour for bootstrap

* Try fix bootstrap

* Fix in deadlock in downtime interpolation

* fmt fix

* Fix: only backup on a slot with thread == 0

* Fix slots logic and interpolation

* fmt fix

* fix tests after slot behaviour change in final_state

* Removed println in final_state is_db_valid

* Updated bootstrap deserializer args for state_length

* Update op_exec_status on pruned executed_ops

* Updated pos_state test, remove roll_counts at 0

* Fix after merge

* typo fix

* removed create_snapshot (unused) feature

* revert node log level (from 3 to 2)

* Use format!() to construct backup directory

* Clean batch allocation in massa_db

* Used assert_eq!() instead of panic!() in finalize

* Use unreachable!() instead of todo!() in massa-hash

* Added debug_assert_eq!() to check Hash size and LsmTree output_size match

* Commit review suggestion

Co-authored-by: Litchi Pi (Tim) <[email protected]>

* Cargo fmt fix

* Updated state_step inconsistant error messages

* Updated XOR hash comment

* Add mip_store in final_state (massalabs#3992)

* Add mip_store in final_state

* Add reset_db in MIP store

* Fix bootstrap test - added the case of empty Versioning bootstrap

We have to rethink of how the cursors work, see follow up issue massalabs#4000

* Rework shutdown start & shutdown end computation for final state check

* Rework delete_prefix

* Rework extend_from_db for MipStore

---------

Co-authored-by: sydhds <[email protected]>
Co-authored-by: Leo-Besancon <[email protected]>

* Fix after merge

* Fix after merge

* Improve error message if update_batches call fail

* Fix lsmtree version in massa_db

* Update MAX_BOOTSTRAPPED_NEW_ELEMENTS to 500

---------

Signed-off-by: Litchi Pi <[email protected]>
Co-authored-by: Litchi Pi <[email protected]>
Co-authored-by: Sydhds <[email protected]>
Co-authored-by: sydhds <[email protected]>

Cargo.toml

@@ -8,6 +8,7 @@ members = [
   "massa-cipher",
   "massa-consensus-exports",
   "massa-consensus-worker",
+  "massa-db",
   "massa-executed-ops",
   "massa-execution-exports",
   "massa-execution-worker",
@@ -53,5 +54,4 @@ opt-level = 3 # Speed-up the CI
 # Do not add as a regular dependency. Never.
 #
 # * sandbox: for testing purpose, genesis timestamps is set as now + 9 seconds.
-# * create_snapshot: for saving the final_state on disk, alongside the ledger.
 # The saved snapshot can then be used to restart the network from the snapshot.

massa-async-pool/Cargo.toml

@@ -9,15 +9,22 @@ nom = "=7.1"
 num = "0.4"
 serde = { version = "1.0", features = ["derive"] }
 rand = "0.8"
+rocksdb = "0.20"
+parking_lot = { version = "0.12", features = ["deadlock_detection"] }
+
 # custom modules
 massa_hash = { path = "../massa-hash" }
 massa_ledger_exports = { path = "../massa-ledger-exports" }
 massa_models = { path = "../massa-models" }
 massa_serialization = { path = "../massa-serialization" }
 massa_signature = { path = "../massa-signature" }
+massa_db = { path = "../massa-db" }
 massa_time = { path = "../massa-time" }
 massa_proto = { path = "../massa-proto" }
 
+[dev-dependencies]
+tempfile = "3.3"
+massa_ledger_worker = { path = "../massa-ledger-worker" }
 
 # for more information on what are the following features used for, see the cargo.toml at workspace level
 [features]