Skip to content

Commit

Permalink
Pruning object tombstones (MystenLabs#14373)
Browse files Browse the repository at this point in the history
## Description 

This PR implements the main logic of pruning sui object tombstones.

Given the fact that we are using range deletes and
`ignore_range_deletions` read option to improve read performance, it
makes tombstone deletion tricky because once a tombstone is deleted, we
need to make sure that all prior versions of the object must also be
deleted. Otherwise, it'll leak object. To not hurt performance much
(pending performance evaluation), we decided to use
scan-and-point-delete when encounter an object tombstone. In this case,
we guarantee that all prior versions are not visible to readers after
the deletion.

The tombstone pruning logic currently is only tested in unit test. I
wanted to enable it in integration test, but the PR becomes messy. It
will come up in a follow up PR.

## Test Plan 

Unit test: this PR
Integration test: follow up PR
cluster test for performance testing

---
If your changes are not user-facing and not a breaking change, you can
skip the following section. Otherwise, please indicate what changed, and
then add to the Release Notes section as highlighted during the release
process.

### Type of Change (Check all that apply)

- [ ] protocol change
- [ ] user-visible impact
- [ ] breaking change for a client SDKs
- [ ] breaking change for FNs (FN binary must upgrade)
- [ ] breaking change for validators or node operators (must upgrade
binaries)
- [ ] breaking change for on-chain data layout
- [ ] necessitate either a data wipe or data migration

### Release notes
  • Loading branch information
halfprice authored Nov 4, 2023
1 parent 2c832ad commit e973547
Show file tree
Hide file tree
Showing 10 changed files with 201 additions and 39 deletions.
1 change: 1 addition & 0 deletions crates/sui-config/data/fullnode-template-with-path.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@ authority-store-pruning-config:
max-checkpoints-in-batch: 10
max-transactions-in-batch: 1000
pruning-run-delay-seconds: 60
enable-pruning-tombstones: false

protocol-key-pair:
path: "protocol.key"
Expand Down
1 change: 1 addition & 0 deletions crates/sui-config/data/fullnode-template.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,4 @@ authority-store-pruning-config:
max-checkpoints-in-batch: 10
max-transactions-in-batch: 1000
pruning-run-delay-seconds: 60
enable-pruning-tombstones: false
8 changes: 8 additions & 0 deletions crates/sui-config/src/node.rs
Original file line number Diff line number Diff line change
Expand Up @@ -543,6 +543,7 @@ pub struct AuthorityStorePruningConfig {
/// number of epochs to keep the latest version of transactions and effects for
#[serde(skip_serializing_if = "Option::is_none")]
pub num_epochs_to_retain_for_checkpoints: Option<u64>,
pub enable_pruning_tombstones: bool,
}

impl Default for AuthorityStorePruningConfig {
Expand All @@ -559,6 +560,7 @@ impl Default for AuthorityStorePruningConfig {
max_transactions_in_batch: 1000,
periodic_compaction_threshold_days: None,
num_epochs_to_retain_for_checkpoints: None,
enable_pruning_tombstones: false,
}
}
}
Expand All @@ -578,6 +580,7 @@ impl AuthorityStorePruningConfig {
max_transactions_in_batch: 1000,
periodic_compaction_threshold_days: None,
num_epochs_to_retain_for_checkpoints,
enable_pruning_tombstones: false,
}
}
pub fn fullnode_config() -> Self {
Expand All @@ -594,6 +597,7 @@ impl AuthorityStorePruningConfig {
max_transactions_in_batch: 1000,
periodic_compaction_threshold_days: None,
num_epochs_to_retain_for_checkpoints,
enable_pruning_tombstones: false,
}
}

Expand All @@ -613,6 +617,10 @@ impl AuthorityStorePruningConfig {
}
})
}

pub fn set_enable_pruning_tombstones(&mut self, enable_pruning_tombstones: bool) {
self.enable_pruning_tombstones = enable_pruning_tombstones;
}
}

#[derive(Debug, Clone, Deserialize, Serialize)]
Expand Down
Loading

0 comments on commit e973547

Please sign in to comment.