[Bug] About vacuum interface.

[triplesheep]

About vacuum(time) -> deleteBeforeTimestamp, it will select files that will be deleted as following (let's ignore directory this case).
I suppose it works like:
All files - files: (files.modifyTime > deleteBeforeTimestamp)
- files: (currentSnapshot.addFiles)
- files: (all_removeFiles.deleteTime > deleteBeforeTimestamp)
However it seems works like:
All files - files: (files.modifyTime > deleteBeforeTimestamp)
- files: (currentSnapshot.addFiles)
- files: (currentSnapshot.removeFile.deleteTime > deleteBeforeTimestamp)
The difference between all_removeFiles.deleteTime and currentSnapshot.removeFile.deleteTime is that currentSnapshot.removeFile will filter out those deleteTime < minFileRetentionTimestamp(computed by TOMBSTONE_RETENTION). So it may delete some RemoveFile with removeFiles.deleteTime > deleteBeforeTimestamp but removeFiles.deleteTime < minFileRetentionTimestamp when set vacuum time param > TOMBSTONE_RETENTION.

For example:
Vacuum time param: 14 days
TOMBSTONE_RETENTION: 7 days
Vacuum will delete files with RemoveFile.deleteTime earlier than 7 days rather than earlier 14 days.

I want to confirm if it works lining with expectations

[rahulsmahadev]

Hi @triplesheep

The docs for Vacuum can be found here
https://docs.delta.io/latest/delta-utility.html

When vacuum is called with time parameter it will override the default retention period. If the time in not provided it will take the default retention period.

So for your example, it would remove files older than 14 days.

[triplesheep]

Hi @rahulsmahadev . I know vacuum will override the default retention period and in my case I supposed to remove files older than 14 days. However, after reading the code I found that it may delete remove files older than 7 days. For current snapshot, we use modify time of AddFileAction to filter and use delete time of RemoveFileAction to filter. Because when snapshot constructing stateReconstruction, it will reply the commit log and form the Acition collection. During that constructing, it will filter out those RemoveFileAction older than TOMBSTONE_RETENTION not than the params we set in vacuum interface and this is my question. Thanks for reply :)

[brkyvz]

@triplesheep You are correct, this is a legitimate bug.

[triplesheep]

@brkyvz Hi :) I have try to fix it and post a pr.