Vacuum parallel delete is affected by AQE

[Kimahriman]

Currently the vacuum.parallelDelete.enabled flag uses the default shuffle partitions as the parallelism control for deleting, since it is the result of a join. However, when the adaptive query engine is enabled, this affects the partitioning of the join, and since it is purely on file metadata, the resulting shuffle is small and AQE will merge these to a very small number of partitions regardless of your default partition setting.

This wasn't a huge deal since AQE is disabled by default, but in Spark 3.2 it will be enabled by default so this will affect more people.

Possible solutions:

• Log something if AQE is enabled saying it will affect the parallelism of the delete
• Try to temporarily turn off AQE while doing the vacuum
• Change the existing setting or add a new setting for exactly how many partitions you want, and just do a repartition before deleting to the count

[rahulsmahadev]

Thanks for reporting this, do you want to work on the potential fix here ?

[tdas]

Yeah, I am inclined to simply turn off AQE and maintain the previous behavior. we can do a simple check.. if parallel delete is enabled, then disable AQE before starting the job.

[Kimahriman]

Yeah I can work on a fix, just wanted to get a consensus on the solution. I'll play around with disabling AQE for it

[tdas]

we are planning to make a delta 1.1 release soon since spark 3.2 is about to be released. so would be awesome if we are able to release with this fix.

[Kimahriman]

I'll try to get figure it out quick, I want to upgrade asap too, glad to hear you are planning to support 3.2 quickly

[Kimahriman]

Got something thrown together so let me know your thoughts. Verified locally