Don't VACUUM your time travel away: the 7-day floor

VACUUM removes data files that are no longer in the latest state of the transaction log and are older than a retention threshold. That threshold defaults to 7 days — and the official docs are blunt about the cost: run VACUUM on a Delta table and you lose the ability to time travel back to a version older than the retention period. The files are physically deleted; the versions they belonged to are gone for good.

Delta Lake ships a safety check that refuses RETAIN values under 7 days unless you set spark.databricks.delta.retentionDurationCheck.enabled to false — the same guard exists in open-source Delta Lake, not only on Databricks. Every RETAIN 0 HOURS horror story crossed that exact line — someone disabled the guard to 'save storage faster.' The tactic: never vacuum with a custom retention without a dry run first. DRY RUN returns up to 1000 files that would be deleted, so you see the blast radius before you pull the trigger.

-- What would this vacuum actually delete, before it deletes it
VACUUM catalog.schema.sales RETAIN 168 HOURS DRY RUN;

Recommended learning: Databricks docs: VACUUM

The failure

A team watched storage costs climb on their CDC-fed sales table — months of upserts had left thousands of superseded parquet files. The fix seemed obvious: a nightly job running VACUUM sales RETAIN 24 HOURS. When Delta's safety check refused, someone disabled it — 'we don't time travel, why keep the files?' Storage dropped 40% overnight. Two weeks later, the quarterly audit asked for the table VERSION AS OF a version from last month, and a downstream incremental reader failed the same day. DESCRIBE HISTORY still listed the versions — the transaction log was untouched — but every parquet file behind them was gone. Nothing to restore from.

1. Audit every vacuum job and the guard

Search your job definitions for VACUUM with RETAIN under 168 HOURS, and confirm the safety check is still on everywhere:

spark.conf.get("spark.databricks.delta.retentionDurationCheck.enabled")  # expect "true"

2. Raise the floor where history matters

Don't lower retention where storage is cheap — raise it on tables with audit or incremental readers, per table:

-- Keep a month of recoverable history on this table
ALTER TABLE catalog.schema.sales
SET TBLPROPERTIES ('delta.deletedFileRetentionDuration' = '30 days');

3. Leave auto-vacuum alone, and validate

If predictive optimization is enabled, Databricks already runs VACUUM automatically — but it uses your table's configured delta.deletedFileRetentionDuration (7-day default), not a fixed 7 days. Raising the property to 30 days in step 2 extends the automatic window too, so set it before relying on auto-maintenance. Setting it below 7 days won't shorten auto-cleanup: predictive optimization keeps a 7-day minimum when it runs VACUUM FULL. And it only covers Unity Catalog managed tables (Premium plan or above, Databricks Runtime 12.2 LTS and above) — external tables still need a manual VACUUM job.

Recommended learning: continue with the site's Recommended learning section.