Your Delta stream dies the day someone runs an UPDATE — change data feed is the fix

One modifying commit breaks a plain stream

Structured Streaming treats a Delta source as an append-only log. The official contract is blunt: any UPDATE, DELETE, MERGE INTO, or OVERWRITE on the source table fails the stream with an error. There is no “tolerate small changes” mode — the day a monthly restatement or a GDPR delete lands on bronze, the silver stream reading it dies on the next micro-batch.

This is why append-only tables stream for months and then break overnight: the failure is not in your query, it is in the source's write pattern changing. The fix is a decision, made before the restatement lands.

-- 1. Enable legacy CDF on the source (Delta tables only).
ALTER TABLE bronze.orders
  SET TBLPROPERTIES ('delta.enableChangeDataFeed' = 'true');

-- 2. Or default it on for all NEW tables in this session:
SET spark.databricks.delta.properties.defaults.enableChangeDataFeed = true;

-- 3. Verify before you cut over — must return true:
SHOW TBLPROPERTIES bronze.orders ('delta.enableChangeDataFeed');

The four documented approaches

When the source table can change, the official documentation compares four approaches. skipChangeCommits is the simple appends-only escape hatch — it ignores modifying commits entirely, and corrections never propagate; the docs recommend it “for most workloads that do not use change data feeds.” Full refresh is simple but expensive at scale. Change data feed processes every change type and is “the most robust approach because your code explicitly handles every type of change event.” Materialized views propagate changes automatically but at higher latency. The standing recommendation: “streaming from the CDC feed of a Delta Lake table rather than directly from the table whenever possible.”

Step 1 — BEFORE: enable the feed (it never backfills)

Illustrative, idempotent setup:

CREATE OR REPLACE TABLE demo.cdf_demo (
  order_id BIGINT, amount DOUBLE
) TBLPROPERTIES ('delta.enableChangeDataFeed' = 'true');  -- version 0
INSERT INTO demo.cdf_demo VALUES (1, 100.00), (2, 200.00), (3, 300.00);  -- version 1
UPDATE demo.cdf_demo SET amount = 150.00 WHERE order_id = 1;             -- version 2

Legacy change data feed records only changes that occur after it is enabled. Enable it on day one — a restatement that lands before enablement is invisible to the feed forever, and an off-then-on gap stays unqueryable.

Step 2 — READ: every change, typed

SELECT _change_type, order_id, amount, _commit_version
FROM table_changes('demo.cdf_demo', 0)
ORDER BY _commit_version, order_id;

Illustrative expected output — expected shape, not measured: version 1 contributes three insert rows; version 2 contributes one update_preimage row (order_id 1, amount 100.00 — the value before) and one update_postimage row (order_id 1, amount 150.00 — the value after). Each record also carries _commit_version and _commit_timestamp; delete is the fourth change type. A table whose schema already uses those column names cannot enable CDF until they are renamed.

Step 3 — CUTOVER: skip the snapshot replay

Illustrative recipe; verified option names only:

(spark.readStream
  .format("delta")
  .option("readChangeFeed", "true")
  .option("startingVersion", 76)      # last-processed version + 1
  .table("bronze.orders")
  .writeStream
  .format("delta")
  .option("checkpointLocation", "/mnt/checkpoints/silver_orders_cdf_v2")  # NEW location
  .start("silver.orders"))

When a CDF stream first starts without a starting bound, it returns the latest snapshot as INSERT records. If the target is already caught up, cut over with startingVersion = last-processed + 1 against a new checkpoint location so that the new starting bound takes effect. An existing checkpoint resumes its saved state instead. A version older than CDF enablement raises an error; a version aged out of table history fails the stream start.

Step 4 — GATES: make the regression fail loudly

-- Gate 1: CDF must be ON before the cutover. Must return true.
SHOW TBLPROPERTIES demo.cdf_demo ('delta.enableChangeDataFeed');
-- Gate 2: never read below the enable version.
DESCRIBE HISTORY demo.cdf_demo;  -- find where the property landed

Gate 1 is a real gate — the cutover must not proceed unless the property is on. Gate 2 leans on the engine: the docs' contract is that a table_changes start version older than CDF enablement raises an error, so start >= enable version is enforced whether the pipeline checks it or not.

Automatic versus legacy CDF

Automatic CDF requires DBR 19+ and a Unity Catalog-registered table; Delta tables need row tracking enabled. It is computed at query time, needs no per-table CDF configuration, and the docs recommend migrating to it. Legacy CDF is materialized at write time, works only with Delta tables, and requires the per-table property. You cannot use both together.

1. Confirm enablement: SHOW TBLPROPERTIES <table> ('delta.enableChangeDataFeed') must return true before any cutover.
2. Find the enable version: DESCRIBE HISTORY — never set a table_changes start or readChangeFeed startingVersion below it; the read raises an error.
3. Decide: skipChangeCommits=true when corrections do not matter downstream, versus readChangeFeed=true when corrections must propagate and downstream must handle update_preimage/update_postimage pairs.
4. Cut over: use a NEW checkpoint location plus startingVersion = last-processed + 1. A new CDF read without that starting bound returns the current snapshot as INSERTs; an existing checkpoint resumes its saved state.
5. Archive rule: if CDF output is your audit trail, write it incrementally to a separate table — CDF is transient and ages out with the table.

Use this when

1. Upstream corrections — restatements, GDPR deletes, or late arrivals — must reach downstream.
2. You need row-level auditability between versions through _change_type, _commit_version, and _commit_timestamp.
3. An incremental silver-to-gold pipeline reads a table that sees MERGE, UPDATE, or DELETE.

Avoid this when

1. The source is strictly append-only — a plain streaming read is simpler, and legacy CDF can add storage overhead in separate change data files.
2. Corrections are handled elsewhere and downstream replays would be wrong — use skipChangeCommits.
3. You need permanent history — CDF is transient and ages out with table retention; archive it yourself.