Your bad rows didn't fail — they went to _rescued_data

The values moved

Auto Loader's rescued data column quietly quarantines anything the schema can't parse — unknown columns, type mismatches, case mismatches — into _rescued_data as JSON, and the load stays green. The nulls don't mean the values were missing. They mean the values were moved.

The one-line monitor that belongs on every bronze table you ingest with Auto Loader:

SELECT * FROM bronze_events WHERE _rescued_data IS NOT NULL

If that query returns rows, your load didn't fail — it buried the evidence. The JSON blob also carries the source file path, so you can trace every rescued record back to the exact file that carried it.

Illustrative demo — deterministic from the setup

Not an observed production result: the setup writes exactly two JSON records. The row counts and outcomes below follow deterministically from those inputs and the declared schema.

Setup — idempotent, creates every object it references

dbutils.fs.rm("dbfs:/tmp/demo_rescue", True)
dbutils.fs.put("dbfs:/tmp/demo_rescue/clean.json", '{"id": 1, "amount": 100}\n', True)
dbutils.fs.put("dbfs:/tmp/demo_rescue/dirty.json", '{"id": 2, "amount": "one hundred"}\n', True)
bronze = (spark.read.schema("id INT, amount INT").option("rescuedDataColumn", "_rescued_data").json("dbfs:/tmp/demo_rescue"))
bronze.createOrReplaceTempView("demo_bronze")

Inspect the outcome

SELECT id,
       amount,
       _rescued_data IS NOT NULL AS was_rescued
FROM demo_bronze
ORDER BY id;

Where malformed data goes

Routing, per official docs:

1. Column not in schema, type mismatch, case mismatch: _rescued_data JSON; the load stays green.
2. Whole record malformed — broken JSON or CSV: the legacy _corrupt_record option, or badRecordsPath.
3. Corrupt file — cannot be opened: skipped and counted in numSkippedCorruptFiles.
4. COPY INTO ingestion: no rescue column at all — badRecordsPath only.

The quarantine pattern for production bronze

1. Ingest with rescuedDataColumn set explicitly, even with a declared schema.
2. Schedule the detector query: count rows where _rescued_data IS NOT NULL per batch, and alert — do not merely log.
3. Triage each rescued field: extract it with get_json_object(_rescued_data, '$.<field>'). Promote it by fixing the upstream producer or adding the column via addNewColumns evolution and reprocessing the file; or quarantine it by routing to a DQ table and holding it from silver.
4. Know your mode: addNewColumns fails the stream on a new column and evolves on restart; rescue mode never evolves — new columns park in the rescue column forever. Pick deliberately; rescue is not a schema-evolution strategy.

Bronze rescue monitor: run per batch, alert on any rows

SELECT COUNT(*) AS total_rows,
       SUM(CASE WHEN _rescued_data IS NOT NULL THEN 1 ELSE 0 END) AS rescued_rows
FROM bronze_events;

Triage the rescued value

SELECT get_json_object(_rescued_data, '$.amount') AS rescued_amount,
       _rescued_data AS rescue_blob
FROM bronze_events
WHERE _rescued_data IS NOT NULL;

Use this when

1. Messy JSON or CSV: ingesting from sources you do not control.
2. Loss-sensitive bronze: losing data is worse than a green load with clearly flagged rows.
3. Backfill resilience: you would rather quarantine an unexpected field than abort at 3 a.m.

Avoid this when

1. A null key is worse than a failed load: strict contracts want an explicit schema, no rescue column, and FAILFAST so the deviation wakes someone up.
2. Downstream consumers cannot tolerate nullable-everything bronze: fail before ambiguity becomes a contract.