Short enough to read between meetings
Every issue is a 10-minute read: one tactic, one pattern, one answered question.
ISSUE 028 · OCTOBER 16, 2026
Your notebook works on the old cluster and dies on serverless — your DBFS mounts are deprecated
DBFS mounts are a deprecated pattern — and the deprecation is now a hard break. Mounts don't work on serverless compute or Unity Catalog, bypass UC governance entirely, and don't exist on workspaces provisioned after September 30, 2026. This issue's six-step ladder migrates /mnt/ paths to Unity Catalog volumes.
THE DIFFERENCE
Why Lakehouse Digest
Code you can run today
Copy-paste PySpark and SQL checked against official Databricks docs.
Primary sources, no hype
Claims link to primary sources — official docs and verified community deep-dives; no invented benchmarks.
THE THREE-PART READ
Your notebook works on the old cluster and dies on serverless — your DBFS mounts are deprecated
DBFS mounts are a deprecated pattern — and the deprecation is now a hard break. Mounts don't work on serverless compute or Unity Catalog, bypass UC governance entirely, and don't exist on workspaces provisioned after September 30, 2026. This issue's six-step ladder migrates /mnt/ paths to Unity Catalog volumes.
Inventory before the move
Find mounts, FileStore dependencies, init scripts, libraries, notebooks, and code before target compute finds them for you.
Read Issue 028 →Register, re-point, verify
Use an external volume for existing cloud data, copy DBFS-root files, and cut reads over to governed paths.
Read Issue 028 →The path has nothing behind it
Serverless and new UC-only workspaces do not provide the legacy mount your notebook expects.
Read Issue 028 →Last verified against official Databricks documentation: October 16, 2026. Correction history: none (new issue).
Continue: Read the full Issue 028 article →
Run the inventory before target compute finds the break.
Run a mount inventory BEFORE your next workspace or serverless move: list every mount with dbutils.fs.mounts(), then grep your notebooks and jobs for /mnt/, dbfs:/FileStore, and dbfs:/databricks/init. Rule of thumb: /mnt/ data becomes a Unity Catalog volume; notebooks and code become workspace files or Git folders; JARs and wheels become volumes too. Init scripts move to volumes on classic compute — but serverless doesn't run init scripts at all, so a serverless move means switching to serverless environments instead.
Continue: Read the full Issue 028 article →
The six-step migration ladder
- Inventory:
dbutils.fs.mounts(); grep notebooks, jobs, and libraries for/mnt/,dbfs:/FileStore, anddbfs:/databricks/init. - Create an external location over the cloud bucket (one-time admin setup; storage credential + metastore admin).
- Register the data without moving it:
CREATE EXTERNAL VOLUME IF NOT EXISTS main.data.raw_files LOCATION 's3://my-bucket/csv-files/';— LOCATION takes a full cloud-storage URI (the external location from step 2 must cover that path). When files already live in cloud storage, no data movement is needed. For brand-new file collections, a managed volume (no LOCATION) is the simpler choice. - Copy DBFS-root files:
dbutils.fs.cp("dbfs:/FileStore/…", "/Volumes/<cat>/<sch>/<vol>/…"); large sets go through the Databricks CLIfs cp -ror a distributed Spark copy. - Re-point every read to
/Volumes/<catalog>/<schema>/<volume>/…and grantUSE CATALOG+USE SCHEMA+READ VOLUME(writers also needWRITE VOLUME). - Verify on the target compute first (
dbutils.fs.ls("/Volumes/<cat>/<sch>/<vol>/")from serverless or the new workspace), then clean up:dbutils.fs.unmount("/mnt/oldpath")— and lock down or delete the moved DBFS-root data. Do not keep the same cloud storage prefix both mounted and registered.
Also: JARs and Python wheels move to UC volumes; notebooks and code move to workspace files or Git folders — not to volumes. Init scripts are compute-scoped: on classic UC compute, store them in UC volumes — but serverless compute does not run init scripts at all, so a serverless move replaces them with serverless environments (a base environment or notebook dependencies).
Recommended learning: DataCamp Databricks courses · DataCamp data engineering
Full walkthrough: Read Issue 028 →
DEV Q&A · 2 MIN
The legacy path is compute-bound.
Replace the mount with a governed volume path, then verify it on target compute before cutting over.
My notebook runs fine on our old cluster but fails on serverless / a new workspace — why?
Because the mount doesn't exist there. Mounts are incompatible with serverless compute, and workspaces provisioned after September 30, 2026 aren't provisioned with DBFS root or mounts at all — the /mnt/... path your notebook references simply has nothing behind it on that compute. The fix is the migration: register the bucket prefix as an external volume, re-point reads to /Volumes/<catalog>/<schema>/<volume>/…, and grant USE CATALOG + USE SCHEMA + READ VOLUME. Verify with dbutils.fs.ls() on the volume path from the target compute before cutting over.
Full walkthrough: Read Issue 028 →
ABOUT THIS ARTICLE
Scope, verification, and sources.
Last verified against official Databricks documentation: October 16, 2026. Correction history: none (new issue). Official documentation — Upgrade a Databricks workspace to Unity Catalog, DBFS mounts, DBFS and Unity Catalog best practices, and Unity Catalog privileges reference (READ VOLUME, WRITE VOLUME, USE CATALOG, and USE SCHEMA). Practitioner case study — not core documentation: From HMS to Unity Catalog: A Self-Service Migration Playbook.
BROWSE BY TOPIC
Follow the problem you're solving.
Jump straight to the permanent issue pages for each topic.
Migration & validation
Data modeling & integrity
Streaming & watermarks
Performance & tuning
Delta Lake & file layout
Debugging & reliability
Ingestion & Auto Loader
Data retention & governance
ARCHIVE
All issues
Issues, newest first.
28 issues
- Your notebook works on the old cluster and dies on serverless — your DBFS mounts are deprecated
DBFS mounts are a deprecated pattern — and the deprecation is now a hard break. Mounts don't work on serverless compute or Unity Catalog, bypass UC governance entirely, and don't exist on workspaces provisioned after September 30, 2026. This issue's six-step ladder migrates
/mnt/paths to Unity Catalog volumes. - Your DELETE was fast, but your reads got slower — deletion vectors are the hidden bill
Deletion vectors make DELETE, UPDATE, and MERGE fast by marking rows as modified in metadata instead of rewriting whole Parquet files — but every read must apply those deletion-vector entries at query time. That read-time merge is the hidden bill. The Auto-Enable rollout is flipping the default to "All new tables", so check your tables — then run the purge runbook below.
- Your table evolved without asking — the silent schema auto-merge
One session setting let every write silently widen every Delta table it touched — no ALTER TABLE, no error. Schema enforcement is the gate; evolution should be a deliberate per-write opt-in, never a session-wide config.
- Your bad rows didn't fail — they went to _rescued_data
Auto Loader silently quarantines unknown columns, type mismatches, and case mismatches in
_rescued_dataas JSON while the load stays green. Detect every rescue withWHERE _rescued_data IS NOT NULL, then route, triage, and promote or quarantine deliberately. - Your foreachBatch wrote the batch twice — at-least-once is the contract, not a bug
The engine re-delivers the same batch_id when a micro-batch fails after committing — and a plain append writes it all twice. The documented fix: txnAppId + txnVersion bound to batch_id on Delta targets, batch_id-keyed idempotency for external sinks, and a fresh txnAppId for every new checkpoint lineage.
- Your expired Kafka offsets are gone — stop trying to re-read them
The restarted job reads committed offsets from its checkpoint — but Kafka retention may have deleted them, and the query dies with KAFKA_DATA_LOSS. startingOffsets is ignored on restart; the three documented recoveries, the never-hand-edit rule, and how to prevent the trap.
- Your shallow clone is a lease, not a copy: one VACUUM ends it
SHALLOW CLONE copies metadata only — num_copied_files is 0 and the data files stay in the source directory. One scheduled VACUUM on the source ends the lease with a FileNotFoundException; the repair, the Unity Catalog caveats, and when to deep-clone instead.
- Your one-second column rename broke three things you didn't check
Metadata-only renames and drops via column mapping — id mode on new tables, name mode on existing — plus the streaming, partition-directory, and CDF breakage the docs warn about, and the full-rewrite price of turning it back off.
- Your DROP TABLE deleted the data, not just the metadata
DROP TABLE on a managed table marks its data files for deletion — on an external table it only unregisters metadata. The type gate, the UNDROP runbook, and why pipelines should use CREATE OR REPLACE instead of drop + recreate.
- Your Delta stream dies the day someone runs an UPDATE — change data feed is the fix
Structured Streaming accepts append inputs only — one UPDATE, DELETE, MERGE, or OVERWRITE fails the stream. Change data feed is the documented fix when corrections must reach downstream, but it records changes only from enablement onward.
- Your time travel expires in 7 days, not 30
Time travel and RESTORE expire when deleted data files are vacuumed (7 days by default) even though the log keeps 30 days of history — audit both windows and set them before you need the rewind.
- Your Reads Are Drowning in Tiny Files
Measure average file size from
DESCRIBE DETAIL, repair existing fragmentation with idempotentOPTIMIZEbin-packing, then prevent recurrence with optimized writes and auto compaction. - Your watermarked dedup is still hoarding state
A watermark placed after dropDuplicates never reaches the dedup operator, and even with the right order the state store stays unbounded unless the event-time column is in the dedup keys. Two traps, one fix.
- Migrated from SAS? Prove the numbers survived: the six-layer validation ladder
A converted CASE mapped LOC → 0.75 while the SAS ground truth said 1.00 — with perfect row counts. The six-layer ladder: inventory, counts, schema contract, exceptAll comparison, epsilon aggregates, and enforced gates.
- COPY INTO won't reload your fixed file: the filename-tracked idempotency trap
COPY INTO skips files it has already loaded — even modified ones. Force re-reads without deduping; the repair is delete-by-business-key then force, or better: versioned drop filenames.
- Rank, don't dedupe: duplicate-safe CDC MERGE patterns
A CDC change window is an ordered timeline per key. Rank to the latest event, filter NULL keys, then MERGE with explicit delete, update, and insert branches; never infer deletion from a key's absence in the window.
- Your INSERT OVERWRITE just deleted data you never touched
INSERT OVERWRITE defaults to static mode and, without a PARTITION clause, truncates all table data — the legacy dynamic-mode escape hatch is not supported on SQL warehouses or serverless. The compute-independent fix: REPLACE USING, plus the full WHERE/USING/ON menu.
- Your PRIMARY KEY isn't protecting you
Delta's enforced-vs-informational split: which constraints actually block writes, the RELY optimizer foot-gun, and the pipeline-code enforcement pattern.
- Late doesn't mean queued: the watermark's silent drop contract
- Expired isn't gone: Auto-TTL's real timing contract and the 17-day math
- Your schemaHint isn't a cast: the Auto Loader mixed-type trap
- The offset that vanished: recover a dead Kafka stream without silently losing data
- One key, one row: dedupe the source before you MERGE
- Don't VACUUM your time travel away: the 7-day floor
- Tame the skewed join: one rule, one check, one fix
- Kill the small-file tax: two flags that keep streaming pipelines fast
- Fix the maxResultSize error — three fixes, ranked.
- Repair the task. Not the whole pipeline.
No issues match — try the Resources page
KNOWLEDGE SEARCH
Ask Lakehouse Digest
Ask a Databricks implementation question. Answers are grounded in Lakehouse Digest issues and linked sources.
Answers may be incomplete. Check the linked official documentation before changing production workloads.
ABOUT
About Lakehouse Digest.
Contact: sbogale@sb9tech.com
Lakehouse Digest is a short, practical newsletter for Databricks engineers, published by the Lakehouse Digest team. Each issue delivers one tactic to try, one production pattern to understand, and one hard developer question answered with code and primary-source links. It is an independent publication and is not affiliated with Databricks, Inc.
RECOMMENDED LEARNING
Build the skill behind the fix.
Topic-led learning paths for readers who want to turn this issue’s tactics into repeatable Databricks practice.
Databricks fundamentals
Strengthen the platform basics behind notebooks, jobs, data workflows, and collaborative development.
Browse DataCamp Databricks courses →The mount-to-volume migration ladder
Inventory legacy paths, register existing cloud data as governed volumes, and verify the cutover on target compute before unmounting.
Read Issue 028 →Practice the platform skills: Databricks · Data engineering
Data engineering workflows
Pipelines, orchestration, and streaming practice that turn this issue's tactics into repeatable habits.
Browse DataCamp data engineering →Partner links: we may earn a commission if you enroll through these recommendations, at no extra cost to you.
FAQ
Questions, answered.
What is Lakehouse Digest?
A short, practical newsletter for Databricks engineers — one tactic, one production pattern, one hard developer question per issue.
How often do issues publish?
Daily.
Is it free?
Yes.
How do I get issues by email?
Email delivery isn't wired up yet — a subscribe form will appear on this page when it launches; nothing here collects your address today.
Who publishes it?
The Lakehouse Digest team — an independent publication, not affiliated with Databricks, Inc.
Can I sponsor an issue?
Yes — email sbogale@sb9tech.com with subject “Sponsoring Lakehouse Digest”.
FREE DOWNLOAD
Lakehouse health-check notebook
Seven diagnostics from Issues 001–009 in one Databricks notebook: small-file detection, VACUUM safety, skew checks, MERGE dedupe validation, streaming checkpoint recovery, driver-memory safety, and failed-job repair. Import it into your workspace and run it against your own tables.
SPONSORSHIP
Sponsor Lakehouse Digest.
A technical audience of Databricks engineers, reading implementation notes they can test the same day.
Launch inventory is one clearly labeled, problem-first placement per issue — built for products that help data teams build, govern, observe, or ship more reliably.
Sponsor Lakehouse Digest →Primary sources for this issue
Official documentation
- Upgrade a Databricks workspace to Unity Catalog — migrate files stored in DBFS
- Mounting cloud object storage on Databricks — deprecation notice
- DBFS and Unity Catalog best practices
- Unity Catalog privileges reference