Issue 002 · September 2026

Fix the maxResultSize error — three fixes, ranked.

Reduce collected data first, fix runaway broadcast joins second, raise the cluster limit last.

The tip

Three fixes, ranked

The error — Total size of serialized results of N tasks (X GB) is bigger than spark.driver.maxResultSize (Y GB) — means one Spark action tried to pull more data to the driver than the limit allows (default 1 GB). The limit counts serialized results across all partitions for actions like collect(), toPandas(), or show() on a huge result. Fix in this order:

Fix 1 — Collect less (the real fix, most of the time)

Aggregate, filter, or sample before the action — or skip the driver entirely and write the result to storage:

# Instead of df.collect() on millions of rows:
summary = df.groupBy("event_type").count().orderBy("event_type")
summary.show()   # small result, safe to display

# Or skip the driver entirely:
df.write.mode("overwrite").parquet("s3://bucket/out/")

Fix 2 — Fix the runaway broadcast

If the failed stage is a BroadcastExchange, Spark tried to ship a too-large table to the driver — usually because stale statistics made a big table look small. Diagnose the plan, then fix the decision, not the limit:

-- Refresh the statistics the optimizer decides on
ANALYZE TABLE events COMPUTE STATISTICS;

# Or stop this session broadcasting the large side (default threshold is 10 MB)
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", "-1")

If a table above the 10 MB threshold is being broadcast, suspect stale stats first.

Fix 3 — Raise the limit last

Only when the workload genuinely must collect that much data: set spark.driver.maxResultSize in the cluster's Spark config to a value above the reported size. Check the current value first in the Spark UI → Environment tab. This needs real driver memory headroom — setting it very high just moves the failure to a driver OOM. Never set it to 0 (unlimited); the limit exists to protect the driver.

How to validate: re-run the action and confirm the stage completes; then check driver memory in the Spark UI — if it ran near the top, you still collected too much and Fix 1 remains the right answer.

Use Fix 3 when: a legitimate one-off export must land on the driver and the driver has memory to spare. Avoid it when: the collect sits inside a recurring pipeline — rewrite the pipeline instead.

Implementation breakdown

The dashboard export that killed the driver

Illustrative scenario: a composite built from common maxResultSize failure modes — not a report of one observed incident.

The failure

A job collects a full orders table to the driver to write a local CSV for a dashboard export. It dies with Total size of serialized results of 196 tasks (32.8 GB) is bigger than spark.driver.maxResultSize (32.0 GB). Someone had already raised the limit once — the data just kept growing past it.

Step 1 — Read the error, name the path

The message tells you how far over the limit the action went. Then open the failed stage in the Spark UI: a failed result stage means the collect path (Fix 1); a failed BroadcastExchange means the broadcast path (Fix 2). The two demand different fixes — don't raise the limit for a broadcast problem.

Step 2 — Apply the matching fix

Collect path: push the aggregation into Spark and collect the summary, or write the full result to cloud storage and let the dashboard read it from there. Broadcast path: run ANALYZE TABLE on the large side, confirm the plan no longer broadcasts it, and only then re-run.

Step 3 — Raise the limit only if a real collect remains

If a genuine driver-side collect is unavoidable, raise spark.driver.maxResultSize in the cluster Spark config and size the driver's memory to match. Treat the raised value as a ceiling you monitor, not a problem solved.

The result: the export stops dying on data growth — because the data no longer funnels through the driver, or the driver is honestly sized for what it must hold.

Dev Q&A

My data is small — why did this fail?

Three usual suspects. Stale stats, bad broadcast: the optimizer thought a large table was under 10 MB and broadcast it; ANALYZE TABLE fixes the estimate. Many partitions: the limit is the total across tasks, so thousands of small results add up past 1 GB. Conversion overhead: toPandas() via Arrow can hold several times the serialized size in memory — the limit trips before the OOM does.

Related: on serverless compute and SQL warehouses the config can't be changed — reduce the collected data or, for BI tools, enable Cloud Fetch so results stream from cloud storage instead of the driver.

About this article

Applies to: Apache Spark on Databricks classic compute. On serverless compute and SQL warehouses spark.driver.maxResultSize cannot be changed — collect less data instead.
Last verified against Databricks docs: September 24, 2026.
Corrections: Sep 24, 2026 — full article restored to the archive (previously a restoration placeholder).