Know the skew rule, then tune the threshold
Default on: spark.sql.adaptive.skewJoin.enabled = true. Skewed means BOTH: partition size > 5 × the per-side median partition size AND > 256MB.
Open the Spark UI, Stages tab, and find your join stage: one task's shuffle read is 20–50× the median and its duration dwarfs everything else. That's a skewed partition. AQE's skew handling is on by default on Databricks and splits such partitions into roughly evenly sized sub-partitions (replicating the other side as needed) for sort merge and shuffle hash joins.
The catch: a partition only counts as skewed when it clears both bars — bigger than 5× the per-side median partition size (spark.sql.adaptive.skewJoin.skewedPartitionFactor) and bigger than 256MB (spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes). On a smaller shuffle — say 10GB total with a 20MB median — a 150MB monster partition slips under the 256MB default and never gets split. The tactic: lower the threshold for that one job so the rule catches it.
spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes", "64MB")How to validate: run the query, then check the final plan — look for the AdaptiveSparkPlan node with isFinalPlan=true. A SortMergeJoin showing isSkew=true is AQE telling you it split a skewed partition. But isSkew=false does not prove the partition missed the detection rule — it can also mean the join wasn't eligible for skew handling (check the join-type table in the Q&A), AQE never applied (streaming queries, or a plan with no exchange), or skew handling was disabled. Rule those out first. Note: SQL EXPLAIN never shows this — the final plan only exists after the query actually runs.
Use this when: one partition dominates the stage but sits under the 256MB default — a smaller shuffle where the median is low. Avoid this when: the join isn't eligible for AQE skew handling (broadcast joins are never skew-optimized — check the join-type table in the Q&A), or the partition still misses the rule after tuning. Note: a single hot key can be divided across tasks — AQE splits skewed partitions into roughly evenly sized tasks and replicates the matching side as needed, so hot-key skew benefits whenever the partition trips the rule and the join/shuffle layout is eligible. Salt the key when AQE remains insufficient (see the pattern).
Caution: don't drop the threshold below 64MB — Spark's own guidance keeps it at or above spark.sql.adaptive.advisoryPartitionSizeInBytes (default 64MB). Near-median partitions get split pointlessly, adding task overhead.
The nightly join with the '-1' user_id
Illustrative scenario: the walkthrough below is a composite example built from common skewed-join failure modes — not a report of one observed incident. The diagnostic steps, code, and validation checks apply to any join dominated by a hot key.
Two ways to take the hot key out of the shuffle — pick one based on the dimension's size, not both in sequence.
The failure
A nightly ETL joins a large events table to a users dimension on user_id. Two or three percent of events are untracked bot traffic stamped with user_id = '-1'. Every one of those rows hashes to the same shuffle partition, so one executor grinds for 40 minutes while the rest finish in three. AQE never fired — the giant partition didn't clear both bars of the skew rule.
Broadcast the dimension when it fits
When the dimension is small enough to fit comfortably in memory on every executor, broadcast it and the big side never shuffles — the hot key can't pile into one giant partition. The LEFT join keeps every events row even when '-1' has no matching row in users (an inner join would silently drop those rows). Databricks' own FAQ notes a statically planned broadcast is usually faster than letting AQE discover it at runtime.
from pyspark.sql.functions import broadcast
# LEFT keeps every events row; a '-1' with no match in users
# just gets NULL user fields. Broadcasting means the big side
# never shuffles, so the hot key can't pile into one partition.
result = events.join(broadcast(users), on="user_id", how="left")Salt the hot key when the dimension is too big to broadcast
Use this instead of Step 1, not after it — pick the option that matches the dimension's size. If the dimension can't be broadcast, scatter only the hot-key rows across N salt buckets and replicate only the hot-key rows of the dimension across the same buckets — clean rows stay on salt 0 on both sides, so shuffle cost stays bounded. Join on the salted key, then drop the salt column to restore the original schema.
from pyspark.sql import functions as F
N = 16 # salt buckets — tune to the hot key's row volume
# Scatter only hot-key events across salt buckets; everything else gets salt 0
salted_events = events.withColumn(
"salt",
F.when(F.col("user_id") == "-1", (F.rand() * N).cast("long")).otherwise(F.lit(0).cast("long")),
)
# Replicate only the hot-key rows of the large dimension; clean rows keep salt 0
salted_users = (
users.filter(F.col("user_id") == "-1")
.crossJoin(spark.range(N).toDF("salt"))
.unionByName(users.filter(F.col("user_id") != "-1").withColumn("salt", F.lit(0).cast("long")))
)
# Join keys collapse (no duplicated columns); LEFT keeps every events row
result = (
salted_events
.join(salted_users, on=["user_id", "salt"], how="left")
.drop("salt")
)Validate the path you chose
Broadcast (Step 1): the final plan should show a BroadcastHashJoin — there is no SortMergeJoin to inspect, so do not look for isSkew=true. In the Stages view the events side has no shuffle exchange at all. Then confirm the LEFT join kept every row: events.count() must equal result.count() — provided users.user_id is unique. Duplicate dimension keys multiply rows in a left join, so if the dimension isn't deduplicated, compare against the pre-change result's count instead.
Salting (Step 2): in the Stages view the max task duration should be back near the median. Confirm the salt column is dropped and row counts match the pre-change result — salting must never change the row count.
AQE threshold (the tip): in the final plan, a SortMergeJoin with isSkew=true confirms where AQE split a partition. Compare before/after stage timings on your own workload.
AQE is on — so why didn't it fix my skewed join?
Three documented reasons: the partition missed the rule, the join type limits the fix, or AQE never applied.
Check these three conditions
The partition didn't trip the rule. AQE treats a partition as skewed only when BOTH hold: larger than
spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes(default 256MB) AND larger thanskewedPartitionFactor × the per-side median partition size(default 5). A 150MB giant in a small shuffle fails the first; a 300MB partition in a shuffle whose median is 100MB fails the second (5 × 100MB = 500MB). Fix: lower the threshold for that job (see the tip), or isolate the hot key (see the pattern).The join type limits it. AQE only splits skew on the sides the join type allows: inner joins on both sides, left outer only on the left, right outer only on the right — full outer joins aren't skew-optimized at all. Broadcast joins are never skew-optimized either (the broadcast side needs no splitting). And when the skew is a single key, AQE can still split the skewed partition into roughly evenly sized tasks — replicating the matching side — whenever the join type and shuffle layout are eligible (the same eligibility rules as above). Salting the key is the fallback for when AQE is ineligible or remains insufficient.
AQE didn't apply at all. It only applies to non-streaming queries with at least one exchange (join, aggregate, window) or sub-query. Streaming skew needs a different playbook.
Bonus from Databricks' own FAQ: prefer AQE's automatic skew handling over the manual skew join hint — the docs recommend the automatic path.
Applies to: batch queries on Databricks Runtime with adaptive query execution (Spark 3.x). Skew handling applies to sort merge and shuffle hash joins only — never broadcast joins; see the join-type table in the Q&A. Streaming queries need a different playbook.
Last verified against Databricks docs: September 30, 2026.
Corrections: Sep 24, 2026 — corrected isSkew=false interpretation (join eligibility and AQE applicability, not only the detection rule); renamed the broadcast step and framed salting as an alternative; labeled the walkthrough an illustrative scenario; made validation specific to each solution path; corrected the hot-key claim (AQE can split a hot-key-dominated partition into roughly evenly sized tasks when the join and shuffle layout are eligible — salting is the fallback when AQE remains insufficient); qualified the row-count check (requires unique dimension keys). Sep 30, 2026 — fixed a residual instance of the hot-key claim in the Q&A (the Sep 24 correction had missed it).