Worked example

A daily job writes 750 GB of Parquet (as stored, after compression) into one table without partitionBy, aiming for 128 MiB files. The cluster has 200 task slots and the job writes about 3 billion rows. The planner's answer:

Output files5,588

Average file128.0 MiB target 128 MiB

Suggested range5,588–7,450 files, averaging 75–100% of target

Data written698.5 GiB 750,000,000,000 bytes

Write waves28 200 slots, last wave 188 tasks

maxRecordsPerFile536,870 rows per 128 MiB file

What this means

5,588 files run in 28 waves and the last one uses only 188 of 200 slots. 5,600 files (still within the size range) would fill every wave.

The same data at other target sizes
Target	Files	Average file	Waves
64 MiB	11,176	64.0 MiB	56
128 MiB yours	5,588	128.0 MiB	28
256 MiB	2,794	256.0 MiB	14
512 MiB	1,397	512.0 MiB	7
1024 MiB	699	1,023.3 MiB	4

Applying it

df.repartition(5600).write.parquet(path)   # full shuffle, even file sizes
df.coalesce(5600).write.parquet(path)      # no shuffle; only when reducing partitions

Starting points, not guarantees: check the files a real run writes and adjust. The caveats below explain why the numbers can differ.

How the planner works

The arithmetic is deliberately simple, so you can check it by hand:

Bytes written. The size you enter is converted to bytes. Decimal units (MB, GB, TB) are powers of 1000 and binary units (MiB, GiB, TiB) powers of 1024; mixing them up is a 7% error at the gigabyte scale and 10% at the terabyte scale. If you enter an uncompressed size, it is divided by your compression ratio.
Per directory. With partitionBy, each directory holds its own files, so the bytes are split evenly across the directories the run writes.
Files per directory = bytes per directory ÷ target, rounded up, so no file in the plan is larger than the target. The total is that times the number of directories.
The range runs from that count up to the count where the average file falls to 75% of the target. Anywhere in the range is a reasonable choice; if you have task slots, the planner looks for a count in the range that fills every write wave.
maxRecordsPerFile = target bytes ÷ average bytes per row, when you give a row count.

Nothing is sampled or estimated behind the scenes: the same inputs always give the same answer.

Output files are not the same as Spark partitions

Spark has several numbers that are all called "partitions", and only one of them decides how many files you get:

Read partitions come from the input files and spark.sql.files.maxPartitionBytes (128 MB by default). They decide how the input is split, not how the output is written.
Shuffle partitions (spark.sql.shuffle.partitions, 200 by default) are used by joins and aggregations. With Adaptive Query Execution, on by default since Spark 3.2, Spark coalesces small shuffle partitions towards spark.sql.adaptive.advisoryPartitionSizeInBytes, so the real number varies from run to run. Size these by shuffle data in memory, not by file size.
Write tasks are the partitions of the final stage. Each task writes one file for every output directory it holds rows for, and more if maxRecordsPerFile splits them. Without partitionBy, files ≈ non-empty write tasks; with it, files can approach tasks × directories.

That is why the planner asks for directories and shows waves only when there is no partitionBy: those are the cases where the file count follows directly from the numbers.

Ways to get the file count you planned

repartition(n) shuffles the data into n evenly sized partitions: the most predictable way to get n similar files, at the cost of a shuffle.
coalesce(n) merges partitions without a shuffle. It is cheaper but only reduces the count, and it can also reduce the parallelism of the work before the write.
repartition(*partition_cols) before partitionBy sends each directory's rows to one task (about one file per directory); add maxRecordsPerFile to split large directories. Beware skew: one huge directory becomes one slow task.
Table formats can do the sizing for you: Delta Lake's optimized writes and OPTIMIZE, or Apache Iceberg's write.target-file-size-bytes (512 MB by default). Use the planner to sanity-check what they produce.

Limitations and assumptions

Compression is data-dependent. An uncompressed-to-Parquet ratio can be anywhere from about 2:1 to more than 10:1; measure one real partition and plan with the size on storage when you can.
Data is assumed to be evenly spread. Skewed keys and skewed partition directories make some files much larger or smaller than the average.
File size is not the only goal. Very large files speed up scans but make each retry and each small update more expensive; streaming and frequently updated tables often use smaller targets.
The planner sizes one write. Appends over time still create small files; schedule compaction for those.

Frequently asked questions

What is a good target file size for Parquet in Spark?

Most teams use 128 MiB to 1 GiB. Around 128–256 MiB suits tables that are updated or read selectively; 512 MiB to 1 GiB suits large, append-only tables that are mostly scanned. Parquet's default row group is 128 MB, so files smaller than that give up some of the format's benefits.

Why does my job write 200 files?

200 is the default for spark.sql.shuffle.partitions. If the last stage before the write is a shuffle, each of its 200 tasks writes a file. With Adaptive Query Execution on, Spark may coalesce those into fewer, larger partitions, so you might see fewer than 200.

Should I use repartition or coalesce before writing?

Use repartition(n) when you want evenly sized files and can afford a shuffle. Use coalesce(n) to cheaply reduce a large number of small partitions, but check that it does not squeeze an expensive transformation onto too few tasks.

Does the planner work for Delta Lake and Iceberg tables?

Yes: they store Parquet files, so the same sizing logic applies. Both formats can also manage file sizes themselves (optimized writes, OPTIMIZE, target-file-size settings); the planner tells you what to expect from them.

Why do I need an account?

The tools are free for members. Signing in lets us keep them free and fast, and protects them from automated abuse. Your input is used only to produce the answer and is not stored.

Spark File Size & Partition Planner

Create a free account to use the planner