Spark File Size & Partition Planner
Work out how many files a Spark job should write for a target file size, with partitionBy, task slots and maxRecordsPerFile, and every assumption shown.
Open the planner →Small, focused utilities for the jobs that come up every week in data engineering. Each one shows its working, so you can check the answer and learn from it, not just copy it.
Work out how many files a Spark job should write for a target file size, with partitionBy, task slots and maxRecordsPerFile, and every assumption shown.
Open the planner →Paste sample JSON records and get a PostgreSQL CREATE TABLE: inferred types, safe column names, jsonb or flattened nesting, and every decision explained.
Open the generator →Describe a table, pick rules (required, unique, accepted values, non-negative, dates) and get read-only PostgreSQL checks, each explained, plus constraints.
Open the generator →Data engineers, analytics engineers, SQL learners and anyone running Spark or PostgreSQL in production. The tools are free for members: create an account once and use all of them. They run on our server in milliseconds, and what you type is used only to produce the answer. It is never stored, logged or sent anywhere else.
Every result comes from plain, deterministic calculations, with no AI guessing and no live database. The pages explain the method, the assumptions and the limits, because a sizing estimate or a generated schema is a starting point for your judgement, not a replacement for it.