Repartition the data

Status

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Background

When to use it

In AWS Glue, data that arrives without enough parallelism is common when the source is gzipped or is read through Java Database Connectivity (JDBC).^[3] In AWS Glue, redistributing data has a modest performance cost, so the optimisation might not compensate for it if the data was already well balanced.^[3] Uneven distribution of data across partitions, termed skew, reduces the efficiency of partitioning, and partitions with excessive load are called hot spots.^[1] Repartition when a few parts hold most of the rows, or the source arrived in too few parts to keep every worker busy, and time the run before and after. (Meridian's view)

In Apache Spark, a shuffle re-distributes data so that it is grouped differently across partitions, which typically involves copying data across machines.^[2] In Apache Spark, operations that can cause a shuffle include repartition and coalesce, most by-key operations, and joins.^[2] In Apache Spark, a shuffle is an expensive operation because it involves disk input and output, data serialisation, and network input and output.^[2] In Apache Spark, a drastic coalesce, such as to one partition, may run the computation on fewer nodes than intended, which repartition avoids by adding a shuffle step.^[4] In AWS Glue, the Autobalance Processing transform can take a number of partitions, though letting the system decide that value is recommended in general.^[3]

The method

Data parallelism distributes data across different nodes, which operate on the data in parallel, and contrasts with task parallelism.^[5] Exploitation of data parallelism started in the 1960s with the development of the Solomon machine.^[5] Database partitioning emerged in the 1980s with systems like Teradata and NonStop SQL.^[1]

With n partitions, round-robin partitioning, the simplest strategy, assigns the ith row in insertion order to partition i mod n, which ensures uniform distribution.^[1] Hash partitioning applies a hash function to a partitioning key to spread skewed data evenly, at the cost of range query efficiency, as adjacent keys scatter across partitions.^[1] Range partitioning assigns continuous ranges of the partitioning key to partitions, which keeps range scans efficient but can create hot spots under certain access patterns.^[1]

In Apache Spark's DataFrame interface, repartition returns a hash-partitioned result, split into a given number of partitions, by given partitioning columns, or both.^[6] In Apache Spark, coalesce reduces the number of partitions without a shuffle, so going from 1000 partitions to 100 has each new partition claim 10 of the current ones.^[4] In Apache Spark, asking coalesce for more partitions than there are leaves the data at its current number of partitions.^[4] The AWS Glue Autobalance Processing transform uses Apache Spark repartition to randomly reassign data among a number of partitions optimal for the cluster capacity.^[3] In AWS Glue, the Autobalance Processing transform can also reorganise data by specified columns when writing partitioned tables, giving output files that are more consolidated.^[3] In Talend, tPartitioner partitions input data before tCollector transfers it to parallel execution processes, whose outputs tDepartitioner assembles for tRecollector to capture.^[7]

Compared with the alternatives

Horizontal partitioning of stored data puts different rows into different tables, such as customers split by ZIP code into CustomersEast and CustomersWest.^[1] In Apache DataFusion, the PARTITION BY clause within a window function's OVER clause divides rows into groups, or partitions, that share the same values of its expressions, and the function is computed across each row's partition.^[8] Repartitioning is a different job from both: it is about how the engine spreads rows over its workers for the steps that follow, not where tables are stored or which rows a calculation sees. (Meridian's view)

Sources

1. [1]Partition (database) - Wikipediasecondary source
2. [2]RDD Programming Guide - Spark 4.3.0 Documentationdocumentation
3. [3]Using the Autobalance Processing transform to optimize your runtime - AWS Gluedocumentation
4. [4]pyspark.sql.DataFrame.coalesce — PySpark 4.3.0 documentationdocumentation
5. [5]Data parallelism - Wikipediasecondary source
6. [6]pyspark.sql.DataFrame.repartition — PySpark 4.3.0 documentationdocumentation
7. [7]Orchestration (Integration) components | Talend Components for Jobs Helpdocumentation
8. [8]Window Functions — Apache DataFusion main documentationdocumentation