Run a lazy query now

Status

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Background

When to use it

In Polars, waiting on execution until all the steps are defined lets the query planner optimise, for example with predicate pushdown, which applies filters as early as possible while reading the dataset.^[2] Projection pushdown, in Polars, selects only the columns a query needs while reading the dataset.^[2] In Polars, the lazy API should in general be preferred unless the intermediate results are of interest or the work is exploratory and the query's shape is not yet known.^[2] Run the query now when a result has to be looked at, handed to code outside the engine, or used several times. (Meridian's view)

A Polars LazyFrame is a query plan that is not guaranteed to cache common subplans, so each time it is reused in a separate downstream query it is computed all over again.^[4] In Polars, an operation on a LazyFrame that does not maintain row order, such as group_by, can change the order every time it is run unless maintain_order=True is set.^[4] Polars' collect can use a streaming engine, which processes queries in batches, reducing memory pressure and often outperforming the in-memory engine.^[5] DataFusion's collect buffers all the resulting record batches into memory, and its execute_stream runs a DataFrame without buffering.^[3] DataFusion's count method actually runs a plan to calculate the count, which may be slow for large or complicated DataFrames.^[3] Lazy evaluation is difficult to combine with imperative features such as exception handling and input/output, because the order of operations becomes indeterminate.^[1] Collect at a point where a failure can be traced back to the steps that built the plan, and reuse the collected table rather than the plan when the same result is needed twice. (Meridian's view)

In RapidMiner, the Materialize Data operator creates a fresh and clean copy of the data in memory, which might be useful after large preprocessing chains with a lot of views or data copies.^[6]

The method

Lazy evaluation was introduced for lambda calculus by Christopher Wadsworth, and for programming languages independently by Peter Henderson and James H. Morris and by Daniel P. Friedman and David S. Wise.^[1] In Polars, each lazy step is added to an internal query graph that is optimised, and executing the query runs the optimised graph by default.^[4] In Polars, explain describes the query plan that will be executed once the results are collected.^[2] In many cases, Polars' eager API calls the lazy API under the hood and immediately collects the result.^[2] Polars' collect_all collects multiple LazyFrames at the same time and applies Common Subplan Elimination on the combined plan, so that diverging queries run only once.^[7] A DataFusion DataFrame is a wrapper around a LogicalPlan, and collect executes the plan with the same planning and execution process DataFusion uses for SQL and other queries.^[3]

Compared with the alternatives

The opposite of lazy evaluation is eager evaluation, sometimes known as strict evaluation, which most programming languages employ.^[1] DataFusion's cache method caches a DataFrame as a memory table.^[3]

Sources

1. [1]Lazy evaluation - Wikipediasecondary source
2. [2]Lazy API - Polars user guidedocumentation
3. [3]DataFrame in datafusion::dataframe - Rustdocumentation
4. [4]Query execution - Polars user guidedocumentation
5. [5]polars.LazyFrame.collect — Polars documentationdocumentation
6. [6]Materialize Data - Rapidminer Documentationdocumentation
7. [7]polars.collect_all — Polars documentationdocumentation