# Plan and run queries

Control / engine

query planning, optimisation, memory, caching, partitioning, parallelism

Page: <https://meridian.online/operations/plan-and-run-queries>

## Status

A feature, not a step.

It lives in how a Protocol runs. [Cargo.toml in arcform at fc62d6f](https://github.com/meridian-online/arcform/blob/fc62d6f861dd/Cargo.toml)

## Variants

- [Cache an intermediate result](https://meridian.online/operations/plan-and-run-queries/cache-an-intermediate-result): Hold a table or column in memory, or in a stored copy, so later steps read it without recomputing it.
- Run within memory: Keep execution inside a memory limit by streaming batches and spilling to disk, and report memory and temporary-file use.
- Parse and optimise the query plan: Turn a query into a plan and rewrite it, pushing filters, projections and limits toward the scan and reordering joins.
- [Repartition the data](https://meridian.online/operations/plan-and-run-queries/repartition-the-data): Split or redistribute rows over a set number of partitions or threads so the work runs in parallel.
- Reload from the source: Read a sheet again from its source, on demand, on a timer or when the file changes, clearing any cache first.
- Maintain the database file: Checkpoint, vacuum and report on the size and storage of the database file and its connections.

## What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

- Apache DataFusion: [`Constant folding`](https://datafusion.apache.org/user-guide/features.html) (read 2026-09-27)
- Apache DataFusion: [`DataFrame API`](https://datafusion.apache.org/user-guide/features.html) (read 2026-09-27)
- Apache DataFusion: [`Join Reordering`](https://datafusion.apache.org/user-guide/features.html) (read 2026-09-27)
- Apache DataFusion: [`Limit Pushdown`](https://datafusion.apache.org/user-guide/features.html) (read 2026-09-27)
- Apache DataFusion: [`Memory limits enforced`](https://datafusion.apache.org/user-guide/features.html) (read 2026-09-27)
- Apache DataFusion: [`Parallel query execution`](https://datafusion.apache.org/user-guide/features.html) (read 2026-09-27)
- Apache DataFusion: [`Parquet: Predicate push down (late materialization)`](https://datafusion.apache.org/user-guide/features.html) (read 2026-09-27)
- Apache DataFusion: [`Parquet: Row Group and Data Page pruning on min/max statistics`](https://datafusion.apache.org/user-guide/features.html) (read 2026-09-27)
- Apache DataFusion: [`Parquet: Row Group pruning on Bloom Filters`](https://datafusion.apache.org/user-guide/features.html) (read 2026-09-27)
- Apache DataFusion: [`Predicate push down`](https://datafusion.apache.org/user-guide/features.html) (read 2026-09-27)
- Apache DataFusion: [`Projection push down`](https://datafusion.apache.org/user-guide/features.html) (read 2026-09-27)
- Apache DataFusion: [`Query Optimizer`](https://datafusion.apache.org/user-guide/features.html) (read 2026-09-27)
- Apache DataFusion: [`SQL Parser`](https://datafusion.apache.org/user-guide/features.html) (read 2026-09-27)
- Apache DataFusion: [`SQL Query Planner`](https://datafusion.apache.org/user-guide/features.html) (read 2026-09-27)
- Apache DataFusion: [`Spilling (to disk) Grouping`](https://datafusion.apache.org/user-guide/features.html) (read 2026-09-27)
- Apache DataFusion: [`Spilling (to disk) Sort`](https://datafusion.apache.org/user-guide/features.html) (read 2026-09-27)
- Apache DataFusion: [`Spilling (to disk) Sort Merge Join`](https://datafusion.apache.org/user-guide/features.html) (read 2026-09-27)
- Apache DataFusion: [`Streaming Execution`](https://datafusion.apache.org/user-guide/features.html) (read 2026-09-27)
- Apache DataFusion: [`Streaming Grouping`](https://datafusion.apache.org/user-guide/features.html) (read 2026-09-27)
- Apache DataFusion: [`Streaming Window Evaluation`](https://datafusion.apache.org/user-guide/features.html) (read 2026-09-27)
- Apache DataFusion: [`repartition`](https://docs.rs/datafusion/latest/datafusion/dataframe/struct.DataFrame.html) (read 2026-09-27)
- DuckDB: [`ANALYZE`](https://duckdb.org/docs/current/sql/statements/analyze) (read 2026-09-27)
- DuckDB: [`CHECKPOINT`](https://duckdb.org/docs/current/sql/statements/checkpoint) (read 2026-09-27)
- DuckDB: [`VACUUM`](https://duckdb.org/docs/current/sql/statements/vacuum) (read 2026-09-27)
- DuckDB: `check_peg_parser` (read 2026-09-27)
- DuckDB: `checkpoint` (read 2026-09-27)
- DuckDB: `disable_peg_parser` (read 2026-09-27)
- DuckDB: `duckdb_approx_database_count` (read 2026-09-27)
- DuckDB: `duckdb_connection_count` (read 2026-09-27)
- DuckDB: `duckdb_external_file_cache` (read 2026-09-27)
- DuckDB: `duckdb_memory` (read 2026-09-27)
- DuckDB: `duckdb_temporary_files` (read 2026-09-27)
- DuckDB: `enable_peg_parser` (read 2026-09-27)
- DuckDB: `force_checkpoint` (read 2026-09-27)
- DuckDB: `pragma_database_size` (read 2026-09-27)
- DuckDB: `pragma_metadata_info` (read 2026-09-27)
- DuckDB: `pragma_storage_info` (read 2026-09-27)
- DuckDB: `sql_auto_complete` (read 2026-09-27)
- AWS Glue: [`Autobalance Processing`](https://docs.aws.amazon.com/glue/latest/dg/transforms-autobalance-processing.html) (read 2026-09-27)
- AWS Glue: [`coalesce`](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-extensions-dynamic-frame.html) (read 2026-09-27)
- AWS Glue: [`repartition`](https://docs.aws.amazon.com/glue/latest/dg/aws-glue-api-crawler-pyspark-extensions-dynamic-frame.html) (read 2026-09-27)
- KNIME Analytics Platform: [`Cache`](https://api.hub.knime.com/nodes/*Qi07PtWEA5wwcXn5?details=full) (read 2026-09-27)
- Power Query M: [`Table.Buffer`](https://learn.microsoft.com/en-us/powerquery-m/table-functions#other) (read 2026-09-27)
- subtoken: [`subtoken_cache_clear`](https://github.com/meridian-online/subtoken/blob/ceb61bd796b15b4d54ca3bb2db8a74792d67026e/README.md) (read 2026-09-27)
- Talend: [`tBufferInput`](https://help.qlik.com/talend/en-US/components/8.0/buffer/buffer-component) (read 2026-09-27)
- Talend: [`tBufferOutput`](https://help.qlik.com/talend/en-US/components/8.0/buffer/buffer-component) (read 2026-09-27)
- Talend: [`tCacheClear`](https://help.qlik.com/talend/en-US/components/8.0/processing/processing-component) (read 2026-09-27)
- Talend: [`tCacheIn`](https://help.qlik.com/talend/en-US/components/8.0/processing/processing-component) (read 2026-09-27)
- Talend: [`tCacheOut`](https://help.qlik.com/talend/en-US/components/8.0/processing/processing-component) (read 2026-09-27)
- Talend: [`tHashInput`](https://help.qlik.com/talend/en-US/components/8.0/technical/technical-component) (read 2026-09-27)
- Talend: [`tHashOutput`](https://help.qlik.com/talend/en-US/components/8.0/technical/technical-component) (read 2026-09-27)
- Talend: [`tManagePartitions`](https://help.qlik.com/talend/en-US/components/8.0/processing/processing-component) (read 2026-09-27)
- Talend: [`tPartition (deprecated)`](https://help.qlik.com/talend/en-US/components/8.0/processing/processing-component) (read 2026-09-27)
- Talend: [`tPartitioner`](https://help.qlik.com/talend/en-US/components/8.0/orchestration/orchestration-component) (read 2026-09-27)
- VisiData: [`cache-col`](https://github.com/saulpw/visidata/blob/7609212ec2febafa9caba7183cbebfaf49627909/visidata/features/freeze.py#L77) (read 2026-09-23)
- VisiData: [`cache-cols`](https://github.com/saulpw/visidata/blob/7609212ec2febafa9caba7183cbebfaf49627909/visidata/features/freeze.py#L78) (read 2026-09-23)
- VisiData: [`reload-every`](https://github.com/saulpw/visidata/blob/7609212ec2febafa9caba7183cbebfaf49627909/visidata/features/reload_every.py#L50) (read 2026-09-23)
- VisiData: [`reload-modified`](https://github.com/saulpw/visidata/blob/7609212ec2febafa9caba7183cbebfaf49627909/visidata/features/reload_every.py#L51) (read 2026-09-23)
- VisiData: [`reload-rows`](https://github.com/saulpw/visidata/blob/7609212ec2febafa9caba7183cbebfaf49627909/visidata/features/reload_every.py#L52) (read 2026-09-23)
- VisiData: [`reload-selected`](https://github.com/saulpw/visidata/blob/7609212ec2febafa9caba7183cbebfaf49627909/visidata/indexsheet.py#L99) (read 2026-09-23)
- VisiData: [`reload-sheet`](https://github.com/saulpw/visidata/blob/7609212ec2febafa9caba7183cbebfaf49627909/visidata/sheets.py#L1289) (read 2026-09-23)
- VisiData: [`s3-refresh-sheet`](https://github.com/saulpw/visidata/blob/7609212ec2febafa9caba7183cbebfaf49627909/visidata/loaders/s3.py#L297) (read 2026-09-23)
- VisiData: [`s3-refresh-sheet-all`](https://github.com/saulpw/visidata/blob/7609212ec2febafa9caba7183cbebfaf49627909/visidata/loaders/s3.py#L303) (read 2026-09-23)