Control / engine · All operations

Cache an intermediate result

A variant of Plan and run queries

Hold a table or column in memory, or in a stored copy, so later steps read it without recomputing it.

Status

In arcform

Not built yet

arcform does not do this yet; it would live in how a Protocol runs. Its place on the roadmap

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

KNIME Analytics Platform (1)
  • Cacheread 2026-09-27

    Materialise the input table so downstream nodes read a stored copy

Power Query M (1)
  • Table.Bufferread 2026-09-27

    Load the table into memory once so later steps see a stable copy

subtoken (1)
Talend (7)
  • tBufferInputread 2026-09-27

    Read rows that tBufferOutput buffered, for processing in another subJob

  • tBufferOutputread 2026-09-27

    Collect rows in a buffer for later access, for example when the Job is exposed as a web service

  • tCacheClearread 2026-09-27

    Remove from memory a Spark RDD that tCacheOut persisted

  • tCacheInread 2026-09-27

    Read the Spark RDDs that tCacheOut persisted

  • tCacheOutread 2026-09-27

    Persist the input Spark RDDs at a chosen storage level for later use in the same Job

  • tHashInputread 2026-09-27

    Read the rows that tHashOutput loaded into cache memory

  • tHashOutputread 2026-09-27

    Load rows into cache memory for fast access later in the Job

VisiData (2)
  • cache-colread 2026-09-23

    Add or reset the value cache of current column

  • cache-colsread 2026-09-23

    Add or reset the value cache of all visible columns

Background

A cache stores data so that future requests for it are served faster, and the data it stores might be the result of an earlier computation.[1] In Apache Spark, persisting or caching a dataset keeps it in memory across operations, which can make later actions on it much faster.[2] The Cache node in KNIME Analytics Platform materialises its input table into a self-contained table that consolidates all columns.[3]

When to use it

In Apache Spark, each transformed resilient distributed dataset (RDD) may by default be recomputed each time an action runs on it, unless it is persisted with the persist or cache method.[2] In Apache Spark, caching is a key tool for iterative algorithms and fast interactive use.[2]

A cached copy may become out of date, or stale, when something other than the cache changes the data it was copied from.[1] In Power Query M, Table.Buffer buffers a table in memory, isolating it from external changes during evaluation.[4] In Power Query M, after a SQL table is loaded into memory with Table.Buffer, later operations are no longer able to query the SQL server.[4] In Power Query M, Table.Buffer might or might not make queries run faster, and can slow them through the added cost of reading all the data and storing it in memory.[4] Say in the pipeline where each cached copy is taken, and clear or rebuild it whenever its source changes, so no step reads an older copy than it expects. (Meridian's view)

The Cache node in KNIME Analytics Platform also lets data be staged at chosen points of a workflow run in streaming mode, giving a snapshot for inspection and debugging.[3] Apache Spark monitors cache use on each node and drops old cached data in least-recently-used order, and its unpersist method removes a dataset from the cache by hand.[2]

The method

A cache hit occurs when requested data is found in a cache and a cache miss when it is not; a hit is served faster than recomputing a result.[1]

When an RDD is persisted in Apache Spark, each node stores in memory the parts of it that it computes and reuses them in other actions on that dataset.[2] Apache Spark's cache is fault-tolerant: if any part of a persisted RDD is lost, it is recomputed using the transformations that originally created it.[2] Apache Spark's storage levels let a persisted RDD be kept on disk, in memory as serialised objects to save space, or replicated across nodes.[2] For Apache Spark, spilling a cached dataset to disk is advised against unless the functions that computed it are expensive or filter a large amount of data, since recomputing may be as fast as reading from disk.[2]

In KNIME Analytics Platform, a chain of transformation nodes stores only the modified columns and references the input for the rest, which can leave a composite of nested tables that is less efficient to iterate over.[3] In DuckDB, a temporary table is visible only to the connection that created it and is dropped automatically when that connection closes.[5] In DuckDB, temporary tables reside in memory even in a persistent database, and spill to disk when memory becomes constrained if a temporary directory is configured.[5]

Compared with the alternatives

A materialised view also caches the result of a query as a concrete table, which may be updated from the base tables from time to time, at the cost of extra storage and possibly out-of-date data.[6] An ordinary view is a virtual table: a query against it is converted into queries against the underlying base tables.[6] A cached intermediate result is closest to a short-lived materialised view: a stored copy made for the steps that follow, rather than a database object kept for other users. (Meridian's view)

Sources

  1. [1]Cache (computing) - Wikipediasecondary source
  2. [2]RDD Programming Guide - Spark 4.3.0 Documentationdocumentation
  3. [3]Cachedocumentation
  4. [4]Table.Buffer - PowerQuery M | Microsoft Learndocumentation
  5. [5]CREATE TABLE Statement – DuckDBdocumentation
  6. [6]Materialized view - Wikipediasecondary source