Control / flow · All operations

Wait for a file, table or signal

A variant of Branch or loop

Hold the run until a file, table, query result or release signal appears, or a time-out passes.

Status

In arcform

Not built yet

arcform does not do this yet; it would live in how a Protocol runs. Its place on the roadmap

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Apache Hop (2)
  • Wait for SQLread 2026-09-27

    Poll a SQL query until its row count meets a condition or a timeout passes

  • Wait for fileread 2026-09-27

    Wait until a file appears, checking at intervals up to a timeout

Apache NiFi (2)
  • Notifyread 2026-09-27

    Cache a release signal so FlowFiles held at the matching Wait processor are released

  • Waitread 2026-09-27

    Hold FlowFiles until a matching release signal from Notify appears in the cache

Azure Data Factory (1)
  • Validation Activityread 2026-09-27

    Continue only once a reference dataset exists or meets criteria, or a time-out passes

Bruin (5)
  • Query sensors (family)read 2026-09-27

    18 platform types (bq.sensor.query, …): wait until a query returns a result

  • Table sensors (family)read 2026-09-27

    14 platform types (bq.sensor.table, …): wait until a named table exists

  • gcs.sensor.objectread 2026-09-27

    Wait until an object matching a name or wildcard exists in a GCS bucket

  • gcs.sensor.prefixread 2026-09-27

    Wait until any object exists under a prefix in a GCS bucket

  • s3.sensor.key_sensorread 2026-09-27

    Wait until an S3 key or wildcard match exists, optionally checking etag, size or modified time

SQLMesh (1)
  • Signalsread 2026-09-27

    Python functions that decide whether a model interval is ready for the scheduler to evaluate

Talend (3)
  • tWaitForFileread 2026-09-27

    Watch a directory and trigger the next component when a file event occurs

  • tWaitForSocketread 2026-09-27

    Listen on a port and trigger a subJob when data arrives

  • tWaitForSqlDataread 2026-09-27

    Poll a database and trigger a subJob when a condition on the presence of rows is met

Background

In Apache Hop, the Wait for file action sleeps and periodically checks whether a file exists, after which the workflow continues.[1] Azure Data Factory's Validation activity lets a pipeline continue only once a dataset it references exists and meets set criteria, or a time-out passes.[2] In Bruin, a sensor waits until a certain external event has occurred, most commonly until data that an external system produces is available.[3]

When to use it

A timeout is a period allowed to elapse before a specified event takes place, unless another specified event occurs first.[4] By default a Bruin sensor gives up after 24 hours and fails.[3] An Azure Data Factory Validation activity times out after 12 hours by default, and waits 10 seconds between attempts unless set otherwise.[2] Apache Hop's Wait for file can wait indefinitely or time out, and its Success on timeout option decides whether the workflow then follows its success or failure path.[1]

Apache Hop's Wait for file can also check that the file's size has not changed since the last check, for when another process is still writing the file in its final place.[1] An Azure Data Factory Validation activity can require a minimum file size in bytes, which is 0 unless set.[2] Ask whoever writes the file to write it under another name and move it into place when complete, so that its arrival means it is whole. (Meridian's view)

The method

Polling can mean checking a device repeatedly for readiness and, when it is not ready, returning to a different task.[5] Polling that returns to other work in between is not as wasteful of processor cycles as busy waiting, in which the computer does nothing but check until the device is ready.[5] Bruin's sensors work as a poking mechanism, checking the status of the external event every few minutes until the event has occurred.[3] In Apache Hop's Wait for file, a check cycle time of 30 or 60 seconds seems to be a good trade-off between how soon the file is detected and the processor time it needs.[1]

Apache NiFi's Wait processor routes incoming FlowFiles, NiFi's single pieces of data, to its wait relationship until a matching release signal from a Notify processor is stored in a cache, then routes a waiting FlowFile to success.[6][7] FlowFiles held by NiFi's Wait processor are routed to expired if they wait longer than the Expiration Duration.[6] NiFi's Wait processor can wait for more than one signal, such as one for each fragment a SplitText processor produced, through its Target Signal Count property.[6]

Sources

  1. [1]Wait for file | Apache Hopdocumentation
  2. [2]Validation activity - Azure Data Factory & Azure Synapse | Microsoft Learndocumentation
  3. [3]Sensor Assets | Bruin CLIdocumentation
  4. [4]Timeout (computing) - Wikipediasecondary source
  5. [5]Polling (computer science) - Wikipediasecondary source
  6. [6]Wait - Apache NiFidocumentation
  7. [7]Apache NiFi User Guidedocumentation