Check that a file exists

Status

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Background

When to use it

Apache Hop's File exists action can serve as an integration point with other systems: a process places a file in a well-known location when its task is complete, and the file's existence is the signal that launches the workflow.^[1] Apache Hop's Create file action creates an empty file, which is useful for creating trigger files from within workflows.^[5] In Apache Hop's File exists action, the result False means the file does not exist or cannot be read.^[1] In Azure Data Factory, a Get Metadata activity fails if the object is not found, unless exists is in its field list, in which case it returns exists: false instead.^[6] Decide whether a missing file is an error or simply an empty day, and choose the form of the check that says so: a flag to route on, or a condition that stops the run. (Meridian's view)

A time-of-check to time-of-use (TOCTOU) bug is caused by a race condition between checking the state of part of a system and using the result of that check.^[7] TOCTOU race conditions are common in Unix between operations on the file system.^[7] One general technique against TOCTOU races is to use error handling instead of pre-checking, under the philosophy of easier to ask for forgiveness than permission (EAFP).^[7] Use an existence check to decide whether to start, and still handle the error when the file has gone or changed by the time it is read. (Meridian's view)

Apache Hop's Check files locked action tests each file by renaming it to the same name as before, an operation that fails on certain operating systems, MS Windows in particular, if the file is locked.^[8] Unix-like operating systems, including Linux and macOS, do not normally lock open files automatically, and file locks under Unix are by default advisory.^[9] Under advisory locking, cooperating processes may use locks to coordinate access to a file, but uncooperative processes are free to ignore the locks.^[9] Treat a passed lock check as weak evidence on Linux or macOS, and prefer a completion marker that the writer creates last. (Meridian's view)

The method

Apache Hop's Checks if files exist action returns true if all the listed files exist and false if any of them do not.^[10] When a path given to Apache Hop's Checks if files exist action is a folder with a regular-expression wildcard, at least one file matching the pattern must exist.^[10] Apache Hop's Check if a folder is empty action verifies that there are no regular files or sub-folders in the folder.^[3] Apache Hop's File Compare action compares the contents of two files and returns true if the files are equal.^[4]

Unlike DuckDB's read functions, DuckDB's glob function returns an empty result instead of raising an error when nothing matches, so it can check whether any files match before a read.^[11]

Sources

1. [1]File Existsdocumentation
2. [2]File existsdocumentation
3. [3]Check if a folder is emptydocumentation
4. [4]File Comparedocumentation
5. [5]Create Filedocumentation
6. [6]Get Metadata activity - Azure Data Factory & Azure Synapsedocumentation
7. [7]Time-of-check to time-of-use - Wikipediasecondary source
8. [8]Check Files Lockeddocumentation
9. [9]File locking - Wikipediasecondary source
10. [10]Checks if files existdocumentation
11. [11]Reading Multiple Filesdocumentation