Acquire / filesystem · All operations

List files and folders

List directories, file metadata, file exists.

Status

In arcform

Runs as a SQL step

arcform carries it as a sql: step today.

Shape of the result
a new table, from no input table
Level
one clause of SQL

The DuckDB step

List files by glob

It reads the orders table the proofs share. Create it first:

SQL
CREATE TABLE orders AS SELECT * FROM (VALUES
  (1,  'Alice Ng',      'north', DATE '2026-01-03', 120.00, 'widget', 'a|b',   '{"channel":"web","items":[1,2]}'),
  (2,  'alice ng',      'north', DATE '2026-01-05',  80.50, 'gadget', 'b',     '{"channel":"shop","items":[3]}'),
  (3,  'Bob  Okafor',   'south', DATE '2026-01-05', 300.00, 'widget', 'a|c',   '{"channel":"web","items":[4,5,6]}'),
  (4,  'Carla Díaz',    'east',  DATE '2026-01-09',  NULL,  'gizmo',  NULL,    '{"channel":"web","items":[]}'),
  (5,  'Bob Okafor',    'south', DATE '2026-01-12',  45.25, 'gadget', 'c',     '{"channel":"phone","items":[7]}'),
  (6,  'Dev Patel',     'west',  DATE '2026-01-12', 510.00, 'widget', 'a',     '{"channel":"web","items":[8,9]}'),
  (7,  'Carla Diaz',    'east',  DATE '2026-01-20',  99.99, 'gizmo',  'b|c',   '{"channel":"shop","items":[10]}'),
  (8,  'Dev Patel',     'west',  DATE '2026-02-02', 510.00, 'widget', 'a',     '{"channel":"web","items":[8,9]}'),
  (9,  'Eve Larsen',    NULL,    DATE '2026-02-03',  15.00, 'gadget', NULL,    '{"channel":"web","items":[11]}'),
  (10, 'Alice Ng',      'north', DATE '2026-02-10', 220.00, 'gizmo',  'a|b|c', '{"channel":"phone","items":[12,13]}')
) t(order_id, customer, region, order_date, amount, product, tags, meta);

Then run the statement:

SQL
COPY orders TO '<your_scratch_dir>/a.csv'; COPY orders TO '<your_scratch_dir>/b.csv';
CREATE TEMP TABLE result AS
SELECT count(*) AS files FROM glob('<your_scratch_dir>/?.csv');
SELECT * FROM result;
SELECT CASE WHEN ((SELECT files = 2 FROM result)) THEN 'as expected' ELSE error('the glob did not find the two files written') END AS verdict;

Replace <your_scratch_dir> with a directory of your own that you can write to before running the statement.

  • The statement writes a.csv and b.csv itself, in <your_scratch_dir>.
  • It needs nothing installed beyond DuckDB itself.
  • The statement checks its own result and raises an error when it is wrong: “the glob did not find the two files written”.
  • It passed on DuckDB v2.0.0 (Cyanoptera) d54cbbe3df.

Variants

Connect to a file system

Open a connection to a local, mounted or test file system for later file steps to use.

List the files in a folder

Return one row per file or subfolder in a directory, optionally matching a pattern, with size and dates.

Read a file's raw content

Load the bytes of a file, or files picked from a folder, into the flow without parsing them as a table.

Read system and repository information

Open process, memory, network, package or git repository details as rows.

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Alteryx Designer (1)
  • Directoryread 2026-09-27

    List the files in a directory with size, created and modified dates

Apache Hop (21)
  • Check files lockedread 2026-09-27

    Succeed only when none of the listed files is locked by another process

  • Check if a folder is emptyread 2026-09-27

    Succeed when a folder holds no files, optionally by wildcard

  • Check if file is lockedread 2026-09-27

    Flag whether the file named in each row is locked by another process

  • Checks if files existsread 2026-09-27

    Succeed only when every listed file exists

  • Compare foldersread 2026-09-27

    Compare the contents of two folders and succeed when identical

  • Copy Filesread 2026-09-27

    Copy files or folders to another location, by wildcard

  • Create a folderread 2026-09-27

    Create a folder

  • Create fileread 2026-09-27

    Create an empty file, for example as a trigger file

  • Delete fileread 2026-09-27

    Delete one file

  • Delete filesread 2026-09-27

    Delete a set of files by folder and wildcard

  • Delete foldersread 2026-09-27

    Delete folders together with their contents

  • Evaluate files metricsread 2026-09-27

    Test the total size or count of a set of files against a threshold

  • File compareread 2026-09-27

    Compare the contents of two files and succeed when equal

  • File existsread 2026-09-27

    Flag whether the file named in each row exists

  • File existsread 2026-09-27

    Succeed when a named file exists

  • Get SubFolder namesread 2026-09-27

    List the subfolders of a directory

  • Get filenamesread 2026-09-27

    List files matching a path and wildcard with size, dates and attributes

  • Get files rowcountread 2026-09-27

    Count the rows in one or more text files

  • Move filesread 2026-09-27

    Move files or folders to another location

  • Process filesread 2026-09-27

    Copy, move or delete the files named by source and target fields

  • Process result filenamesread 2026-09-27

    Copy, move or delete the files held in the result file list

Apache NiFi (6)
  • DeleteFileread 2026-09-27

    Delete a file from the local file system

  • FetchFileread 2026-09-27

    Read a file from local disk into the content of an incoming FlowFile

  • GetFileread 2026-09-27

    Read files from a local directory into new FlowFiles

  • GetFileResourceread 2026-09-27

    Create FlowFiles from the content of a configured file resource

  • ListFileread 2026-09-27

    List files in a local directory, one FlowFile per file, for FetchFile to read

  • Object and file store delete, copy, move and tag processors (family)read 2026-09-27

    10 processors, DeleteS3Object, CopyS3Object, TagS3Object, DeleteGCSObject, MoveAzureDataLakeStorage, DeleteSFTP, …

Azure Data Factory (2)
  • Delete Activityread 2026-09-27

    Delete files or folders from on-premises or cloud storage

  • Get Metadataread 2026-09-27

    Retrieve metadata of data, such as of a file, folder or table, for later activities to use

DuckDB (1)
  • globread 2026-09-27

    glob table function

Duckle (4)
  • Artifactsread 2026-09-25

    one row per file with uri, media type, size, sha256 and modified time via read_blob

  • File Listread 2026-09-25

    one row per file in a directory via glob()

  • File Operationread 2026-09-25

    copy, move or delete one file

  • Git Repositoryread 2026-09-25

    read the commit log or file tree of a local git working copy

EasyMorph (4)
  • File commandread 2026-09-27

    Copy, move, rename, delete or unzip files

  • List of filesread 2026-09-27

    List the files of a folder as rows

  • List of foldersread 2026-09-27

    List the subfolders of a folder as rows

  • Split delimited fileread 2026-09-27

    Split a large text file into chunk files by row count or by a column's values, without loading it

KNIME Analytics Platform (16)
OpenRefine (1)
  • Get data from this computerread 2026-09-27

    Load one or more local files into a project; archives are unpacked and their files offered for selection

Pentaho Data Integration (6)
Power Query M (3)
  • File.Contentsread 2026-09-27

    Read a file at a path as a binary value

  • Folder.Contentsread 2026-09-27

    List the files and folders directly inside a folder, with properties and a link to content

  • Folder.Filesread 2026-09-27

    List every file in a folder and its subfolders, with properties and a link to content

RapidMiner Studio (16)
Talend (9)
  • tCreateTemporaryFileread 2026-09-27

    Create a temporary file in a given directory, kept or deleted after the Job

  • tFileCompareread 2026-09-27

    Compare two files and output the comparison result

  • tFileCopyread 2026-09-27

    Copy a file or folder into a target directory

  • tFileDeleteread 2026-09-27

    Delete files from a directory

  • tFileExistread 2026-09-27

    Check whether a file exists

  • tFileListread 2026-09-27

    Iterate over the files or folders of a directory that match a file mask

  • tFilePropertiesread 2026-09-27

    Output one row holding the main properties of a file

  • tFileRowCountread 2026-09-27

    Count the rows of a file by reading it row by row

  • tFileTouchread 2026-09-27

    Create an empty file, or update the modification and access dates of an existing one

VisiData (10)
  • dir (loader)read 2026-09-23

    Load a directory as a Directory Sheet

  • fdir (loader)read 2026-09-23

    Load a list of file paths as a FileListSheet

  • open-cpustatsread 2026-09-23

    Open sheet of CPU usage statistics

  • open-dir-currentread 2026-09-23

    Open Directory Sheet listing files in the current directory

  • open-dir-parentread 2026-09-23

    Open Directory Sheet for the parent directory

  • open-memstatsread 2026-09-23

    Open sheet of memory usage statistics

  • open-pingread 2026-09-23

    Open sheet of ping statistics for a given host

  • open-processesread 2026-09-23

    Open sheet of system processes

  • open-python-packagesread 2026-09-23

    Open sheet of installed Python packages

  • open-rlimitsread 2026-09-23

    Open resource limits of the process in current row

Background

In DuckDB, the glob table function returns the filenames found at a search path, in a single column named file.[1] In DuckDB, the search path given to glob may contain glob pattern matching syntax, such as * for any characters and ** for any number of subdirectories.[1][2] Globbing, in its original meaning, expands a name pattern into the list of names in a file system directory that match it.[3]

When to use it

In DuckDB, if a glob pattern given to a read function such as read_parquet matches no files, the read raises an error.[2] Starting with DuckDB 2.0, the CSV, JSON and Parquet readers accept an allow_empty option, which returns an empty result instead of raising that error.[2] List the folder as its own step when an empty delivery is a normal day, so an empty folder reads as zero files rather than a failed run. (Meridian's view)

Traditionally, globs do not match hidden files in the form of Unix dotfiles, and a pattern must start with a dot to match them.[3] Normally, a glob never matches the path separator character, / on Linux, Unix and macOS or \ on Windows.[3]

Apache NiFi's ListFile processor stores the timestamp of the newest file after each listing, so its next run lists only files added or modified after that date.[4] Apache NiFi's ListFile has a Minimum File Age property, and any file younger than that age, by its last modification date, is ignored.[4] Treat a listing as a snapshot of the folder at one moment, and leave files that changed in the last few minutes for the next run when writers may still be at work. (Meridian's view)

The method

The ls shell command, which lists files including special files such as directories, appeared in the first version of AT&T UNIX.[5] The name ls was inherited from Multics and is short for list.[5] The long output format of ls, selected with -l, adds each file's type, permissions, hard link count, owning user and group, size and last-modified timestamp.[5]

The glob command, short for global, originates in the earliest versions of Bell Labs' Unix, where a separate program, /etc/glob, expanded wildcard characters in unquoted arguments.[3] Glob was originally written in the B programming language, and was the first piece of mainline Unix software to be developed in a high-level programming language.[3] The Portable Operating System Interface (POSIX) is a family of standards for maintaining compatibility between operating systems.[6] Today, globbing is standardised by the POSIX.2 specification.[3]

In DuckDB's glob syntax, * matches any number of any characters, including none, and ** matches any number of subdirectories, including none.[2] In DuckDB's glob syntax, ? matches any single character, [abc] matches one character given in the brackets, and [a-z] matches one character from the range.[2] DuckDB's read_text and read_blob functions accept a filename, a list of filenames or a glob pattern, and output the content of each file with metadata such as its size and last modified time.[7]

Compared with the alternatives

Standard SQL's LIKE operator uses a glob-like syntax, in which the percent sign matches zero or more characters and the underscore matches exactly one.[3] In Apache Hop, the Get filenames transform selects files with regular expressions, such as .*\.dat$ for files with an extension of .dat.[8]

DuckDB can read multiple CSV, Parquet and JSON files at the same time, using either the glob syntax or a list of files to read.[2] In DuckDB, the filename argument of a read function adds a column that indicates which file each row came from.[2] Reach for a listing when the files themselves are the subject, such as which arrived and how large they are, and read through a glob when only their rows matter. (Meridian's view)

Sources

  1. [1]Utility Functionsdocumentation
  2. [2]Reading Multiple Filesdocumentation
  3. [3]glob (programming) - Wikipediasecondary source
  4. [4]ListFile - Apache NiFidocumentation
  5. [5]ls - Wikipediasecondary source
  6. [6]POSIX - Wikipediasecondary source
  7. [7]Directly Reading Filesdocumentation
  8. [8]Get filenamesdocumentation