Download files to a local path

Status

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Background

When to use it

For remote CSV files, DuckDB downloads the whole file in most cases, because of the format's row-based nature.^[7] For Parquet files, DuckDB can use the Parquet metadata and range requests to download only the parts of the file a query actually needs.^[7] Download first when a row-based file will be read more than once, or when a later step reads only local paths; leave a columnar file where it is when a query needs few of its columns. (Meridian's view)

EasyMorph's Download folder setting overwrites files with the same names that already exist in the local folder.^[3] For a single file, EasyMorph lets the user choose to overwrite an existing file of the same name, do nothing, or halt project execution.^[3] Apache Hop's Get a file with FTP action can add the system date and the system time to each retrieved file's name.^[1] Decide before the first run what happens to a local file of the same name, because the folder should say which download each file came from. (Meridian's view)

Apache Hop's Get a file with SFTP action returns true when the files were retrieved without errors, including when no file matched.^[8] After a download step, check that it fetched the files expected, not only that it succeeded. (Meridian's view)

Apache Hop's Get a file with SFTP action can delete each remote file after it is downloaded, an operation that usually cannot be undone.^[8] Apache Hop's Get a file with FTP action removes or moves the files on the server only after all selected files have been successfully transferred.^[1] Keep the remote copy until the local one has been checked, unless the server is a drop folder whose files are meant to be taken once. (Meridian's view)

The method

Apache Hop's Get a file with FTP action does not crawl a server, and retrieves files from one remote directory exclusively.^[1] In Apache Hop's FTP and SFTP actions, a wildcard written as a regular expression selects multiple files to retrieve.^[1][8] Apache Hop's Get a file with SFTP action can preserve each source file's last modified timestamp; otherwise the downloaded file is stamped with the moment of download.^[8]

In FTP's image type, commonly called binary mode, the sending machine sends each file byte by byte and the recipient stores the bytestream as it receives it.^[2] FTP's ASCII type converts text between the two hosts' character representations, newlines included, which makes it inappropriate for files that contain data other than ASCII.^[2] Transfer data files in binary mode, so that the bytes that arrive are the bytes that were sent. (Meridian's view)

Compared with the alternatives

DuckDB's httpfs extension can query files at web addresses directly, with read-only access, for all files supported by DuckDB or its extensions.^[7] Reading in place keeps one copy of the data and no local folder to manage; downloading gives later steps a fixed copy that does not change under them, and serves steps that read only local paths. (Meridian's view)

Sources

1. [1]Get a file with FTP | Apache Hopdocumentation
2. [2]File Transfer Protocol - Wikipediasecondary source
3. [3]AMAZON S3 COMMAND [EasyMorph Help]documentation
4. [4]Amazon S3 - Wikipediasecondary source
5. [5]Read SFTP - Rapidminer Documentationdocumentation
6. [6]SSH File Transfer Protocol - Wikipediasecondary source
7. [7]HTTP(S) Support – DuckDBdocumentation
8. [8]Get a file with SFTP | Apache Hopdocumentation