Upload files to a remote store

Status

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Background

When to use it

Reach for a file upload when the file already exists and must arrive unchanged, such as a report, an archive or a model an earlier step built. (Meridian's view)

FTP does not encrypt its traffic, so usernames, passwords, commands and data can be read by anyone able to capture packets on the network.^[2] The SSH File Transfer Protocol (SFTP) uses the Secure Shell protocol (SSH) to transfer files and, unlike FTP, encrypts both commands and data.^[2] Prefer SFTP to plain FTP for any upload that carries credentials or private data. (Meridian's view)

In Apache Hop, the Put A File With FTP action can select the files to upload from a local directory with a regular expression, and can skip files that already exist in the remote directory.^[3] Apache Hop's Put A File With FTP action has an option to remove files from the local directory after they have been uploaded.^[3] Delete or move the local copy only once the upload is known to have succeeded. (Meridian's view)

In Apache Hop, the Put a file with SFTP action fails the workflow when there are no files to process, unless its Success when no file option is checked.^[4] Apache Hop's Put a file with SFTP action can send the files an earlier pipeline or workflow action generated, and ignores any of those files it cannot find.^[4] Check what arrived on the server, by name or by count, rather than trusting the step's success alone. (Meridian's view)

The method

The original specification for FTP was written by Abhay Bhushan and published on 16 April 1971.^[2] SFTP was designed by the Internet Engineering Task Force (IETF) as an extension of SSH version 2.0, and is seen as a replacement of FTP due to superior security.^[5] SFTP is not FTP run over SSH, but a new protocol designed from the ground up by an IETF working group, SECSH.^[5] An SFTP client's capabilities include resuming interrupted transfers, directory listings and remote file removal.^[5]

In RapidMiner, the file the Write SFTP operator uploads is given as a file object, which operators such as Write CSV or Write File create.^[6] Apache Hop's SFTP Put transform uploads a file, or the content of a field, to an SFTP server once per incoming row, taking the file, the folder and the remote name from fields.^[7] In RapidMiner, setting the Write SFTP operator's overwrite parameter to true overwrites a file of the same name in the remote directory on the server.^[6]

In Amazon Simple Storage Service (Amazon S3), a single operation uploads an object of up to 5 GB, and multipart upload uploads a single object of up to 50 TB in parts.^[8] A large file already on disk is better sent to Amazon S3 in parts than in one operation. (Meridian's view)

Compared with the alternatives

Uploading contrasts with downloading, where data is received over a network.^[1] Moving files within a computer system, as opposed to over a network, is called file copying.^[1] Compared with the Secure Copy Protocol (SCP), which only allows file transfers, SFTP allows a range of operations on remote files that make it more like a remote file system protocol.^[5] Upload a file when its exact bytes matter; write the pipeline's rows instead when the format is yours to choose. (Meridian's view)

Sources

1. [1]Upload - Wikipediasecondary source
2. [2]File Transfer Protocol - Wikipediasecondary source
3. [3]Put a file with FTP | Apache Hopdocumentation
4. [4]Put a file with SFTP | Apache Hopdocumentation
5. [5]SSH File Transfer Protocol - Wikipediasecondary source
6. [6]Write SFTP - Rapidminer Documentationdocumentation
7. [7]SFTP Put | Apache Hopdocumentation
8. [8]Uploading objects - Amazon Simple Storage Servicedocumentation