Acquire / http · All operations

Call a web API for each row

A variant of Fetch from the web

Send one HTTP request per input row, built from its values, and add the response to that row.

Status

In arcform

Not built yet

arcform does not do this yet. Its place on the roadmap

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

Apache Hop (3)
  • HTTP Postread 2026-09-27

    Send an HTTP POST per row with body and parameters and return the response

  • HTTP clientread 2026-09-27

    Call a URL with HTTP GET per row and return the response body, status and headers

  • REST Clientread 2026-09-27

    Call a REST service per row with any method, headers and body, returning the response

Azure Data Factory (1)
  • External callread 2026-09-27

    Call an external endpoint row by row and add the response to the row

EasyMorph (1)
  • Iterate Web Requestread 2026-09-27

    Send one HTTP request per row to a web API and append the responses into one table

KNIME Analytics Platform (6)
  • DELETE Requestread 2026-09-27

    Send an HTTP DELETE per row and append the response

  • GET Requestread 2026-09-27

    Send an HTTP GET per row and append the response

  • HEAD Requestread 2026-09-27

    Send an HTTP HEAD per row and append the response headers

  • PATCH Requestread 2026-09-27

    Send an HTTP PATCH with a body per row and append the response

  • POST Requestread 2026-09-27

    Send an HTTP POST with a body per row and append the response

  • PUT Requestread 2026-09-27

    Send an HTTP PUT with a body per row and append the response

OpenRefine (1)
Pentaho Data Integration (3)
  • HTTP Postread 2026-09-27

    Send an HTTP POST request to a web service with parameters set per row

  • HTTP clientread 2026-09-27

    Call a web service over HTTP from a base URL with parameters set per row

  • REST Clientread 2026-09-27

    Call RESTful web services identified by URL

RapidMiner Studio (2)
  • Enrich Data by Webserviceread 2026-09-27

    Call a web service per row and attach values extracted by regex, XPath or string matching

  • Get Pagesread 2026-09-27

    Fetch the pages for the URLs in a column and store the content in a new column

Talend (1)

Background

In OpenRefine, Add column by fetching URLs builds a uniform resource locator (URL) from each cell of a column and fetches the page or data it points to.[1][2] Azure Data Factory's external call transformation calls Representational State Transfer (REST) endpoints row by row to add custom results to a data flow.[3][4] KNIME's GET Request node can send one request per URL listed in an input table, and each request gives one row of the output table.[5]

When to use it

OpenRefine's Add column by fetching URLs is more useful for fetching metadata files than web pages, though a page's entire contents can be fetched and parsed afterwards.[1] In OpenRefine, fetching URLs can take quite some time, and servers may not want to fulfil hundreds or thousands of page requests in seconds.[1] OpenRefine's throttle delay sets the time between requests, and defaults to 5 seconds per row of the dataset.[1] Multiply the number of rows by the delay before starting, because a call per row over a large table can run for hours. (Meridian's view) Think of a call per row as a lookup against a web service: each input row keeps its columns and gains the response, where a single request reads a table from the service. (Meridian's view)

KNIME's GET Request node can either fail or pause for a fixed cooldown and retry when it receives rate-limiting errors with status 429.[5] KNIME's GET Request node can insert a missing value for a request that fails with a client-side (4XX) or server-side (5XX) error, instead of failing the node.[5] Keep each row's status code and error message beside its response, so the rows whose call failed can be found and fetched again rather than left quietly empty. (Meridian's view)

In OpenRefine, authorisation credentials set as request headers for fetching URLs are logged in the operation history in plain text.[1] Keep access keys out of a call that records its settings in plain text, and pass them through the tool's credential store where it has one. (Meridian's view)

The method

EasyMorph's Iterate Web Request action can append the responses into one dataset or save them as files in a designated folder.[6] In Azure Data Factory's external call transformation, the body, headers and status returned by the call can each be stored in a named column for downstream transformations.[3]

KNIME's GET Request node sets a number of concurrent requests, and can add a delay between consecutive requests to avoid overloading the web service.[5] KNIME's GET Request node can retry server-side errors automatically, with a delay that increases exponentially with each retry attempt.[5] Start with one request at a time and a delay, and raise the concurrency only when the service's documentation allows it. (Meridian's view)

Sources

  1. [1]Column editing | OpenRefinedocumentation
  2. [2]URL - Wikipediasecondary source
  3. [3]External call data transformation in mapping data flows - Azure Data Factory | Microsoft Learndocumentation
  4. [4]REST - Wikipediasecondary source
  5. [5]GET Requestdocumentation
  6. [6]ITERATE WEB REQUEST [EasyMorph Help]documentation