Acquire / http · All operations

Fetch from the web

HTTP GET or POST, REST, SOAP, GraphQL, web scraping.

Status

In arcform

Not built yet

arcform does not do this yet. Its place on the roadmap

arcform has http_fetch for downloading a file from a URL.

arcform has html_link_discover for collecting links from an index page.

Shape of the result
a new table, from no input table
Level
needs an arcform operator

Variants

Download a file from a URL

Fetch a file from a web address, saving it or reading it as a table.

Call a REST API

Send an HTTP request to a web API, with headers, body and pagination, and turn the response into rows.

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Alteryx Designer (1)
  • Downloadread 2026-09-27

    Fetch data from a URL, or transfer by FTP and SFTP, into a field or a file

Apache Hop (6)
  • HTTPread 2026-09-27

    Download a file from a web server over HTTP, or upload one

  • HTTP Postread 2026-09-27

    Send an HTTP POST per row with body and parameters and return the response

  • HTTP clientread 2026-09-27

    Call a URL with HTTP GET per row and return the response body, status and headers

  • OData inputread 2026-09-27

    Read entities from OData V2 and V4 service endpoints

  • REST Clientread 2026-09-27

    Call a REST service per row with any method, headers and body, returning the response

  • Web services lookupread 2026-09-27

    Call a SOAP web service described by WSDL per row and return the result

Apache NiFi (3)
  • HandleHttpRequestread 2026-09-27

    Run an HTTP server and create a FlowFile per request; pairs with HandleHttpResponse

  • InvokeHTTPread 2026-09-27

    HTTP client: call a configurable endpoint and method; content is the body for PUT, POST, PATCH

  • ListenHTTPread 2026-09-27

    Run an HTTP server on a base path and turn incoming requests into FlowFiles

AWS Glue (1)
  • REST APIread 2026-09-27

    Connection type for a REST API, set up by the user

Azure Data Factory (3)
Bruin (1)
Duckle (6)
  • GraphQLread 2026-09-25

    POST a GraphQL query and walk the response data path

  • HTTPread 2026-09-25

    read CSV, Parquet or JSON from an HTTP(S) URL

  • ODataread 2026-09-25

    OData v4 source; a REST alias following @odata.nextLink

  • RESTread 2026-09-25

    generic HTTP GET/POST source with JSON pointer row path, pagination and cursor state

  • SOAPread 2026-09-25

    SOAP or XML API source; a REST alias with XML response parsing

  • Webhookread 2026-09-25

    listen on a local port and collect inbound HTTP requests as rows

EasyMorph (6)
  • Download fileread 2026-09-27

    Download a file from the internet

  • Import from ODataread 2026-09-27

    Read from Microsoft Graph or another OData version 3 or 4 provider

  • Import from Web APIread 2026-09-27

    Read data from a web API

  • Incoming HTTP Requestread 2026-09-27

    Receive the HTTP request sent to a custom API endpoint hosted by EasyMorph Hub

  • Iterate Web Requestread 2026-09-27

    Send one HTTP request per row to a web API and append the responses into one table

  • Web Requestread 2026-09-27

    Send an HTTP request to a web API and receive the response

KNIME Analytics Platform (10)
  • Container Input (Raw HTTP)read 2026-09-27

    Read body, headers and query string of the HTTP request that called the workflow

  • DELETE Requestread 2026-09-27

    Send an HTTP DELETE per row and append the response

  • GET Requestread 2026-09-27

    Send an HTTP GET per row and append the response

  • HEAD Requestread 2026-09-27

    Send an HTTP HEAD per row and append the response headers

  • KNIME Web Interaction (Labs) (extension)read 2026-09-27

    Extension outside the base install: 9 current nodes: Clicker (Labs); Content Retriever (Labs); Navigator (Labs); Screenshotter (Labs), …

  • KNIME Web Service Client (extension)read 2026-09-27

    Extension outside the base install: 1 current node: Generic Web Service Client

  • PATCH Requestread 2026-09-27

    Send an HTTP PATCH with a body per row and append the response

  • POST Requestread 2026-09-27

    Send an HTTP POST with a body per row and append the response

  • PUT Requestread 2026-09-27

    Send an HTTP PUT with a body per row and append the response

  • Webpage Retrieverread 2026-09-27

    Fetch an HTML page by HTTP GET and output it cleaned as XHTML or a string

OpenRefine (2)
Pentaho Data Integration (6)
  • Call Endpointread 2026-09-27

    Call a Pentaho Server API endpoint from inside a transformation

  • HTTP Postread 2026-09-27

    Send an HTTP POST request to a web service with parameters set per row

  • HTTP clientread 2026-09-27

    Call a web service over HTTP from a base URL with parameters set per row

  • REST Clientread 2026-09-27

    Call RESTful web services identified by URL

  • RSS Inputread 2026-09-27

    Read items from RSS feeds

  • Web services lookupread 2026-09-27

    Look up information from a SOAP web service described by WSDL

Power Query M (1)
RapidMiner Studio (8)
  • Crawl Webread 2026-09-27

    Crawl web pages from a start URL by link rules, storing pages and links

  • Enrich Data by Webserviceread 2026-09-27

    Call a web service per row and attach values extracted by regex, XPath or string matching

  • GET (REST)read 2026-09-27

    Send an HTTP GET request to a REST service and return the response

  • Get Pageread 2026-09-27

    Fetch one page over HTTP as a document

  • Get Pagesread 2026-09-27

    Fetch the pages for the URLs in a column and store the content in a new column

  • Process Documents from Webread 2026-09-27

    Crawl web pages and run an inner text pipeline on each to build a table

  • Read RSS Feedread 2026-09-27

    Read the entries of an RSS feed from a URL into a table

  • Read URLread 2026-09-27

    Read semicolon-separated CSV data from a URL into a table

Talend (18)
  • tAPIOperationread 2026-09-27

    Handle the requests of an API operation defined in the data service contract

  • tESBConsumerread 2026-09-27

    Call a method of a web service and return its response

  • tESBProviderRequestread 2026-09-27

    Expose the Job as a web service and pass each request message to the next component

  • tFileFetchread 2026-09-27

    Download a file over HTTP, HTTPS, FTP or SMB

  • tFindRegexlibExpressionsread 2026-09-27

    Fetch the regular expressions matching a request from the regexlib.com web service

  • tHTTPClientread 2026-09-27

    Call a web service on an HTTP server, such as REST or SOAP, with authentication

  • tHttpRequestread 2026-09-27

    Send an HTTP request and output the response; removed in favour of tHTTPClient

  • tLastRegexlibExpressionsread 2026-09-27

    Fetch the N most recent regular expressions matching a query from regexlib.com

  • tODataInputread 2026-09-27

    Read data from an OData web service

  • tRESTread 2026-09-27

    Send HTTP requests to a REST web service and read the responses; removed from recent releases

  • tRESTClientread 2026-09-27

    Send HTTP or HTTPS requests to a REST service and read the responses

  • tRESTRequestread 2026-09-27

    Serve a REST endpoint, accepting GET, POST, PUT, PATCH and DELETE requests from clients

  • tRSSInputread 2026-09-27

    Read the entries of RSS feeds from URLs

  • tRestWebServiceLookupInputread 2026-09-27

    Send requests to a REST web service and read the responses in a Spark Streaming Job

  • tSOAPread 2026-09-27

    Send a SOAP message to a web service and return the response

  • tWebServiceread 2026-09-27

    Call a method of a web service and return its values

  • tWebServiceInputread 2026-09-27

    Call a method of a web service and return the result

  • tXMLRPCInputread 2026-09-27

    Call a method of an RPC service and return the result

VisiData (3)
  • http (URL loader)read 2026-09-23

    Fetch an http or https URL, then parse by filetype; also scheme https

  • scrape (loader)read 2026-09-23

    Load a web page as a sheet of HTML elements

  • scrape-cellsread 2026-09-23

    Fetch the URLs in current column of selected rows and open them as HTML documents

Background

Hypertext Transfer Protocol (HTTP) is a request–response protocol: a server answers a client's request with a response that may contain the requested resource.[1] A web application programming interface (API) on a server consists of one or more endpoints to a defined request–response message system.[2] In HTTP, the GET method is the primary mechanism of information retrieval.[3]

When to use it

An HTTP request method is safe when its defined semantics are essentially read-only, so the client does not request or expect a change of state on the server.[3] The HTTP POST method asks the target resource to process the representation enclosed in the request under the resource's own semantics.[3] Read data with GET wherever an API offers it, and treat a POST as a request that may change something on the server unless the API's documentation says otherwise. (Meridian's view)

An HTTP method is idempotent when several identical requests have the same intended effect on the server as a single one; PUT, DELETE and the safe methods are idempotent.[3] An idempotent HTTP request can be repeated automatically if a communication failure occurs before the client reads the server's response.[3] Retry a failed GET freely, and read an API's documentation before retrying a POST. (Meridian's view)

The first digit of an HTTP status code defines its class: 1xx informational, 2xx successful, 3xx redirection, 4xx client error and 5xx server error.[3] Check the status of every response before treating its body as data. (Meridian's view) The HTTP status code 429 Too Many Requests means the user has sent too many requests in a given amount of time, and its response may include a Retry-After header.[4] A server sends the Retry-After header field to say how long the user agent ought to wait before a follow-up request, as an HTTP date or a number of seconds.[3] Servers are not required to use the 429 status code, and when limiting resource usage they may instead drop connections.[4] Pace a pipeline's requests, and honour Retry-After rather than retrying at once. (Meridian's view)

An HTTP response can carry an ETag, an entity tag that serves as an opaque validator for telling apart representations of the same resource.[3] The HTTP If-None-Match header field makes a request conditional on the server's selected representation having an entity tag that matches none of those listed.[3] The HTTP status 304 Not Modified answers a conditional GET or HEAD request whose condition was false, so the server need not transfer the representation again.[3] When a pipeline downloads the same file on a schedule, keep the ETag of the last download and send it back in If-None-Match, so an unchanged file is not fetched twice. (Meridian's view)

The method

In HTTP, a transaction starts with a client submitting a request; the server tries to satisfy it and returns a response describing the disposition of the request.[1] An HTTP response message includes headers and, where required, a body, which is typically the requested resource, although an error message may be returned instead.[1] The HTTP GET method requests transfer of a current selected representation of the target resource.[3]

Representational State Transfer (REST) is a software architectural style created to describe the design and guide the development of the architecture of the World Wide Web.[5] The computer scientist Roy Fielding introduced and defined the term representational state transfer in 2000, in his doctoral dissertation.[5] The informal term RESTful is more commonly associated with the design of HTTP-based APIs and their use of HTTP methods than with REST as originally formulated.[5] RESTful web APIs use HTTP methods to access resources, and typically use JavaScript Object Notation (JSON) or Extensible Markup Language (XML) to transmit data.[2][6][7]

Compared with the alternatives

Most web pages are designed for human end-users and not for ease of automated use, so specialised tools have been developed to scrape them.[8] Where a site offers an API or a file to download, use it before scraping the site's pages. (Meridian's view)

Sources

  1. [1]HTTP - Wikipediasecondary source
  2. [2]Web API - Wikipediasecondary source
  3. [3]RFC 9110: HTTP Semanticsstandard
  4. [4]RFC 6585: Additional HTTP Status Codesstandard
  5. [5]REST - Wikipediasecondary source
  6. [6]JSON - Wikipediasecondary source
  7. [7]XML - Wikipediasecondary source
  8. [8]Web scraping - Wikipediasecondary source

Used in

15 Protocols use this operation: