Acquire / http · All operations

Scrape or crawl web pages

A variant of Fetch from the web

Fetch web pages, following links from a start page if asked, and keep their content or HTML elements.

Status

In arcform

Not built yet

arcform does not do this yet. Its place on the roadmap

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

KNIME Analytics Platform (2)
  • KNIME Web Interaction (Labs) (extension)read 2026-09-27

    Extension outside the base install: 9 current nodes: Clicker (Labs); Content Retriever (Labs); Navigator (Labs); Screenshotter (Labs), …

  • Webpage Retrieverread 2026-09-27

    Fetch an HTML page by HTTP GET and output it cleaned as XHTML or a string

RapidMiner Studio (3)
  • Crawl Webread 2026-09-27

    Crawl web pages from a start URL by link rules, storing pages and links

  • Get Pageread 2026-09-27

    Fetch one page over HTTP as a document

  • Process Documents from Webread 2026-09-27

    Crawl web pages and run an inner text pipeline on each to build a table

VisiData (2)
  • scrape (loader)read 2026-09-23

    Load a web page as a sheet of HTML elements

  • scrape-cellsread 2026-09-23

    Fetch the URLs in current column of selected rows and open them as HTML documents

Background

Web scraping is data scraping used for extracting data from websites, and the term typically refers to automated processes using a bot or web crawler.[1] Scraping a web page involves fetching it and then extracting data from it.[1] A web crawler starts from a list of web addresses called seeds, finds the hyperlinks in the pages it retrieves and adds them to the list of addresses to visit.[2]

When to use it

Most web pages are designed for human end-users and not for ease of automated use.[1] Scrape only when the site offers no feed, interface or file to download that holds the same data, and expect a scraper to need repair when the page's layout changes. (Meridian's view) The legality of web scraping varies across the world, and scraping may be against the terms of service of some websites, though the enforceability of those terms is unclear.[1]

The Robots Exclusion Protocol, originally defined by Martijn Koster in 1994, lets service owners control how automatic clients known as crawlers may access their content.[3] The rules of the Robots Exclusion Protocol are not a form of access authorisation.[3] RapidMiner's Crawl Web operator queries robots.txt for each page it visits, and has an ignore robot exclusion parameter meant to be enabled only for one's own sites.[4]

The robots exclusion standard does not suggest an interval between visits to the same server, though some commercial search engines read an extra Crawl-delay parameter giving the seconds between requests.[2] RapidMiner's Crawl Web operator bounds a crawl with max pages and max crawl depth, where a depth of 1 means crawling only the direct links on the starting page.[4] Set a page limit, a depth and a delay before the first run, and identify the crawler honestly in the user agent it sends. (Meridian's view)

Programs such as Selenium or Playwright control a web browser that fully loads each page, which is especially useful for scraping data from dynamic sites.[1] When data shows in the browser but not in the fetched page, the page is probably built in the browser, and a scraper that drives a browser is the one to reach for. (Meridian's view)

The method

Once a page is fetched, its content may be parsed, searched and reformatted, and its data copied into a spreadsheet or loaded into a database.[1] A simple approach to extracting information from web pages uses the UNIX grep command or the regular expression facilities of programming languages to find text matching a pattern.[1] Once a browser has loaded a whole page, developers can parse its Document Object Model (DOM) using an expression language such as XPath.[1]

KNIME's Webpage Retriever node fetches web pages by Hypertext Transfer Protocol (HTTP) GET requests and parses their Hypertext Markup Language (HTML) with the jsoup library.[5][6][7] KNIME's Webpage Retriever node cleans the parsed HTML by removing comments and, optionally, replacing relative links with absolute ones.[5] VisiData's scrape loader lists the HTML elements of a page as parsed by beautifulsoup4, with a selector for each, and can add a column of the elements matching a given selector.[8]

A web crawler's list of addresses still to visit is called the crawl frontier, and its addresses are visited recursively under a set of policies.[2] The behaviour of a web crawler combines policies that include a selection policy stating which pages to download and a politeness policy stating how to avoid overloading websites.[2] The MercatorWeb crawler follows an adaptive politeness policy: if a document took t seconds to download from a server, the crawler waits 10t seconds before the next page.[2]

Compared with the alternatives

Web crawlers are typically operated by search engines for web indexing, and can also be used for web scraping.[2]

Sources

  1. [1]Web scraping - Wikipediasecondary source
  2. [2]Web crawler - Wikipediasecondary source
  3. [3]RFC 9309: Robots Exclusion Protocolstandard
  4. [4]Crawl Web - Rapidminer Documentationdocumentation
  5. [5]Webpage Retrieverdocumentation
  6. [6]HTTP - Wikipediasecondary source
  7. [7]HTML - Wikipediasecondary source
  8. [8]visidata/visidata/loaders/scrape.py at 7609212ec2febafa9caba7183cbebfaf49627909 · saulpw/visidata · GitHubdocumentation