Send to a web endpoint

Status

Variants

Send rows to a web endpoint

Send the rows to a URL as HTTP requests, one per row or in a batch.

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Background

When to use it

Webhooks are commonly used to trigger builds in continuous integration systems or to notify bug tracking systems.^[4] Because webhooks use HTTP, they can be integrated into web services without adding new infrastructure.^[4] Reach for a web endpoint when the system that should act on the results already listens for HTTP requests; when the destination reads files or tables, write to those instead. (Meridian's view)

An HTTP method is idempotent if the intended effect on the server of a single request is the same as the effect of several identical requests.^[6] Among HTTP methods, PUT and DELETE are idempotent, while POST and PATCH are not guaranteed to be idempotent.^[6] Successive identical POST requests may have additional effects, such as creating the same order several times.^[3] A client can safely retry a request that uses an idempotent method, for example when there is doubt as to whether the request reached the server.^[6] Before retrying a failed POST, find out whether the endpoint can recognise a repeated request, or a retry may create the same record twice. (Meridian's view)

The HTTP status 429 Too Many Requests means the client has sent too many requests in a given amount of time, a mechanism commonly called rate limiting.^[7] A 429 response may include a Retry-After header to indicate how long the client should wait before making the request again.^[7] When sending one request per row, pace the requests, and wait as long as Retry-After asks before trying again. (Meridian's view)

In Rapidminer, the POST (REST) operator fails by default when a web service returns an HTTP error status in the 4xx or 5xx range.^[8] Decide what an error status should do to the run, stop it or record the error and carry on, rather than leaving the choice to a default. (Meridian's view)

Server-side request forgery (SSRF) is an attack that abuses an application to interact with the internal or external network or the machine itself.^[9] One enabler of SSRF is the mishandling of URLs, as with a custom webhook whose handler or callback URL users specify.^[9] When users can control the external URL an application requests, as with webhooks, an allow list cannot be used, because the list of destinations is often unknown upfront and keeps changing.^[9] Treat a destination URL that arrives in the data as untrusted input. (Meridian's view) A webhook's incoming POST request should be authenticated to avoid a spoofing attack, and its timestamp verified to avoid a replay attack.^[4] A webhook can include a shared secret or digital signature so that the receiver can verify it.^[4]

The method

An HTTP message, request or response, has a start-line, an optional set of headers, an empty line marking the end of the metadata, and an optional body.^[2] The start-line of an HTTP/1.x request names the method, the request target, which is usually a URL, and the protocol version.^[2] In HTTP/2, messages are wrapped in binary framing, but the underlying semantics are the same as in the text-based messages of HTTP/1.x.^[2]

In Apache NiFi, the HttpRecordSink formats records with a Record Writer and sends them as the body of an HTTP POST request to a configured URL.^[10] NiFi's HttpRecordSink caps the records in each request's body with its Maximum Batch Size property, where zero sends all records together in a single request.^[10]

In Azure Data Factory, a Webhook activity calls an endpoint and passes it a callback URL, and the pipeline run waits for the callback before it proceeds to the next activity.^[11] The Azure Data Factory Webhook activity supports only the POST method and requires a Content-Type header.^[11] The Azure Data Factory Webhook activity waits 10 minutes for the callback by default.^[11]

Jeff Lindsay coined the term webhook in 2007, from the computer programming term hook.^[4]

Sources

1. [1]HTTP - Wikipediasecondary source
2. [2]HTTP messages - HTTP | MDNdocumentation
3. [3]POST request method - HTTP | MDNdocumentation
4. [4]Webhook - Wikipediasecondary source
5. [5]URL - Glossary | MDNdocumentation
6. [6]Idempotent - Glossary | MDNdocumentation
7. [7]429 Too Many Requests - HTTP | MDNdocumentation
8. [8]POST (REST) - Rapidminer Documentationdocumentation
9. [9]Server Side Request Forgery Prevention - OWASP Cheat Sheet Seriessecondary source
10. [10]HttpRecordSink - Apache NiFidocumentation
11. [11]Webhook activity - Azure Data Factory & Azure Synapse | Microsoft Learndocumentation