Read from an application's API

Status

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Background

When to use it

In Apache Hop's Salesforce Input transform, leaving the Query all records box unchecked retrieves only the first 2000 records, because of Salesforce's default pagination settings.^[4] In GitHub's REST API, the per_page query parameter sets how many results a page returns, and a value above the endpoint's maximum is reduced to that maximum without an error.^[5] Check the number of rows a read returns against the number the application shows before trusting the table. (Meridian's view)

GitHub's REST API allows 60 requests per hour for unauthenticated requests and 5,000 requests per hour for an authenticated user's personal rate limit.^[6] The Hypertext Transfer Protocol (HTTP) status code 429 Too Many Requests means the user has sent too many requests in a given amount of time, and its response may carry a Retry-After header saying how long to wait.^[7][8] GitHub's REST API sends x-ratelimit-remaining and x-ratelimit-reset headers with each response, giving the requests left in the current window and the time that window resets.^[6] Continuing to make requests while rate limited on GitHub's REST API may result in the integration being banned.^[6] Plan a large read to fit inside the application's request budget, and have it wait when told to slow down rather than retry at once. (Meridian's view)

Apache Hop's Salesforce Input transform can retrieve all, updated or deleted records, and with Query all records cleared it takes a start date and an end date to bound the range.^[4] Talend's tSalesforceGetUpdated and tSalesforceGetDeleted components collect the data updated, or deleted, during a specific period of time from a Salesforce object.^[9] Before relying on repeated reads of changed records alone, find out how the application reports the records that were deleted. (Meridian's view) In Apache Hop's Salesforce Input transform, Salesforce field API names such as CustomField__c stay the same when an administrator changes a field's label, while labels can change.^[4] Name columns by the application's stable field identifiers rather than by the labels people see on screen. (Meridian's view)

The method

The Singer specification describes a tap as an application that takes a configuration file and an optional state file and outputs an ordered stream of record, state and schema messages.^[3] In the Singer specification, a tap's configuration typically includes the credentials for the API or data source.^[3] In the Singer specification, state persists information between a tap's invocations, and a common use is to record the spot in the stream where the last invocation left off.^[3]

GitHub's REST API gives the URLs of the previous, next, first and last pages in a link header, and each paginated endpoint uses the page, before and after, or since query parameters.^[5] Stripe's list API methods use cursor-based pagination, in which the starting_after parameter takes the ID of the last object received to fetch the next page.^[10] A Stripe list response carries a has_more value that is false when the set returned is the end of the list.^[10]

The dlt library is an open-source Python library that loads data from various, often messy data sources into well-structured datasets.^[11] The dlt library's REST API source defines an API's endpoints, their relationships, pagination and authentication by declarative configuration, and tries to detect the pagination from the first response.^[12] The dlt library describes incremental loading as loading only new or changed data and not old records already loaded, which needs track kept of the state of the load.^[13] In the dlt REST API source, an incremental section names a cursor path and an initial value, and on each later run the start value becomes the maximum cursor value the last run saw.^[12] The dlt library offers three write dispositions: a full load replaces the destination dataset, append adds the new data, and merge merges new data in using a merge key or primary key.^[13] The dlt library's lag, or attribution window, reacquires recent data during incremental loading, for example always capturing the last 7 days when fetching daily analytics reports.^[14]

The dlt library's requests wrapper, a customised Python Requests client, retries a failing request up to 5 times by default with an exponentially increasing delay, and retries on status 429 and every status in the 500 range.^[15] When a 429 or 503 response carries a Retry-After header, the dlt requests wrapper uses that value in place of its standard retry delay.^[15]

Sources

1. [1]Software as a service - Wikipediasecondary source
2. [2]Web API - Wikipediasecondary source
3. [3]getting-started/docs/SPEC.md at master · singer-io/getting-started · GitHubdocumentation
4. [4]Salesforce Input | Apache Hopdocumentation
5. [5]Using pagination in the REST API - GitHub Docsdocumentation
6. [6]Rate limits for the REST API - GitHub Docsdocumentation
7. [7]RFC 6585: Additional HTTP Status Codesstandard
8. [8]429 Too Many Requests - HTTP | MDNsecondary source
9. [9]Salesforce components | Talend Components for Jobs Helpdocumentation
10. [10]Pagination | Stripe API Referencedocumentation
11. [11]Getting started with dlt | dlt Docsdocumentation
12. [12]REST API source | dlt Docsdocumentation
13. [13]Incremental loading in Python with dlt | dlt Docsdocumentation
14. [14]Lag / Attribution window | dlt Docsdocumentation
15. [15]REST API helpers | dlt Docsdocumentation

Used in