URL

Worked samples

DuckDB 1.5.5 · finetype 0.6.64

Shape

Background

Origin

RFC 1738, titled Uniform Resource Locators (URL), was published in December 1994.^[1] RFC 1738 was written by T. Berners-Lee, L. Masinter and M. McCahill as editors.^[1] RFC 1738 specifies a Uniform Resource Locator as the syntax and semantics of formalised information for location and access of resources via the Internet.^[1] RFC 1738 gives the common syntax for Internet schemes as //<user>:<password>@<host>:<port>/<url-path>.^[1] The schemes RFC 1738 covers include ftp, http, gopher, mailto, news, nntp, telnet, wais and file.^[1] Wikipedia traces the URL to a collaboration started at the IETF Living Documents birds of a feather session in 1992.^[2]

Tim Berners-Lee wrote that he wanted the URI syntax to separate the part the web browser has to know about, such as www.example.com, from the opaque rest that the client requests from the server.^[3] Berners-Lee wrote that the relative URI syntax is just unix pathname syntax reused without apology.^[3] Berners-Lee wrote that he copied the //computername/file/path form from the Apollo domain system, which had extended filename syntax that way for a network file system.^[3] Berners-Lee wrote in his FAQ that he regretted that the syntax is so clumsy.^[3]

RFC 3986, Uniform Resource Identifier (URI): Generic Syntax, was published in January 2005 as STD 66.^[4] RFC 3986 updates RFC 1738.^[4] RFC 3986 says the term Uniform Resource Locator refers to the subset of URIs that, in addition to identifying a resource, provide a means of locating the resource by describing its primary access mechanism.^[4] RFC 3986 recommends that future specifications use the general term URI rather than the more restrictive terms URL and URN.^[4] RFC 3986 splits the example foo://example.com:8042/over/there?name=ferret#nose into a scheme, an authority, a path, a query and a fragment.^[4] The WHATWG URL Standard sets out to align RFC 3986 and RFC 3987 with contemporary implementations and to obsolete those RFCs in the process.^[5] The WHATWG URL Standard also states a goal to standardise on the term URL.^[5]

Use

A URL is most commonly used to reference a web page over HTTP or HTTPS, but URLs are also used for file transfer, email, database access and many other applications.^[2] The WHATWG URL Standard lists ftp, file, http, https, ws and wss as special schemes.^[5] The WHATWG URL Standard gives default ports of 21 for ftp, 80 for http and 443 for https.^[5] Wikipedia describes protocol-relative URLs, such as //example.com, as URLs with no protocol specified that use the protocol of the current page.^[2] When a column mixes full addresses with protocol-relative ones, decide which scheme the source meant before you treat the two as the same thing. (Meridian's view)

RFC 3986 says schemes are case-insensitive, though their canonical form is lowercase.^[4] RFC 3986 says that although host is case-insensitive, producers and normalisers should use lowercase for registered names.^[4] In RFC 3986's percent-encoding, %20 stands for the space character.^[4] The WHATWG URL parser removes any leading and trailing C0 control or space from its input, and removes all ASCII tab or newline.^[5] The WHATWG URL Standard's examples parse https://example.org/foo bar to https://example.org/foo%20bar.^[5] The WHATWG URL Standard's examples parse https://EXAMPLE.com/../x to https://example.com/x.^[5] Before you join or deduplicate on a URL column, consider lowercasing the scheme and host and trimming whitespace, because comparing raw strings can split one address into several. (Meridian's view) Keep the original string alongside any normalised form, so that a cleaning step can always be checked against what the source actually wrote. (Meridian's view)

Python's documentation says that the urlsplit() and urlparse() functions do not perform validation of inputs.^[6] Python's urlsplit() recognises a netloc only if it is properly introduced by //, and otherwise presumes the input is a relative URL starting with a path.^[6] A column of bare host names such as www.example.com is better treated as domain names than as URLs, and a scheme should be added only when you know which one was meant. (Meridian's view) Split URLs into their parts with a parser rather than a hand-written regular expression, and then check the parts you care about, such as the scheme and host. (Meridian's view)

RFC 3986 warns that, because the userinfo subcomponent appears before the host, it can be used to construct a URI that appears to identify one trusted naming authority while actually identifying another.^[4] RFC 3986 gives ftp://cnn.example.com&story=breaking_news@10.0.0.1/top_story.htm as an example whose host is actually 10.0.0.1.^[4] When a URL column feeds anything a person will click, read the host from a parser rather than from the start of the string. (Meridian's view)

Where you'll meet it

The Common Crawl Foundation's URL Index is an index to the WARC files and URLs in the Common Crawl corpus, in Apache Parquet format.^[7] Common Crawl's URL Index table has a url column next to columns that split it into parts, such as url_host_name, url_protocol, url_port, url_path and url_query.^[7]

Sources

1. [1]RFC 1738: Uniform Resource Locators (URL) | RFC Editorstandard
2. [2]URL - Wikipediasecondary source
3. [3]Frequently asked questions by the Press - Tim BLsecondary source
4. [4]RFC 3986: Uniform Resource Identifier (URI): Generic Syntax | RFC Editorstandard
5. [5]URL Standardstandard
6. [6]urllib.parse — Parse URLs into components — Python 3.15.0 documentationdocumentation
7. [7]URL Indexorganisation

Detection notes

Use it

Schema

Aliases