MeridianMERIDIAN

User Agent String

technology.internet.user_agent

HTTP User-Agent header string identifying the client application. Typically contains browser, OS, and version information.

Casts to
VARCHAR
Standard

Shape

Length 10 to 500 characters.

Pattern
^(Mozilla/|curl/|python-requests/|Wget/|Go-http-client/|axios/|PostmanRuntime/|kube-probe/|Java/|okhttp/|Apache-HttpClient/|libcurl/|node-fetch/|Dalvik/|CFNetwork/|Lynx/|Links |Scrapy/|Googlebot/|Bingbot/|Slackbot|Twitterbot/|facebookexternalhit/|LinkedInBot/|Prometheus/|Datadog/|Ruby/|Dart/|grpc-|HTTPie/|bot|spider|crawl)
ValueLengthShape
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.3660rejected
Mozilla/5.0 (iPhone; CPU iPhone OS 15_0 like Mac OS X)54rejected
curl/7.64.111rejected
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.359rejected: one character short

Background

Origin

RFC 9110 says that, by convention, product identifiers are listed in decreasing order of their significance for identifying the user agent software.[1] RFC 9110 encourages implementations not to use the product tokens of other implementations to declare compatibility with them.[1]

WebAIM's history of the user-agent string records that NCSA Mosaic called itself NCSA_Mosaic/2.0 (Windows 3.1).[2] According to WebAIM's history, Netscape called itself Mozilla/1.0 (Win3.1), and webmasters began sending frames only to browsers identifying as Mozilla.[2] WebAIM's history describes Internet Explorer declaring itself Mozilla compatible, with a string such as Mozilla/1.22 (compatible; MSIE 2.0; Windows 95), so that it would be sent frames.[2] MDN says Mozilla/5.0 is a general token meaning the browser is Mozilla-compatible, and that for historical reasons almost every browser today sends it.[3] The history of impersonation is why a modern browser string reads like a list of every engine it has ever had to impersonate, and why the first token tells you almost nothing about which browser actually made the request. (Meridian's view)

Use

RFC 9110 says servers often use the User-Agent field to identify the scope of reported interoperability problems, to work around user agent limitations, and for analytics on browser or operating system use.[1] The Apache HTTP Server's Combined Log Format records the User-Agent request header as its final quoted field.[4] Apache's documentation describes the logged User-Agent as the identifying information that the client browser reports about itself.[4]

By default, curl sends a User-Agent of the form curl/VERSION.[5] MDN lists curl/7.64.1 and PostmanRuntime/7.26.5 as examples of library and network tool user-agent strings.[3] Wikipedia notes that user agent spoofing may make collected statistics of web browser usage inaccurate.[6]

Chrome's User-Agent Reduction plan replaces the minor, build and patch parts of the Chrome version with 0.0.0.[7] Under User-Agent Reduction, Chrome's desktop platform token is one of a fixed set of literal values, such as Windows NT 10.0; Win64; x64 or Macintosh; Intel Mac OS X 10_15_7.[7] The Chromium plan's final phase, in Chrome 113, gives all page loads the reduced user-agent string.[7]

Sources

  1. [1]RFC 9110: HTTP Semanticsstandard
  2. [2]WebAIM: History of the browser user-agent stringsecondary source
  3. [3]User-Agent header - HTTP | MDNsecondary source
  4. [4]Log Files - Apache HTTP Server Version 2.4documentation
  5. [5]curl - How To Usedocumentation
  6. [6]User-Agent header - Wikipediasecondary source
  7. [7]User-Agent Reductiondocumentation

Detection notes

v1 migration: Was internet.user_agent. v13: Added prefix-based pattern for validation branch signal. Checks for known UA prefixes (Mozilla/, curl/, python-requests/, etc.) and common bot/spider/crawler identifiers. Parse with dedicated UA parser for full details.

Use it

CLI · DuckDB
$ finetype infer -i "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" --mode column
→ technology.internet.user_agent

SELECT ft_infer('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36');
-- → 'technology.internet.user_agent'
Cast expression
CAST({col} AS VARCHAR)

Schema

https://meridian.online/schemas/technology.internet.user_agent · Frictionless type string

finetype taxonomy technology.internet.user_agent -o json-schema

Type Registry