MeridianMERIDIAN

S3 URI

technology.cloud.s3_uri

Amazon S3 bucket/key URI in the format s3://bucket-name/key-path. Used in data pipelines, ETL tools, and cloud storage references.

Worked samples

DuckDB 1.5.5 · finetype 0.6.64

ValueAs VARCHARbucketkey
s3://my-bucket/data/file.csvs3://my-bucket/data/file.csvmy-bucketdata/file.csv
s3://production-logs/2024/01/events.parquets3://production-logs/2024/01/events.parquetproduction-logs2024/01/events.parquet
s3://data-lake-raw/ingestion/batch-001/part-00000.jsons3://data-lake-raw/ingestion/batch-001/part-00000.jsondata-lake-rawingestion/batch-001/part-00000.json

Shape

Pattern
^s3://[a-z0-9][a-z0-9.\-]{1,61}[a-z0-9](/.*)?$
ValueLengthShape
s3://my-bucket/data/file.csv28matches
s3://production-logs/2024/01/events.parquet43matches
s3://data-lake-raw/ingestion/batch-001/part-00000.json54matches
s3.//my-bucket/data/file.csv28rejected: separator swapped

Background

Origin

AWS launched Amazon S3 in the United States on March 14, 2006.[1] Amazon S3 stores objects organised into buckets, and each object is identified by a unique, user-assigned key.[1] The AWS CLI command reference defines an S3Uri as the location of an S3 object, prefix or bucket, written in the form s3://amzn-s3-demo-bucket/mykey.[2] The AWS CLI requires an S3 path argument to begin with s3:// to denote that it refers to an S3 object.[2] In the AWS CLI's S3Uri form, prefixes are separated by forward slashes, so the object myobject under the prefix myprefix in amzn-s3-demo-bucket is s3://amzn-s3-demo-bucket/myprefix/myobject.[2]

The Amazon S3 data model is a flat structure with no hierarchy of subbuckets or subfolders.[3] The Amazon S3 console presents folders by reading key name prefixes and the / delimiter.[3]

General purpose bucket names must be between 3 and 63 characters long.[4] General purpose bucket names can consist only of lowercase letters, numbers, periods and hyphens.[4] General purpose bucket names must begin and end with a letter or number.[4] An S3 object key name is a sequence of Unicode characters encoded in UTF-8, with a maximum length of 1,024 bytes.[3] S3 object key names are case sensitive.[3]

Use

DuckDB can query a file directly by an s3:// URL, as in FROM 's3://your-bucket/filename.extension'.[5] Hadoop's original s3:// client is no longer included in Hadoop.[6]

Amazon S3 virtual-hosted-style URLs take the form https://bucket-name.s3.region-code.amazonaws.com/key-name.[7] Amazon S3 path-style URLs take the form https://s3.region-code.amazonaws.com/bucket-name/key-name.[7] AWS says path-style URLs will be discontinued in the future.[7]

AWS recommends avoiding periods in bucket names, except for buckets used only for static website hosting, because virtual-host-style addressing over HTTPS does not work with them unless you perform your own certificate validation.[4] AWS lists characters such as the ampersand, the plus sign, the space and the question mark as ones that might require special handling in key names and most likely must be URL encoded.[3] AWS recommends not using characters such as the backslash, the percent character and the pound sign in key names.[3] Treat everything after the first slash following the bucket as an opaque key, and do not URL-decode or path-normalise it unless you know how the producing tool wrote it. (Meridian's view)

Where you'll meet it

The Registry of Open Data on AWS lists the NOAA Global Historical Climatology Network Daily data with the access command aws s3 ls --no-sign-request s3://noaa-ghcn-pds/.[8]

Sources

  1. [1]Amazon S3secondary source
  2. [2]s3 — AWS CLI Command Referencedocumentation
  3. [3]Naming Amazon S3 objectsdocumentation
  4. [4]General purpose bucket naming rulesdocumentation
  5. [5]S3 API Supportdocumentation
  6. [6]Hadoop-AWS module: Integration with Amazon Web Servicesdocumentation
  7. [7]Virtual hosting of general purpose bucketsdocumentation
  8. [8]NOAA Global Historical Climatology Network Daily (GHCN-D) - Registry of Open Data on AWSorganisation

Detection notes

Bucket names: 3-63 chars, lowercase, no underscores. Key paths can contain any characters after the bucket name.

Use it

CLI · DuckDB
$ finetype infer -i "s3://my-bucket/data/file.csv" --mode column
→ technology.cloud.s3_uri

SELECT ft_infer('s3://my-bucket/data/file.csv');
-- → 'technology.cloud.s3_uri'
Cast expression
CAST({col} AS VARCHAR)

Schema

https://meridian.online/schemas/technology.cloud.s3_uri · Frictionless type string

finetype taxonomy technology.cloud.s3_uri -o json-schema

Type Registry