# S3 URI Key: `technology.cloud.s3_uri` Amazon S3 bucket/key URI in the format s3://bucket-name/key-path. Used in data pipelines, ETL tools, and cloud storage references. Page: Samples: `s3://my-bucket/data/file.csv`, `s3://production-logs/2024/01/events.parquet`, `s3://data-lake-raw/ingestion/batch-001/part-00000.json` ## Background ### Origin AWS launched Amazon S3 in the United States on March 14, 2006.[[1]](https://en.wikipedia.org/wiki/Amazon_S3) Amazon S3 stores objects organised into buckets, and each object is identified by a unique, user-assigned key.[[1]](https://en.wikipedia.org/wiki/Amazon_S3) The AWS CLI command reference defines an S3Uri as the location of an S3 object, prefix or bucket, written in the form s3://amzn-s3-demo-bucket/mykey.[[2]](https://docs.aws.amazon.com/cli/latest/reference/s3/) The AWS CLI requires an S3 path argument to begin with s3:// to denote that it refers to an S3 object.[[2]](https://docs.aws.amazon.com/cli/latest/reference/s3/) In the AWS CLI's S3Uri form, prefixes are separated by forward slashes, so the object myobject under the prefix myprefix in amzn-s3-demo-bucket is s3://amzn-s3-demo-bucket/myprefix/myobject.[[2]](https://docs.aws.amazon.com/cli/latest/reference/s3/) The Amazon S3 data model is a flat structure with no hierarchy of subbuckets or subfolders.[[3]](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html) The Amazon S3 console presents folders by reading key name prefixes and the / delimiter.[[3]](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html) General purpose bucket names must be between 3 and 63 characters long.[[4]](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html) General purpose bucket names can consist only of lowercase letters, numbers, periods and hyphens.[[4]](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html) General purpose bucket names must begin and end with a letter or number.[[4]](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html) An S3 object key name is a sequence of Unicode characters encoded in UTF-8, with a maximum length of 1,024 bytes.[[3]](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html) S3 object key names are case sensitive.[[3]](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html) ### Use DuckDB can query a file directly by an s3:// URL, as in FROM 's3://your-bucket/filename.extension'.[[5]](https://duckdb.org/docs/current/core_extensions/httpfs/s3api.html) Hadoop's original s3:// client is no longer included in Hadoop.[[6]](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html) Amazon S3 virtual-hosted-style URLs take the form https://bucket-name.s3.region-code.amazonaws.com/key-name.[[7]](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html) Amazon S3 path-style URLs take the form https://s3.region-code.amazonaws.com/bucket-name/key-name.[[7]](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html) AWS says path-style URLs will be discontinued in the future.[[7]](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html) AWS recommends avoiding periods in bucket names, except for buckets used only for static website hosting, because virtual-host-style addressing over HTTPS does not work with them unless you perform your own certificate validation.[[4]](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html) AWS lists characters such as the ampersand, the plus sign, the space and the question mark as ones that might require special handling in key names and most likely must be URL encoded.[[3]](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html) AWS recommends not using characters such as the backslash, the percent character and the pound sign in key names.[[3]](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html) Treat everything after the first slash following the bucket as an opaque key, and do not URL-decode or path-normalise it unless you know how the producing tool wrote it. (Meridian's view) ### Where you'll meet it The Registry of Open Data on AWS lists the NOAA Global Historical Climatology Network Daily data with the access command aws s3 ls --no-sign-request s3://noaa-ghcn-pds/.[[8]](https://registry.opendata.aws/noaa-ghcn/) ## Sources 1. [Amazon S3](https://en.wikipedia.org/wiki/Amazon_S3) — secondary source 2. [s3 — AWS CLI Command Reference](https://docs.aws.amazon.com/cli/latest/reference/s3/) — documentation 3. [Naming Amazon S3 objects](https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html) — documentation 4. [General purpose bucket naming rules](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html) — documentation 5. [S3 API Support](https://duckdb.org/docs/current/core_extensions/httpfs/s3api.html) — documentation 6. [Hadoop-AWS module: Integration with Amazon Web Services](https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html) — documentation 7. [Virtual hosting of general purpose buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html) — documentation 8. [NOAA Global Historical Climatology Network Daily (GHCN-D) - Registry of Open Data on AWS](https://registry.opendata.aws/noaa-ghcn/) — organisation