California Housing dataset

A point map of California's census districts by longitude and latitude, each point coloured by the district's median house value.

20,640 California districts (census block groups) from the 1990 Census, each with its median income, housing age, average rooms and occupancy, population, latitude and longitude, and median house value. Pace and Barry published it on StatLib in 1997, and the recipe reads it from there. Latitude and longitude are plain numeric columns, which is all a point map of the districts needs.

Load it

These commands install arc, Meridian’s open-source Protocol runner, clone the open-analytics repository and run this Protocol, which writes build/california_housing.parquet in open-analytics/examples/california-housing.

shell

curl -fsSL https://install.meridian.online/arc | bash
git clone https://github.com/meridian-online/open-analytics
cd open-analytics/examples/california-housing && arc run

arc runs the Protocol’s SQL on DuckDB 2.x, which its installer puts in place when no DuckDB is on your PATH, saying what to add to PATH; the last step, which describes the Parquet, needs finetype 0.6.60 or later on your PATH, because this Protocol’s describe step passes nominations.

Columns

Name	Type	Description
median_income	number	Median household income for the district, in tens of thousands of 1990 US dollars.
house_age	number	Median age of the district's houses, in years.
avg_rooms	number	Mean rooms per household. A district average, so values above ten are small households in large houses rather than errors.
avg_bedrooms	number	Mean bedrooms per household, on the same district-average basis.
population	number	Number of people resident in the district.
avg_occupancy	number	Mean household members per household.
latitude	number	District centroid latitude. With longitude, this is a map that needs no embedding behind it.
longitude	number	District centroid longitude.
median_house_value	number	Median house value for the district, in hundreds of thousands of 1990 US dollars. Capped at 5.00001 in the source, which is a ceiling rather than a measurement.

Source and licence

Licence

No licence declared at source

Sources

Pace, R. Kelley and Ronald Barry, "Sparse Spatial Autoregressions", Statistics and Probability Letters 33 (1997) 291–297; houses.zip on StatLib

Meridian hosts none of this data: the Protocol downloads it, from the address its fetch step names, when you run it.

The steps

5 stepsduckdb >=1.2

Every step runs in order. op: steps are typed, versioned operators; sql: steps are DuckDB models.

01fetch_housingop: http_fetch@1
Fetch houses.zip from StatLib, where Pace and Barry published it, pinned by sha256. StatLib has no revisions, so the hash is the only record of which bytes the recipe was written against.
producesbuild/houses.zip
with
```
url: https://lib.stat.cmu.edu/datasets/houses.zip
out: build/houses.zip
sha256: 8b18f0a01cf9c99a65174d18fa582aa31971dfe55a26ad794f3299937c3708d7
headers:
  User-Agent: Meridian Protocol (open-analytics demo; research@meridian.online)
```
02extract_housingop: archive_extract@1
Extract cadata.txt from the zip.
readsbuild/houses.zipproducesbuild/statlib/cadata.txt
with
```
archive: build/houses.zip
members: [cadata.txt]
dest: build/statlib
```
03loadsql: models/load.sql
Split StatLib's space-padded lines into their nine numbers, derive the per-household averages the way scikit-learn does, and put house value in hundreds of thousands of dollars.
readsbuild/statlib/cadata.txtproducescalifornia_housing
model
```
models/load.sql
```
04export_california_housingop: parquet_export@1
Write build/california_housing.parquet, the file this Protocol ends in, on your own disk. ORDER BY ALL makes the bytes reproducible.
readscalifornia_housingproducesbuild/california_housing.parquet
with
```
input: california_housing
dest: build/california_housing.parquet
compression: zstd
order_by: ALL
```
05describeop: datapackage_describe@1
Emit datapackage.json from the built Parquet: finetype types every column, and descriptor.overrides.json adds the curated title, licence, sources and per-field prose. nominations.finetype.json names the type of the one column finetype cannot settle on its own, avg_bedrooms.
readsbuild/california_housing.parquetdescriptor.overrides.jsonnominations.finetype.jsonproducesdatapackage.json
with
```
parquet: build/california_housing.parquet
overrides: descriptor.overrides.json
nominations: nominations.finetype.json
out: datapackage.json
```