SuperGPQA dataset

A map of SuperGPQA's questions, one point per question, placed by a UMAP projection of its text and coloured by discipline. — Data: SuperGPQA (ODC-BY) and its fifteen upstream datasets, credited below

26,529 graduate-level multiple-choice questions spanning 13 disciplines, 72 fields and 285 subfields, each with four to ten answer options, the correct answer, a difficulty label and whether it requires calculation. The recipe also places each question on a two-dimensional map by what its text is about (an embedding of the text, reduced with UMAP); read the map as regions of related questions, not as distances. SuperGPQA is published under ODC-BY, which requires credit to the original creators of the fifteen datasets it draws on, and compliance with each one's own licence; all fifteen are credited in the sources below.

Load it

These commands install arc, Meridian’s open-source Protocol runner, clone the open-analytics repository and run this Protocol, which writes build/supergpqa.parquet in open-analytics/examples/supergpqa.

shell

curl -fsSL https://install.meridian.online/arc | bash
git clone https://github.com/meridian-online/open-analytics
cd open-analytics/examples/supergpqa && arc run

arc runs the Protocol’s SQL on DuckDB 2.x, which its installer puts in place when no DuckDB is on your PATH, saying what to add to PATH; the last step, which describes the Parquet, needs finetype 0.6.54 or later on your PATH.

This Protocol needs a DuckDB embedding extension on disk before it will run; its README says how to build it and where to put it.

Columns

Name	Type	Description
uuid	string	The source's own identifier for the question. Distinct on all 26,529 rows, which is what makes this Protocol's `ORDER BY ALL` exports a total order and pins the row order the UMAP fit was taken over.
question	string	The question text. THIS is the column that was embedded and projected.
answer_options	string	The 4 to 10 answer options the source offers, joined with ' \|\| ' — the source's `options` array flattened to a single delimited string, because a list column is not a shape the export/describe pair in this directory has handled. The delimiter is ' \|\| ' and not the ' \| ' examples/scienceqa uses, because 31 of these 26,529 rows carry ' \| ' inside an option and splitting on it would return the wrong options for those 31. models/corpus.sql stops the run if ' \|\| ' ever appears inside an option.
answer	string	The text of the correct option, as the source records it.
answer_letter	string	The correct option's letter, A through J. It selects `answer` out of `answer_options` on all 26,529 rows; models/corpus.sql stops the run if the letter and the text ever name different options.
discipline	string	The broadest of the source's three subject levels — 13 distinct values across these rows, e.g. Engineering.
field	string	The middle subject level — 72 distinct values, e.g. Electronic Science and Technology.
subfield	string	The finest subject level — 285 distinct values, e.g. Circuits and Systems. This is the count the source's own title refers to as its graduate disciplines.
difficulty	string	The source's difficulty label: `easy` (7,731 rows), `middle` (11,748) or `hard` (7,050).
is_calculation	boolean	Whether the source marks the question as requiring calculation — true on 11,229 of these 26,529 rows.
projection_x	number	The first UMAP coordinate of this row's question text. A position on a map, not a measurement: read the map as regions. See the package description.
projection_y	number	The second UMAP coordinate of this row's question text. Read with projection_x and only against rows sharing this row's projection_fit_id.
projection_fit_id	string	A fingerprint of the exact feature matrix and knobs the UMAP fit consumed, the same value on every row. Two exports carrying the same value came from one fit and their coordinates are comparable; two carrying different values are two maps, and a position in one says nothing about a position in the other.

Source and licence

Licence

Open Data Commons Attribution License. It asserts no copyright over the underlying content; what it requires is credit to the original creators of any third-party data included, and compliance with the respective licences of the referenced datasets. The fifteen datasets SuperGPQA's own card names are listed individually under `sources`.

Sources

Meridian hosts none of this data: the Protocol downloads it, from the address its fetch step names, when you run it.

The steps

8 stepsduckdb >=1.2

Every step runs in order. op: steps are typed, versioned operators; sql: steps are DuckDB models.

01fetch_supergpqaop: http_fetch@1

Fetch SuperGPQA-all.jsonl from the m-a-p/SuperGPQA repository on Hugging Face, pinned to a commit revision and by sha256.

producesbuild/supergpqa.src.jsonl

with

url: https://huggingface.co/datasets/m-a-p/SuperGPQA/resolve/4430d4458112c7d4497fdcf94d7cc223313d6acf/SuperGPQA-all.jsonl
out: build/supergpqa.src.jsonl
sha256: 28b998e70205ee95e540317b5adc06a06552a3961fb50b153df126b833f7a910
headers:
  User-Agent: Meridian Protocol (open-analytics demo; research@meridian.online)

02corpussql: models/corpus.sql
Read the JSONL with all ten columns and their types declared rather than inferred, and flatten each question's options into one ' || '-delimited string. Stops the run if any of the fifteen upstream datasets the licence requires credit for is missing from the descriptor.
readsbuild/supergpqa.src.jsonldescriptor.overrides.jsonproducescorpus
model
```
models/corpus.sql
```
03export_corpusop: parquet_export@1
Write the corpus Parquet the embedder reads, in ORDER BY ALL order (uuid is distinct and sorts first), so the projection is fitted over the same row order every run.
readscorpusproducesbuild/supergpqa.corpus.parquet
with
```
input: corpus
dest: build/supergpqa.corpus.parquet
compression: zstd
order_by: ALL
```
04embed_questionsop: text_embed@1
Embed each question's text with a static sentence embedder (the staticembed DuckDB extension), one 256-dimensional vector per question.
readsbuild/supergpqa.corpus.parquetvendor/staticembed.duckdb_extensionproducesbuild/supergpqa.embedded.parquet
with
```
input: build/supergpqa.corpus.parquet
text_column: question
extension: vendor/staticembed.duckdb_extension
out: build/supergpqa.embedded.parquet
vector_column: embedding
```
needs the staticembed DuckDB extension at vendor/staticembed.duckdb_extension, which has no published URL; absent, the run stops at once naming the file. The Protocol's README says how to build it.
05project_questionsop: umap_project@1
Reduce the vectors to two map coordinates, projection_x and projection_y, with UMAP over cosine distance. The seed is frozen and the work single-threaded, so two runs land on the same map. Read it as regions, not distances.
readsbuild/supergpqa.embedded.parquetproducesbuild/supergpqa.projected.parquet
with
```
input: build/supergpqa.embedded.parquet
columns: [embedding]
out: build/supergpqa.projected.parquet
metric: cosine
```
06loadsql: models/load.sql
Drop the vectors and keep everything else, with the two map coordinates and the fit's fingerprint.
readsbuild/supergpqa.projected.parquetproducessupergpqa
model
```
models/load.sql
```
07export_supergpqaop: parquet_export@1
Write build/supergpqa.parquet, the file this Protocol ends in, on your own disk. ORDER BY ALL makes the bytes reproducible.
readssupergpqaproducesbuild/supergpqa.parquet
with
```
input: supergpqa
dest: build/supergpqa.parquet
compression: zstd
order_by: ALL
```
08describeop: datapackage_describe@1
Emit datapackage.json from the built Parquet: finetype types every column, and descriptor.overrides.json adds the curated title, licence, sources and per-field prose.
readsbuild/supergpqa.parquetdescriptor.overrides.jsonproducesdatapackage.json
with
```
parquet: build/supergpqa.parquet
overrides: descriptor.overrides.json
out: datapackage.json
```