ScienceQA dataset

ScienceQA is a set of multiple-choice science questions with worked explanations, published by Pan Lu and colleagues with their NeurIPS paper, 'Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering'. The questions span natural, social and language science across the school grades, and each comes with its choices, the correct answer, a hint, the lecture text it is drawn from and a worked solution. The recipe reads the test split, the one the dataset's own leaderboard scores against, and keeps whether a question came with a figure but not the figure itself. The licensor publishes it under Creative Commons Attribution-NonCommercial-ShareAlike, which restricts commercial use of whatever you build from it.

Load it

These commands install arc, Meridian’s open-source Protocol runner, clone the open-analytics repository and run this Protocol, which writes build/scienceqa.parquet in open-analytics/examples/scienceqa.

shell

curl -fsSL https://install.meridian.online/arc | bash
git clone https://github.com/meridian-online/open-analytics
cd open-analytics/examples/scienceqa && arc run

arc runs the Protocol’s SQL on DuckDB 2.x, which its installer puts in place when no DuckDB is on your PATH, saying what to add to PATH; the last step, which describes the Parquet, needs finetype 0.6.54 or later on your PATH.

Columns

Name	Type	Description
question	string	The question text.
answer_choices	string	The answer options the source offers, joined with ' \| ': the source's `choices` array flattened to a single delimited string.
answer_index	integer	Zero-based index into the pipe-delimited answer_choices identifying the correct option, as the source records it. Meaningless on its own without answer_choices or answer_text.
answer_text	string	The text of the correct choice, joined from the source's `choices[answer]` so a reader does not have to re-parse answer_choices to learn it.
hint	string	A short context hint the source provides for the question, empty where none is given.
task	string	The question format the source assigns, e.g. closed choice.
grade	string	Grade level the question targets, as the source labels it.
subject	string	Top-level subject: natural science, social science or language science.
topic	string	Subject subdivision, e.g. figurative-language.
category	string	Finer-grained topic label than subject/topic, e.g. Literary devices.
skill	string	The specific skill the question exercises, as the source names it.
lecture	string	Background teaching text the question draws on, empty where the source gives none.
solution	string	Worked explanation of the correct answer, empty where the source gives none.
has_image	boolean	Whether the source shipped a figure with this question. The image bytes themselves are not carried into this export.

Source and licence

Licence

Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International — NonCommercial: the licensor restricts commercial use of the output, not the act of building it. Verified against the licensor's project site and source repository, not the mirror's YAML tag, which reads plain CC BY-SA and disagrees with its own card body.

Sources

Meridian hosts none of this data: the Protocol downloads it, from the address its fetch step names, when you run it.

The steps

4 stepsduckdb >=1.2

Every step runs in order. op: steps are typed, versioned operators; sql: steps are DuckDB models.

01fetch_scienceqaop: http_fetch@1

Fetch the test split's Parquet from the derek-thomas/ScienceQA mirror on Hugging Face, pinned to a commit revision and by its content hash.

producesbuild/scienceqa.src.parquet

with

url: https://huggingface.co/datasets/derek-thomas/ScienceQA/resolve/f18b0a70359ebfb41f658fd564208d0355b013f4/data/test-00000-of-00001-f0e719df791966ff.parquet
out: build/scienceqa.src.parquet
sha256: 235e92d1f30155266df76bc9f28fc6e4fcb6bec2c6a8c7d67f9086ea6b392a84
headers:
  User-Agent: Meridian Protocol (open-analytics demo; research@meridian.online)

02loadsql: models/load.sql
Reduce each question's embedded picture to has_image, whether it had one, flatten its choices into one delimited string, and join the correct answer's text in as answer_text.
readsbuild/scienceqa.src.parquetproducesscienceqa
model
```
models/load.sql
```
03export_scienceqaop: parquet_export@1
Write build/scienceqa.parquet, on your own disk. ORDER BY ALL makes the bytes reproducible: the only rows it leaves unordered are exact duplicates of each other, which cannot change the output.
readsscienceqaproducesbuild/scienceqa.parquet
with
```
input: scienceqa
dest: build/scienceqa.parquet
compression: zstd
order_by: ALL
```
04describeop: datapackage_describe@1
Emit datapackage.json from the built Parquet: finetype types every column, and descriptor.overrides.json adds the curated title, licence, sources and per-field prose.
readsbuild/scienceqa.parquetdescriptor.overrides.jsonproducesdatapackage.json
with
```
parquet: build/scienceqa.parquet
overrides: descriptor.overrides.json
out: datapackage.json
```