Ask a language model

Status

Variants

Prompt a model for each row

Send a prompt built from each row to a language model and store its reply in a new column.

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Background

When to use it

In Apache SeaTunnel, an LLM transform given the prompt "Determine whether someone is Chinese or American by their name" adds an llm_output column holding the answer for each row.^[2] Reach for a model where a person could make the judgement from the text but no rule written in SQL captures it: a label, a cleaned-up spelling, a field buried in a sentence. (Meridian's view)

Hallucinations are output from a model that is syntactically fluent and appears factually sound but is factually incorrect.^[1] Biased or inaccurate training data can make an LLM's output less reliable.^[1] In OpenAI's API, LLM output is non-deterministic, and model behaviour changes between model snapshots and families.^[4] Treat a column of model answers as a draft: keep the model's name and the prompt beside it, and check a sample of rows by hand before anything downstream relies on it. (Meridian's view)

LLMs have trouble telling user instructions apart from instructions in content the user did not write, such as web pages and uploaded files.^[1] Text in the column being sent is that kind of content, so a cell that reads like an instruction may be followed as one. (Meridian's view)

VisiData's experimental LLM analysis asks for confirmation before analysing more than 100 rows, warning that the run may take some time and money.^[5] Run the prompt over a few rows first, and count what the whole column will cost before sending it. (Meridian's view)

The method

LLMs are typically based on the transformer neural network architecture.^[1] Generative pre-trained transformers (GPTs) are a type of LLM pre-trained to predict the next word, and GPTs are then often fine-tuned to follow instructions and to behave as assistants.^[1] Prompt engineering is the process of structuring natural language inputs, known as prompts, to produce specified outputs from a generative artificial intelligence (AI) model.^[6] In 2020, OpenAI researchers showed that GPT-3 could work out what format to use from a few rounds of questions and answers given as examples in its input, a technique called few-shot prompting.^[1]

The DuckDB ai community extension returns each model answer as an ordinary DuckDB value, such as a VARCHAR, a BOOLEAN, a DOUBLE or a typed STRUCT.^[7] The DuckDB ai extension's ai_extract_record function returns a STRUCT whose fields are shaped by a JSON Schema, so the answer arrives as typed fields rather than free text.^[7] In Apache SeaTunnel, the LLM transform's output_data_type option sets the type of its output column to STRING, INT, BIGINT, DOUBLE or BOOLEAN.^[2] VisiData's experimental LLM analysis asks the model to reply in JSON with a score and an analysis, and searches the raw reply for a score when the reply does not parse as JSON.^[5]

The DuckDB ai extension's ai_usage and ai_usage_summary functions track calls, tokens, failures and estimated cost.^[7] The DuckDB ai extension's ai_try_complete function returns a response and an error together, so one failed row does not stop the query.^[7] Flock, an experimental DuckDB extension, introduces MODEL and PROMPT objects as first-class SQL entities.^[8] Flock is part of ongoing research by the Data & AI Systems (DAIS) Laboratory at Polytechnique Montréal.^[8]

Sources

1. [1]Large language model - Wikipediasecondary source
2. [2]LLM | Apache SeaTunneldocumentation
3. [3]Prompt Tooldocumentation
4. [4]Model optimization | OpenAI APIdocumentation
5. [5]visidata/visidata/experimental/llm.py at 7609212ec2febafa9caba7183cbebfaf49627909 · saulpw/visidata · GitHubdocumentation
6. [6]Prompt engineering - Wikipediasecondary source
7. [7]ai – DuckDB Community Extensionsdocumentation
8. [8]flock – DuckDB Community Extensionsdocumentation