Model / text · All operations

Embed text as vectors

Text or image to vector.

Status

In arcform

Runs as a SQL step

arcform carries it as a sql: step today, with the subtoken extension.

Shape of the result
one row out per row in, with columns added or changed
Level
needs a DuckDB extension

The DuckDB step

Embed text and rank by cosine similarity

It reads the orders table the proofs share. Create it first:

SQL
CREATE TABLE orders AS SELECT * FROM (VALUES
  (1,  'Alice Ng',      'north', DATE '2026-01-03', 120.00, 'widget', 'a|b',   '{"channel":"web","items":[1,2]}'),
  (2,  'alice ng',      'north', DATE '2026-01-05',  80.50, 'gadget', 'b',     '{"channel":"shop","items":[3]}'),
  (3,  'Bob  Okafor',   'south', DATE '2026-01-05', 300.00, 'widget', 'a|c',   '{"channel":"web","items":[4,5,6]}'),
  (4,  'Carla Díaz',    'east',  DATE '2026-01-09',  NULL,  'gizmo',  NULL,    '{"channel":"web","items":[]}'),
  (5,  'Bob Okafor',    'south', DATE '2026-01-12',  45.25, 'gadget', 'c',     '{"channel":"phone","items":[7]}'),
  (6,  'Dev Patel',     'west',  DATE '2026-01-12', 510.00, 'widget', 'a',     '{"channel":"web","items":[8,9]}'),
  (7,  'Carla Diaz',    'east',  DATE '2026-01-20',  99.99, 'gizmo',  'b|c',   '{"channel":"shop","items":[10]}'),
  (8,  'Dev Patel',     'west',  DATE '2026-02-02', 510.00, 'widget', 'a',     '{"channel":"web","items":[8,9]}'),
  (9,  'Eve Larsen',    NULL,    DATE '2026-02-03',  15.00, 'gadget', NULL,    '{"channel":"web","items":[11]}'),
  (10, 'Alice Ng',      'north', DATE '2026-02-10', 220.00, 'gizmo',  'a|b|c', '{"channel":"phone","items":[12,13]}')
) t(order_id, customer, region, order_date, amount, product, tags, meta);

Then run the statement:

SQL
INSTALL subtoken FROM community; LOAD subtoken;
CREATE TEMP TABLE result AS
WITH e AS (SELECT order_id, product, customer, subtoken_embed(customer || ' bought a ' || product) AS v FROM orders),
     q AS (SELECT subtoken_embed('someone purchased a gadget') AS v)
SELECT e.order_id, e.customer, e.product, len(e.v) AS dim, list_cosine_similarity(e.v, q.v) AS cos FROM e, q;
SELECT customer, product, round(cos, 4) AS cos FROM result ORDER BY cos DESC LIMIT 3;
SELECT CASE WHEN ((SELECT min(dim) = 256 AND max(dim) = 256 AND arg_max(product, cos) = 'gadget' AND min(cos) FILTER (WHERE product = 'gadget') > max(cos) FILTER (WHERE product <> 'gadget') FROM result)) THEN 'as expected' ELSE error('the orders for a gadget did not rank above the others for a query about a gadget') END AS verdict;
  • It needs the subtoken extension.
  • The statement checks its own result and raises an error when it is wrong: “the orders for a gadget did not rank above the others for a query about a gadget”.
  • It passed on DuckDB v2.0.0 (Cyanoptera) d54cbbe3df.

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Apache Hop (1)
  • Embed textread 2026-09-27

    Turn a text field into an embedding vector through an AI provider

Apache SeaTunnel (1)
  • Embeddingread 2026-09-27

    Turn text, image or video fields into float32 vectors through an embedding model provider

Duckle (1)
  • Embeddingsread 2026-09-25

    per-row embedding from an OpenAI-compatible endpoint

KNIME Analytics Platform (1)
RapidMiner Studio (2)
  • Embeddings (FastEmbed)read 2026-09-27

    Compute text embeddings locally with FastEmbed and store them as a new column

  • Embeddings (OpenAI)read 2026-09-27

    Compute text embeddings with the OpenAI API and store them as a new column

subtoken (1)
  • subtoken_embedread 2026-09-27

    Return the 256-wide FLOAT[] static embedding of one string, from a model compiled into the binary

Talend (1)
  • tEmbeddingAIread 2026-09-27

    Compute embeddings of input data with a chosen embedding model

Background

In natural language processing, a sentence embedding represents a natural language text as a vector of numbers that encodes meaningful semantic information.[1] In RapidMiner, an embedding operator stores each text's vector in a new column, as input to machine learning or to similarity-based retrieval.[2][3]

When to use it

Sentence embeddings are used for semantic search, in which chunks of documents are embedded and stored, and a query in natural language is embedded too.[1] A top k similarity search between the query embedding and the document chunk embeddings then retrieves the most relevant chunks, as context for question answering.[1] In Apache Hop, the Embed text transform is the middle step of a retrieval pipeline: chunk the documents, embed the chunks, then write them to a vector store.[4] Reach for an embedding when the question is which texts mean much the same thing, whatever words they use; where the question is whether a text holds a given word or code, a plain match answers it more clearly. (Meridian's view)

In RapidMiner, the FastEmbed operator produces embeddings with 768 dimensions under its default setting, while the OpenAI models text-embedding-3-small and text-embedding-ada-002 produce 1536.[2][3] In Apache Hop, the Embed text transform's Include model metadata option adds the model name and the vector width to each row.[4] In an Apache Hop pipeline, mixing vectors from two models in one index produces silently poor results.[4] Keep the model's name beside every stored vector, compare only vectors made by the same model, and re-embed the whole column when the model changes. (Meridian's view)

In Apache Hop, a row whose text is empty or only whitespace is passed on with empty output fields rather than being dropped, and keeps its place in the stream.[4] Decide what an empty text should mean before embedding a column, and filter or fill it on purpose rather than finding the blanks later in a similarity ranking. (Meridian's view) In Apache Hop, embedding calls are billed and rate limited per request, and the transform's Batch size option, the number of rows buffered before the provider is called, is the main throughput control.[4]

The method

In machine learning, embedding is a representation learning technique that maps complex, high-dimensional data into a lower-dimensional vector space of numerical vectors.[5] As a technique, embedding learns its vectors from data such as words, images or user interactions, unlike manually designed methods such as one-hot encoding.[5]

The idea underlying distributional semantics, that "a word is characterized by the company it keeps", was proposed in a 1957 article by John Rupert Firth.[6] The first generation of semantic space models was the vector space model for information retrieval.[6] Reducing the number of dimensions of the vector space model with linear algebraic methods such as singular value decomposition led to latent semantic analysis in the late 1980s.[6] In 2000, Bengio et al., in a series of papers titled Neural probabilistic language models, set out to reduce the high dimensionality of word representations by learning a distributed representation for words.[6] Word2Vec, a technique for obtaining vector representations of words, was developed by Tomáš Mikolov, Kai Chen, Greg Corrado, Ilya Sutskever and Jeff Dean at Google, and presented in 2013.[7]

Two embeddings can be compared by Euclidean distance, by the cosine of the angle between them, or by their dot product.[8] Unlike the cosine, the dot product is proportional to vector length, and examples that appear very frequently in the training set tend to have embedding vectors with large lengths.[8] Once the vectors are normalised to a length of 1, Euclidean distance, cosine and dot product become proportional to each other.[8] In DuckDB, array_cosine_similarity computes the cosine similarity between two arrays of the same size, and the array elements cannot be NULL.[9] Prefer cosine similarity for comparing text embeddings unless the length of a vector is meant to carry weight. (Meridian's view)

A straightforward way to turn word embeddings into a sentence embedding is to compute the average of the word vectors.[1] State of the art sentence embeddings are based on the learned hidden layer representation of dedicated sentence transformer models.[1] Sentence-BERT (SBERT), from Nils Reimers and Iryna Gurevych in 2019, uses siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine similarity.[10]

Compared with the alternatives

A vector space model of words, in its simplest form, results in a very sparse vector space of high dimensionality.[6] Think of an embedding as the dense counterpart of that sparse vector, and reach for it when texts that share no words should still be found alike. (Meridian's view)

A supervised similarity measure, the distance between embeddings, suits large datasets with hard-to-combine features but gives no insight into results, while a manual measure built from feature data does.[8]

Sources

  1. [1]Sentence embedding - Wikipediasecondary source
  2. [2]Embeddings (FastEmbed) - Rapidminer Documentationdocumentation
  3. [3]Embeddings (OpenAI) - Rapidminer Documentationdocumentation
  4. [4]Embed text | Apache Hopdocumentation
  5. [5]Embedding (machine learning) - Wikipediasecondary source
  6. [6]Word embedding - Wikipediasecondary source
  7. [7]Word2Vec - Wikipediasecondary source
  8. [8]Measuring similarity from embeddings | Machine Learning | Google for Developerssecondary source
  9. [9]Array Functions – DuckDBdocumentation
  10. [10][1908.10084] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkssecondary source

Used in

2 Protocols use this operation: