Model / stats · All operations

Estimate density from data

A variant of Use a probability distribution

Fit a density to a sample, by kernels or a density tree, and read it at query points.

Status

In arcform

Runs as a SQL step

arcform carries it as a sql: step today.

The DuckDB step

Estimate a density by Gaussian kernels and by a density tree, and read it at query points

SQL
CREATE TEMP TABLE sample AS SELECT x::DOUBLE AS x FROM (VALUES (1.0), (2.0), (2.5), (3.0), (7.0), (8.0), (8.5), (15.0)) t(x);
CREATE TEMP TABLE query_points AS SELECT q::DOUBLE AS q FROM (VALUES (2.0), (4.0), (7.5), (12.0), (20.0)) t(q);
-- kernels: every query point against the whole sample, so a cross join and an aggregate; bandwidth 1.0
CREATE TEMP TABLE kernel AS
SELECT q.q, sum(exp(-0.5 * power((q.q - s.x) / 1.0, 2)) / sqrt(2 * pi())) / (count(*) * 1.0) AS density
FROM query_points q CROSS JOIN sample s GROUP BY q.q;
-- density tree: split the domain [0, 16) at the median of the points in each part, twice; a leaf has density count / (n * width)
CREATE TEMP TABLE leaves AS
WITH RECURSIVE tree(lo, hi, depth) AS (
  SELECT 0.0::DOUBLE, 16.0::DOUBLE, 0
  UNION ALL
  SELECT c.lo, c.hi, p.depth + 1
  FROM (SELECT t.lo, t.hi, t.depth, median(s.x) AS mid FROM tree t JOIN sample s ON s.x >= t.lo AND s.x < t.hi WHERE t.depth < 2 GROUP BY t.lo, t.hi, t.depth) p,
       LATERAL (VALUES (p.lo, p.mid), (p.mid, p.hi)) c(lo, hi))
SELECT lo, hi, (SELECT count(*) FROM sample s WHERE s.x >= lo AND s.x < hi) / ((SELECT count(*) FROM sample) * (hi - lo)) AS density
FROM tree WHERE depth = 2;
CREATE TEMP TABLE result AS
SELECT k.q, k.density AS kernel_density, coalesce(l.density, 0) AS tree_density
FROM kernel k LEFT JOIN leaves l ON k.q >= l.lo AND k.q < l.hi;
SELECT * FROM result ORDER BY q;
-- the kernel values are the closed form sum of normal densities at these points; the tree has four leaves of two points each, whose
-- densities times widths sum to 1, and a point outside the domain has density 0
SELECT CASE WHEN ((SELECT list(round(kernel_density, 9) ORDER BY q) = [0.154368819, 0.0543116, 0.118264870, 0.000679981, 0.000000186] FROM result)
  AND (SELECT list(round(tree_density, 9) ORDER BY q) = [0.111111111, 0.090909091, 0.076923077, 0.032258065, 0.0] FROM result)
  AND (SELECT count(*) = 4 AND abs(sum(density * (hi - lo)) - 1) < 1e-12 FROM leaves)) THEN 'as expected' ELSE error('the kernel and tree densities at 2, 4, 7.5, 12 and 20 should match the reference values, and the four leaves should hold the whole sample') END AS verdict;
  • The statement builds its own data.
  • It needs nothing installed beyond DuckDB itself.
  • The statement checks its own result and raises an error when it is wrong: “the kernel and tree densities at 2, 4, 7.5, 12 and 20 should match the reference values, and the four leaves should hold the whole sample”.
  • It passed on DuckDB v2.0.0 (Cyanoptera) d54cbbe3df.

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

mlpack (2)
  • mlpack_detread 2026-09-27

    Density estimation trees: train one, or predict density at query points

  • mlpack_kderead 2026-09-27

    Kernel density estimation at query points, with dual-tree algorithms

Background

Density estimation constructs an estimate, based on observed data, of an unobservable underlying probability density function.[1] The data are usually thought of as a random sample from the population whose density is being estimated.[1] The most basic form of density estimation is a rescaled histogram.[1]

When to use it

Density estimates can give a valuable indication of such features as skewness and multimodality in the data.[1] Density estimation is frequently used in anomaly detection or novelty detection: an observation in a very low-density region is likely to be an anomaly or a novelty.[1] Density estimates suit the presentation of conclusions to a client because they are fairly easily comprehensible to non-mathematicians.[1]

Kernel density estimation (KDE) is a non-parametric method to estimate the probability density function of a random variable based on kernels as weights.[2] In KDE, the bandwidth is a smoothing parameter, and there is always a trade-off between the bias of the estimator and its variance.[2] The KDE bandwidth is a free parameter which exhibits a strong influence on the resulting estimate.[2] In an example of KDE on a sample from the standard normal distribution, a bandwidth of 0.05 is too small and leaves spurious data artifacts, while a bandwidth of 2 oversmooths and obscures much of the underlying structure.[2] Silverman's rule of thumb for the KDE bandwidth is easy to compute but can yield widely inaccurate estimates when the density is not close to being normal.[2] Review studies comparing automatic bandwidth selectors for KDE reached a general consensus that plug-in selectors and cross-validation selectors are the most useful over a wide range of data sets.[2] Bandwidth selection for KDE of heavy-tailed distributions is relatively difficult.[2] Draw the estimate at more than one bandwidth before reading peaks or gaps from it. (Meridian's view)

In mlpack, kernel density estimation applies a kernel function to each reference point for each query point, a computation of complexity O(N^2) for N query points and N reference points.[3] On a large table, read the estimate at a modest set of query points rather than at every row. (Meridian's view)

The method

Commonly used kernels include the uniform, triangular, biweight, triweight, Epanechnikov (parabolic) and normal kernels.[2] The Epanechnikov kernel is optimal in a mean square error sense, though the loss of efficiency is small for the other commonly used kernels.[2] Due to its convenient mathematical properties, the normal kernel is often used.[2] In a KDE with a Gaussian kernel, a Gaussian density function is placed at each data point, and the sum of the density functions is computed over the range of the data.[1] In some fields such as signal processing and econometrics, KDE is also termed the Parzen–Rosenblatt window method, after Emanuel Parzen and Murray Rosenblatt, who are usually credited with independently creating it in its current form.[2] Under weak assumptions, no non-parametric estimator can converge at a faster rate than the kernel estimator, though its n−4/5 rate, for a sample of size n, is slower than the typical n−1 convergence rate of parametric methods.[2] When the bandwidth varies with the location of either the estimate or the samples, the method is termed adaptive or variable bandwidth kernel density estimation.[2]

The mlpack KDE program uses an approximate dual-tree or single-tree algorithm for acceleration.[3] The mlpack KDE program lets the user specify the maximum relative error tolerance for each query value.[3]

Parikshit Ram and Alexander G. Gray of the Georgia Institute of Technology developed density estimation trees (DETs), the natural analog of classification trees and regression trees for density estimation, in a 2011 paper.[4] A DET is a piecewise constant estimator structured as a decision tree, learned by minimising the integrated squared error.[4] In a DET, the density estimate in a leaf is the number of observations in the leaf divided by the product of the total number of observations and the volume of the leaf.[4] DETs empirically exhibit the interpretability, adaptability and feature selection properties of supervised decision trees while incurring a slight loss in accuracy over other nonparametric density estimators.[4] The query time of a DET is of the order of the depth of the tree, which for N observations is of order N in the worst case but empirically much closer to log N in practice.[4] In mlpack, the optimal DET can be trained on a set of data using cross-validation.[3] The mlpack DET program can also provide density estimates for a set of test points.[3]

Compared with the alternatives

Kernel density estimates are closely related to histograms, but can be endowed with properties such as smoothness or continuity by using a suitable kernel.[2] The average shifted histogram is an alternative to kernel density estimation that is fast to compute and gives a smooth curve estimate of the density without using kernels.[5] Histograms are fast nonparametric estimators during training as well as query time, but they lack the adaptability of decision trees and become prohibitively inaccurate as the number of dimensions increases.[4] DETs are similar to variable bin-width histograms, but are restricted to a hierarchical partitioning of the data.[4]

Mixtures of Gaussians are widely used parametric density estimators, because of their interpretability as clusters and their simple implementation.[4] Nonparametric density estimators have been shown to be significantly more accurate than parametric ones and require fewer assumptions on the data.[4]

Sources

  1. [1]Density estimationsecondary source
  2. [2]Kernel density estimationsecondary source
  3. [3]mlpack documentationdocumentation
  4. [4]Density Estimation Treessecondary source
  5. [5]Histogramsecondary source