Model / explain · All operations

Explain each prediction

A variant of Explain a model

Attribute each scored row's prediction to the input values that pushed it up or down, as with SHAP or local surrogates.

Status

In arcform

Not built yet

arcform does not do this yet. Its place on the roadmap

What other tools call it

Meridian's classification; an independent blind rating agreed on 95.7% of 324 rows (variant, version 3).

KNIME Analytics Platform (1)
RapidMiner Studio (5)
  • Explain Predictionsread 2026-09-27

    For each scored row, show which column values support or contradict the prediction

  • GLM Contributionread 2026-09-27

    Add columns giving each input's contribution (value times coefficient) to a GLM prediction

  • Get Decision Tree Pathread 2026-09-27

    Return the decision path to the leaf for each row scored by a decision tree

  • Local Interpretation (LIME)read 2026-09-27

    Fit local surrogate models around single rows to explain a complex model (LIME)

  • Model Simulatorread 2026-09-27

    Interactive what-if simulator: change model inputs and see prediction and explanation

Yggdrasil Decision Forests (2)
  • analyze_predictionread 2026-09-27

    Explain one row's prediction by attributing it to feature values

  • predict_shapread 2026-09-27

    Compute SHAP values per feature for each row of a dataset

Background

SHapley Additive exPlanations (SHAP) explains one prediction by computing each feature's contribution to it, as Shapley values from game theory.[1][2] Local Interpretable Model-agnostic Explanations (LIME) explains a prediction by learning an interpretable model locally around it.[3][4] Running SHAP for every row gives a matrix with one row per data instance and one column per feature.[1]

When to use it

Understanding the reasons behind a prediction matters for trust when acting on that prediction or when choosing whether to deploy a new model.[3] Reach for a per-row explanation when one decision has to be justified to the person it affects, and pair it with a check of how well the model performs. (Meridian's view)

LIME's Python implementation defines the neighbourhood of a row with an exponential smoothing kernel on the normalised data, whose width is 0.75 times the square root of the number of columns.[4] Changing the kernel width can turn a LIME explanation around, and there is no good way to find the best kernel or width.[4] KernelSHAP, the estimator introduced in the original SHAP paper, is slow, which makes it impractical for computing Shapley values for many rows.[1] SHAP explanations can be created to mislead on purpose and hide biases, so the receiver of a SHAP explanation cannot be sure it is truthful.[1]

The method

In cooperative game theory, the Shapley value fairly distributes a total gain among players by averaging each player's marginal contribution across all possible coalitions.[5] The Shapley value is named after Lloyd Shapley, who introduced it.[5] In SHAP, the feature values of a row act as players in a coalition, and the Shapley values say how to fairly distribute the prediction, the payout, among them.[1] Shapley values for explaining machine learning predictions were first suggested by Štrumbelj and Kononenko in 2011, and Lundberg and Lee proposed SHAP in 2017 as a new way to estimate them.[1] KernelSHAP, TreeSHAP and the permutation method are three ways to estimate Shapley values for explaining predictions.[1] TreeSHAP, proposed by Lundberg, Erion and Lee in 2019, is a fast, model-specific estimator for tree-based models such as decision trees, random forests and gradient-boosted trees.[1] The permutation method ensures that a row's SHAP values add up to its prediction minus the average prediction.[1]

LIME was proposed by Marco Tulio Ribeiro, Sameer Singh and Carlos Guestrin in 2016.[3][4] LIME generates perturbed samples, gets the black box model's predictions for them, and trains an interpretable model weighted by each sample's proximity to the row being explained.[4] For tabular data, LIME creates samples by perturbing each feature individually, drawing from a normal distribution with the feature's mean and standard deviation.[4] The local model in LIME should approximate the predictions well near the row, which is called local fidelity, but it does not have to be a good global approximation.[4]

A decision tree's prediction for a row can be decomposed along its decision path into the mean outcome plus a contribution from each split, which can be summed per feature.[6]

Compared with the alternatives

The big difference between SHAP and LIME is the weighting of sampled instances: LIME weights them by closeness to the original row, SHAP by the weight the coalition would get in the Shapley value estimation.[1] SHAP identifies a class of additive feature importance measures that unifies six existing methods.[2] With SHAP, global interpretations are consistent with the local explanations, since the Shapley values are the atomic unit of the global interpretations.[1]

Sources

  1. [1]18 SHAP – Interpretable Machine Learningsecondary source
  2. [2][1705.07874] A Unified Approach to Interpreting Model Predictionssecondary source
  3. [3][1602.04938] "Why Should I Trust You?": Explaining the Predictions of Any Classifiersecondary source
  4. [4]14 LIME – Interpretable Machine Learningsecondary source
  5. [5]Shapley value - Wikipediasecondary source
  6. [6]9 Decision Tree – Interpretable Machine Learningsecondary source