Model / explain · All operations

Explain a model

Variable importance, partial dependence, SHAP, model inspection.

Status

In arcform

Not built yet

arcform does not do this yet. Its place on the roadmap

Shape of the result
a table describing the input rather than holding its rows
Level
needs an arcform operator

Variants

Rank inputs by importance

Measure how much each input column contributes to a trained model across a dataset.

What other tools call it

Meridian's classification; an independent blind rating agreed on 86.2% of 312 rows (exact class, version 2).

Alteryx Designer (2)
  • Model Coefficientsread 2026-09-27

    Extract coefficient names and values from a regression model into a table

  • Variance Inflation Factorsread 2026-09-27

    Report variance inflation factors for each variable of a regression model

KNIME Analytics Platform (6)
RapidMiner Studio (11)
  • Cluster Model Visualizerread 2026-09-27

    Visual summary of a centroid cluster model: heat map, centroid charts, cluster descriptions

  • Create Formularead 2026-09-27

    Extract the prediction formula from a model that can express one

  • Explain Predictionsread 2026-09-27

    For each scored row, show which column values support or contradict the prediction

  • Extract Cluster Prototypesread 2026-09-27

    Output the centroids or prototypes of a cluster model as a table

  • GLM Contributionread 2026-09-27

    Add columns giving each input's contribution (value times coefficient) to a GLM prediction

  • Generate Partial Dependency Plot Dataread 2026-09-27

    Compute partial dependence data showing one column's influence on a model's prediction

  • Get Decision Tree Pathread 2026-09-27

    Return the decision path to the leaf for each row scored by a decision tree

  • Local Interpretation (LIME)read 2026-09-27

    Fit local surrogate models around single rows to explain a complex model (LIME)

  • Model Simulatorread 2026-09-27

    Interactive what-if simulator: change model inputs and see prediction and explanation

  • Visualize Model by SOMread 2026-09-27

    Plot a model's predictions over a 2-D self-organising map of the data

  • Weight by Tree Importanceread 2026-09-27

    Derive column importance from the split points of a trained random forest

Yggdrasil Decision Forests (15)
  • GBT model accessors (group)read 2026-09-27

    3 accessors: num_trees_per_iteration, early_stopping_triggered, activation

  • analyzeread 2026-09-27

    Report variable importances, partial dependence, conditional expectation and SHAP over a dataset

  • analyze_model_and_datasetread 2026-09-27

    Write an HTML report interpreting a model against a dataset

  • analyze_predictionread 2026-09-27

    Explain one row's prediction by attributing it to feature values

  • compute_variable_importancesread 2026-09-27

    Compute permutation importances on a dataset and store them in the model's metadata

  • describeread 2026-09-27

    Render a text or HTML description of the model, its features, trees and training logs

  • get_tree / get_all_trees / iter_treesread 2026-09-27

    Read trees out of the model as editable Python objects

  • in_bag_example_indicesread 2026-09-27

    Return the row indices bootstrap-sampled to train one tree, replayed from the training seed

  • num_examples_per_treeread 2026-09-27

    Return how many rows were sampled to grow each isolation tree

  • num_trees / num_nodesread 2026-09-27

    Count the trees and nodes in the model

  • predict_shapread 2026-09-27

    Compute SHAP values per feature for each row of a dataset

  • print_treeread 2026-09-27

    Print one tree as indented text

  • show_modelread 2026-09-27

    Print a model's features, label, variable importances, structure, engines and training logs

  • variable_importancesread 2026-09-27

    Return each feature's importance under every measure the model stores

  • winner_takes_allread 2026-09-27

    Report whether trees vote one class each or average their class distributions

Background

Explainable AI explores methods that bring out the reasoning behind a model's decisions or predictions, to make them more understandable and transparent.[1] Interpretation methods applied after a model is trained can be model-agnostic, such as permutation feature importance, or model-specific.[2] Local methods explain individual predictions, while global methods describe how features affect predictions on average.[2]

When to use it

Explainability is useful for ensuring that a model is not making decisions based on irrelevant or otherwise unfair criteria.[1] A model can learn tricks that satisfy its training goal without reflecting what its designers wanted, as a 2017 image recognition system did when it learned to look for a copyright tag associated with horse pictures.[1] In scikit-learn, the inspection tools are there to help understand a model's predictions and what affects them, to evaluate its assumptions and biases, design a better model or diagnose performance issues.[3]

A feature of low importance for a bad model could be very important for a good model, so a model's predictive power should be evaluated on held-out data before importances are computed.[4] Permutation importance computed on the training data can falsely highlight irrelevant features as important when the model overfits.[5] Permutation importance does not reflect a feature's intrinsic predictive value, only how important the feature is for a particular model.[4] When two features are correlated and one is permuted, the model still has access to it through the other, which lowers the reported importance of both, though both might be important.[4] Like all model-agnostic methods, permutation feature importance has a problem when features are dependent, because shuffling produces unrealistic or at least unlikely data points.[5]

Often several models perform about equally well but give different interpretations, which is called the Rashomon effect.[2] Treat an explanation as a description of one model, and check that the model predicts well on data it has not seen before reading anything about the world into it. (Meridian's view)

The method

Model-agnostic methods ignore what is inside the model and analyse only how its output changes when its inputs change.[2] A model-agnostic method samples from the data, intervenes on it, gets the model's predictions for the changed data and aggregates the results.[2] Partial dependence plots, SHapley Additive exPlanations (SHAP), permutation feature importance and Local Interpretable Model-Agnostic Explanations (LIME) are among the popular techniques for classification and regression models.[1]

Permutation feature importance measures how much a model's error grows when one feature's values are shuffled, which breaks the relationship between the feature and the outcome.[5][4] Breiman introduced the permutation measure for random forests in 2001, and Fisher, Rudin and Dominici later proposed a model-agnostic version, which they called model reliance.[5] In scikit-learn, a feature's permutation importance is the model's reference score minus its mean score over K repeated shuffles of that feature's column.[4] In scikit-learn, repeating the shuffles with different permutations also gives a measure of the variance in the estimated importances.[4] In scikit-learn, permutation importance depends on the scoring function chosen, and different metrics might lead to significantly different importances, in particular for imbalanced classification problems.[4]

Compared with the alternatives

Tree-based models provide another importance measure, based on the mean decrease in impurity (MDI) at the trees' splits.[4] MDI can give high importance to features that may not be predictive on unseen data when the model is overfitting, which permutation importance avoids because it can be computed on unseen data.[4] Impurity-based importance for trees is strongly biased towards high-cardinality features, typically numerical ones, over binary features or categorical features with few categories.[4] Permutation feature importance is based on the decrease in model performance, while SHAP feature importance is based on the magnitude of feature attributions, averaged in absolute value per feature.[6] To understand which features a model actually used, SHAP importance and partial dependence-based importance are alternatives that do not rely on error measures.[5]

Sources

  1. [1]Explainable artificial intelligence - Wikipediasecondary source
  2. [2]4 Methods Overview – Interpretable Machine Learningsecondary source
  3. [3]5. Inspection — scikit-learn 1.9.1 documentationdocumentation
  4. [4]5.2. Permutation feature importance — scikit-learn 1.9.1 documentationdocumentation
  5. [5]23 Permutation Feature Importance – Interpretable Machine Learningsecondary source
  6. [6]18 SHAP – Interpretable Machine Learningsecondary source