How Variable Importance Is Calculated In Random Forest

Estimate and visualize how variable importance is calculated in random forest models using both Mean Decrease in Impurity and Permutation Importance. Enter your model metrics below to see the math, interpret the result, and compare baseline performance against a feature-shuffled scenario.

Formula summary: MDI = feature impurity decrease / all-feature impurity decrease. Permutation Importance = baseline score – permuted score.

How variable importance is calculated in random forest

Variable importance in a random forest tells you how much each predictor helps the ensemble make accurate predictions. It is one of the most useful model interpretation tools in tree based machine learning because it gives a ranked view of which inputs drive the forest’s decisions. Although the phrase sounds simple, there are actually two major families of calculation behind it: impurity based importance and permutation based importance. Knowing the difference matters because the numbers are produced in different ways, can answer slightly different questions, and can behave differently when predictors are correlated.

A random forest is made of many decision trees, each built on a bootstrap sample of the training data and typically using a random subset of candidate features at each split. Every tree makes predictions, and the forest aggregates them. Since a tree repeatedly chooses splits that improve purity or reduce error, we can track how often and how strongly a feature helps. That idea leads directly to Mean Decrease in Impurity, often abbreviated MDI. A second idea is even more intuitive: if a feature is really important, model performance should drop when you randomly shuffle that feature and destroy its relationship with the target. That leads to permutation importance.

Key idea: MDI measures how much a feature improves splits inside the forest. Permutation importance measures how much predictive performance falls when the feature’s information is broken.

The core logic behind Mean Decrease in Impurity

In each decision tree, every split is chosen because it reduces some criterion of disorder. In classification, that criterion is usually Gini impurity or entropy. In regression, it is often variance reduction or mean squared error reduction. Suppose a split uses feature X1. If that split dramatically separates the target values, the impurity reduction is large. If the split barely helps, the reduction is small.

To calculate impurity based importance for one feature across the entire random forest, you sum the impurity decreases produced by all splits that used that feature in all trees. In practice, many software libraries then normalize those totals so the feature importances sum to 1 or 100 percent. This creates an easy ranking. The more total reduction a feature contributes, the more important it is considered under the impurity framework.

The generic form looks like this:

1. For each tree, find every split where the feature was selected.
2. Compute the impurity decrease at that split.
3. Weight the decrease by the number or proportion of samples that reach the node.
4. Sum those weighted decreases across the tree.
5. Average or sum across all trees in the forest.
6. Normalize across all features if desired.

For a classification tree using Gini impurity, the decrease at a node is conceptually:

weighted impurity decrease = parent impurity – left child weighted impurity – right child weighted impurity

If feature X3 was used in many high value splits near the top of trees, it tends to receive a large MDI score. That is why impurity based importance often favors features that repeatedly create strong early splits. Your calculator above reflects this by asking for the total impurity decrease contributed by one feature and the total decrease across all features. Dividing one by the other yields a normalized contribution share.

• Raw MDI total: total impurity decrease attributed to the feature
• Average per tree: raw total divided by number of trees
• Average per split: raw total divided by number of splits where the feature was used
• Normalized MDI percentage: feature decrease divided by all-feature decrease times 100

How permutation importance is calculated

Permutation importance is based on model performance rather than tree splitting statistics. Start with a trained random forest and measure its score on a validation set, test set, or out-of-bag observations. Then randomly shuffle the values of a single feature. This preserves that feature’s marginal distribution but breaks its association with the target and with the other predictors. Run the model again on the permuted data. If the feature was useful, the score will drop. If the feature was not very useful, the score will stay about the same.

The basic formula is straightforward:

Permutation importance = baseline score – permuted score

For example, if baseline accuracy is 0.913 and accuracy after shuffling a feature becomes 0.857, the absolute importance is 0.056. A common relative version divides the drop by the baseline score, giving about 6.13 percent in this case. This method is attractive because it directly answers a practical question: how much predictive signal is lost when this feature’s information is destroyed?

1. Train the random forest.
2. Evaluate the baseline score on held out or out-of-bag data.
3. Shuffle one feature column.
4. Evaluate the score again.
5. Subtract the shuffled score from the baseline score.
6. Repeat multiple times and average for a more stable estimate.

Permutation importance often feels more intuitive to business users because the result is tied to actual predictive performance. If shuffling feature X causes a large drop, that feature clearly matters to the model as currently used. However, if two predictors are highly correlated, shuffling one may not hurt much because the other still carries similar information. In that case, permutation importance can spread importance across a correlated feature set.

Why the two methods can disagree

It is common for MDI and permutation importance to rank features differently. That is not necessarily a problem. They are measuring different things. MDI asks how much a feature improved split quality while the forest was built. Permutation importance asks how much prediction quality worsens after the model is already trained and that feature is disrupted.

Several practical reasons explain the mismatch:

• Correlation between predictors: when features overlap in information, permutation importance may be lower for each individual feature than expected.
• High cardinality bias: impurity based methods can favor variables with many possible split points, such as continuous variables or categorical variables with many levels.
• Split opportunity effects: a feature that is available for many candidate splits may accumulate more impurity decrease simply because it appears often.
• Validation context: permutation importance depends on the evaluation dataset and metric, such as accuracy, AUC, or R-squared.

Dataset	Rows	Features	Target type	Why it matters for importance studies
Iris	150	4	3 class classification	Small, clean benchmark often used to show stable importance rankings.
Wine	178	13	3 class classification	Useful for seeing how chemistry variables separate classes in tree models.
Wisconsin Breast Cancer Diagnostic	569	30	Binary classification	Shows how importance behaves when many measurements are related to one diagnosis.
Ames Housing	2,930	79 explanatory variables	Regression	Helpful for variance reduction and feature importance in larger tabular settings.

The table above uses well known benchmark datasets with real dimensions that are commonly used in machine learning education and experimentation. They illustrate why dataset structure affects importance interpretation. A tiny, low dimensional dataset often yields more stable rankings, while larger datasets with many related predictors can produce broader, more nuanced patterns.

The mathematics of impurity decrease in more detail

Suppose a node contains N observations and has impurity I(parent). It splits into a left child with N_L observations and impurity I(left), and a right child with N_R observations and impurity I(right). The weighted decrease is:

Delta I = I(parent) – (N_L/N) x I(left) – (N_R/N) x I(right)

If that node split uses Feature B, then Feature B receives credit for that Delta I. Across all relevant nodes and trees, the credits are summed. A normalized importance score may then be computed as:

Importance of Feature B = sum of Delta I for Feature B / sum of Delta I for all features

This normalized value is often what practitioners see in random forest software outputs. If Feature B has a normalized importance of 0.202, it means about 20.2 percent of all impurity reduction in the forest was attributed to splits on that feature. That is a relative internal contribution, not a causal effect and not necessarily a direct estimate of how much performance will drop if the feature is removed.

Why bootstrap sampling matters

Random forests rely on bootstrap sampling, where each tree is trained on a sample drawn with replacement from the original dataset. This matters for variable importance because different trees see different versions of the data, and that creates diversity in the splits. It also enables out-of-bag evaluation, which is often used as a convenient internal validation set for permutation importance.

Bootstrap fact	Approximate statistic	Importance interpretation
Unique observations included in one bootstrap sample	About 63.2%	Each tree is trained on a slightly different subset, which stabilizes aggregate importance.
Observations left out of a tree sample	About 36.8%	These out-of-bag cases can be used to estimate prediction error and permutation importance.
Importance stability as tree count rises	Typically improves with hundreds of trees	More trees reduce variance in both split usage patterns and permutation estimates.

Those bootstrap percentages are standard results from sampling with replacement and are central to understanding why random forests are both robust and interpretable. A single tree can be unstable, but averaging across many trees makes variable importance far more reliable.

Common pitfalls when interpreting variable importance

Variable importance is useful, but it is not a substitute for domain knowledge, causal reasoning, or careful validation. Several pitfalls appear frequently in practice.

• Importance is not causality. A feature can be highly predictive without causing the outcome.
• Correlated variables split credit. If two predictors carry nearly the same information, one may appear weaker than expected.
• MDI can favor continuous or high cardinality variables. More potential split points can lead to more opportunities to reduce impurity.
• Permutation scores depend on the chosen metric. Accuracy, AUC, and R-squared may emphasize different aspects of performance.
• Weak validation design can mislead interpretation. If the evaluation set is too small or not representative, importance rankings may fluctuate.

Best practices for a trustworthy importance analysis

1. Use a large enough forest, often hundreds of trees, to reduce variance in rankings.
2. Report both impurity based and permutation based importance when possible.
3. Evaluate permutation importance on held out or out-of-bag data rather than only training data.
4. Repeat permutations multiple times and average the results.
5. Inspect correlated predictors before making high stakes decisions from rankings.
6. Complement global importance with partial dependence, SHAP style analysis, or localized diagnostics when deeper interpretation is required.

When you use the calculator on this page, you are essentially reproducing the core arithmetic behind both major methods. For MDI, you are converting raw impurity reduction into an average per tree, an average per split, and a normalized share of all impurity decrease. For permutation importance, you are measuring the direct drop in model score after the feature has been shuffled. Together, those outputs give a balanced view: one internal to tree construction and one external through model performance.

Which measure should you trust more?

In many applied settings, permutation importance is preferred for communication because it is directly tied to predictive degradation. If the score drops a lot after shuffling, the feature is carrying useful information. However, MDI is still valuable because it is fast, built into many random forest implementations, and can reveal how the forest structurally relied on a feature during training.

A practical recommendation is simple: use MDI for a fast first pass, then confirm key findings with permutation importance. If both methods agree that a variable is dominant, confidence increases. If they disagree sharply, investigate correlation, feature encoding, class imbalance, leakage risk, and the evaluation metric.

Authoritative references for deeper study

For readers who want primary or highly credible educational material, these sources are excellent starting points:

• University of California, Berkeley: Leo Breiman’s Random Forests page
• National Library of Medicine: Bias in random forest variable importance measures
• U.S. Forest Service: Random forest classification and variable importance in applied research

In short, variable importance in random forest is calculated either by accumulating how much each feature reduces impurity across many trees or by measuring how much predictive performance falls when the feature is randomly permuted. Both approaches are useful, both have limitations, and both become much more informative when interpreted in the context of data quality, feature correlation, and the business or scientific question behind the model.