Decision Trees — Entropy, Gini & Random Forest

A decision tree predicts by asking a sequence of questions such as "completed the practice quiz?" or "income above $50{,}000$?" In a classification tree, the best question is usually the one that makes the child nodes less mixed than the parent node. That is where entropy and Gini impurity come in.

Random forests use the same basic idea, but they average many trees instead of trusting one tree on its own. If you only need the core idea, remember this: entropy and Gini help a tree choose splits, and a random forest helps reduce the instability of a single tree.

Decision Tree Entropy And Gini: What They Measure

Entropy and Gini impurity are both ways to score how mixed a classification node is.

If a node contains class probabilities $p_1, p_2, \dots, p_k$, then one common entropy formula is

$$H = -\sum_{i=1}^k p_i \log_2 p_i$$

This formula is used for classification trees. The base of the logarithm changes the scale, but it does not change which split ranks best.

Gini impurity is

$$G = 1 - \sum_{i=1}^k p_i^2$$

Both scores are $0$ when a node is perfectly pure. Both get larger when the classes are more mixed.

In practice, entropy and Gini often rank candidate splits similarly. Entropy has a direct information-theory interpretation, while Gini is slightly simpler to compute.

How A Decision Tree Chooses A Split

For entropy, a common rule is information gain:

$$\text{Information Gain} = H(\text{parent}) - \sum_j \frac{n_j}{n} H(\text{child}_j)$$

Here, $n$ is the number of samples in the parent node and $n_j$ is the number in child node $j$.

For Gini, the idea is parallel: compute the weighted child impurity and prefer the split that reduces it the most.

The condition matters: entropy and Gini are standard for classification trees. A regression tree usually uses a different rule, such as variance reduction, because the target is numeric rather than categorical.

Worked Example: Entropy And Gini For One Split

Suppose a node contains $6$ training examples for a pass/fail prediction:

• $3$ are Pass
• $3$ are Fail

So the parent node is evenly mixed.

Its entropy is

$$H_{\text{parent}} = -\frac{3}{6}\log_2\left(\frac{3}{6}\right) - \frac{3}{6}\log_2\left(\frac{3}{6}\right) = 1$$

Its Gini impurity is

$$G_{\text{parent}} = 1 - \left(\frac{3}{6}\right)^2 - \left(\frac{3}{6}\right)^2 = 0.5$$

Now test the split "completed practice quiz?"

• Yes branch: $4$ examples, with $3$ Pass and $1$ Fail
• No branch: $2$ examples, with $0$ Pass and $2$ Fail

For the Yes branch,

$$H_{\text{yes}} = -\frac{3}{4}\log_2\left(\frac{3}{4}\right) - \frac{1}{4}\log_2\left(\frac{1}{4}\right) \approx 0.811$$

and

$$G_{\text{yes}} = 1 - \left(\frac{3}{4}\right)^2 - \left(\frac{1}{4}\right)^2 = 0.375$$

For the No branch, the node is pure, so

$$H_{\text{no}} = 0, \qquad G_{\text{no}} = 0$$

The weighted entropy after the split is

$$\frac{4}{6}(0.811) + \frac{2}{6}(0) \approx 0.541$$

So the information gain is

$$1 - 0.541 \approx 0.459$$

The weighted Gini after the split is

$$\frac{4}{6}(0.375) + \frac{2}{6}(0) = 0.25$$

So the Gini decrease is

$$0.5 - 0.25 = 0.25$$

Both measures say this split is better than leaving the parent node unsplit, because the weighted impurity goes down in both cases.

Why Decision Trees Make Sense Intuitively

A tree is easy to read because it mirrors the way people often explain decisions: "if this is true, go left; otherwise, go right." That makes trees useful when you need a model that can be inspected, explained, or turned into human-readable rules.

They are also flexible. A tree can capture nonlinear patterns and feature interactions without forcing one global equation onto the whole dataset.

Why Random Forests Often Work Better

A single tree is easy to interpret, but it can be unstable. A small change in the data can produce a noticeably different tree.

A random forest reduces that instability by building many trees instead of one. The usual recipe is:

• sample the training data with replacement for each tree
• consider only a random subset of features at each split
• combine predictions across trees

For classification, the forest usually predicts by majority vote. For regression, it usually averages the tree outputs.

The tradeoff is straightforward. A random forest is often more accurate and more stable than a single tree, but it is harder to explain as one clean set of rules.

Common Mistakes With Decision Trees

Treating Entropy And Gini As Different Kinds Of Prediction

They are split criteria, not separate model families. The model is still a decision tree either way.

Forgetting The Classification Condition

Entropy and Gini are standard for classification trees. If the target is numeric, the tree usually uses a variance-based or error-based rule instead.

Chasing Perfect Purity Too Deeply

If you keep splitting until every leaf is nearly perfect on the training set, the tree may overfit. Depth limits, minimum leaf sizes, or pruning are there for a reason.

Assuming Random Forest Explains Itself

A forest often predicts better, but it is less transparent than a single tree. If interpretability is the main requirement, one carefully controlled tree may still be the better tool.

When To Use A Decision Tree Or Random Forest

Decision trees appear in classification and regression tasks across finance, medicine, operations, marketing, and many other applied settings. They are useful when the relationship between inputs and outputs is not well described by a straight-line model and when rule-like explanations matter.

Use a single tree when interpretability matters most and you need to inspect the decision path. Use a random forest when prediction quality and stability matter more than having one compact tree you can read line by line.

Try A Similar Problem

Take a small labeled dataset with two classes and test two possible first splits. Compute the class proportions in each child node, then compare the weighted entropy or weighted Gini. Solving one small case by hand is often the fastest way to make the split logic stick.