Decision Trees — Entropy, Gini & Random Forest

A decision tree predicts by asking a sequence of questions such as "completed the practice quiz?" or "income above $50{,}000$?" Building one is a procedure: define what you are predicting, score candidate questions, keep the best, and stop before the tree memorizes noise. In a classification tree, the best question is usually the one that makes the child nodes less mixed than the parent node, which is where entropy and Gini impurity come in.

Use this procedure when you need a model you can read as human-readable rules and the relationship between inputs and outputs is not a straight line.

The Build Procedure, Step By Step

1. Define the target. Decide whether the tree is predicting a class label or a numeric value.
2. Test candidate splits. For classification, compare possible questions by how much they reduce impurity.
3. Pick the best split. Use the split with the strongest impurity reduction under the chosen rule, such as information gain or Gini decrease.
4. Stop before overfitting. Limit depth or require enough samples in each node so the tree does not memorize noise.

Step 2 needs a way to score how mixed a node is. If a node contains class probabilities $p_1, p_2, \dots, p_k$, then one common entropy formula is

$$H = -\sum_{i=1}^k p_i \log_2 p_i$$

This formula is used for classification trees; the base of the logarithm changes the scale but not which split ranks best. Gini impurity is

$$G = 1 - \sum_{i=1}^k p_i^2$$

Both scores are $0$ when a node is perfectly pure and get larger when the classes are more mixed. In practice they often rank candidate splits similarly. Entropy has a direct information-theory interpretation, while Gini is slightly simpler to compute.

For step 3 with entropy, a common rule is information gain:

$$\text{Information Gain} = H(\text{parent}) - \sum_j \frac{n_j}{n} H(\text{child}_j)$$

Here $n$ is the number of samples in the parent node and $n_j$ is the number in child node $j$. For Gini, the idea is parallel: compute the weighted child impurity and prefer the split that reduces it the most. The condition matters: entropy and Gini are standard for classification trees, while a regression tree usually uses variance reduction because the target is numeric.

The Whole Procedure On One Split

Suppose a node contains $6$ training examples for a pass/fail prediction: $3$ Pass and $3$ Fail, so the parent node is evenly mixed.

Its entropy is

$$H_{\text{parent}} = -\frac{3}{6}\log_2\left(\frac{3}{6}\right) - \frac{3}{6}\log_2\left(\frac{3}{6}\right) = 1$$

Its Gini impurity is

$$G_{\text{parent}} = 1 - \left(\frac{3}{6}\right)^2 - \left(\frac{3}{6}\right)^2 = 0.5$$

Now test the split "completed practice quiz?"

• Yes branch: $4$ examples, with $3$ Pass and $1$ Fail
• No branch: $2$ examples, with $0$ Pass and $2$ Fail

For the Yes branch,

$$H_{\text{yes}} = -\frac{3}{4}\log_2\left(\frac{3}{4}\right) - \frac{1}{4}\log_2\left(\frac{1}{4}\right) \approx 0.811$$

and

$$G_{\text{yes}} = 1 - \left(\frac{3}{4}\right)^2 - \left(\frac{1}{4}\right)^2 = 0.375$$

For the No branch, the node is pure, so

$$H_{\text{no}} = 0, \qquad G_{\text{no}} = 0$$

The weighted entropy after the split is

$$\frac{4}{6}(0.811) + \frac{2}{6}(0) \approx 0.541$$

So the information gain is

$$1 - 0.541 \approx 0.459$$

The weighted Gini after the split is

$$\frac{4}{6}(0.375) + \frac{2}{6}(0) = 0.25$$

So the Gini decrease is

$$0.5 - 0.25 = 0.25$$

Both measures say this split is better than leaving the parent node unsplit, because the weighted impurity goes down in both cases.

When One Tree Is Not Enough: Random Forests

A single tree is easy to interpret because it mirrors how people explain decisions: "if this is true, go left; otherwise, go right." It can also capture nonlinear patterns and feature interactions. But it can be unstable, since a small change in the data can produce a noticeably different tree.

A random forest reduces that instability by building many trees instead of one. The usual recipe is:

• sample the training data with replacement for each tree
• consider only a random subset of features at each split
• combine predictions across trees

For classification, the forest usually predicts by majority vote; for regression, it usually averages the tree outputs. The tradeoff is straightforward: a random forest is often more accurate and more stable than a single tree, but it is harder to explain as one clean set of rules.

Where The Procedure Goes Wrong, And How To Check

Treating entropy and Gini as different kinds of prediction

They are split criteria, not separate model families. Self-check: the model is still a decision tree either way.

Forgetting the classification condition

Entropy and Gini are standard for classification trees. Self-check: if the target is numeric, switch to a variance-based or error-based rule.

Chasing perfect purity too deeply

If you keep splitting until every leaf is nearly perfect on the training set, the tree may overfit. Self-check: confirm step 4's stopping rule (depth limit, minimum leaf size, or pruning) is actually in place.

Assuming a random forest explains itself

A forest often predicts better but is less transparent. Self-check: if interpretability is the main requirement, one carefully controlled tree may be the better tool.

When To Use A Decision Tree Or Random Forest

Decision trees appear in classification and regression tasks across finance, medicine, operations, marketing, and many other applied settings. They are useful when the relationship between inputs and outputs is not well described by a straight-line model and when rule-like explanations matter. Use a single tree when interpretability matters most and you need to inspect the decision path; use a random forest when prediction quality and stability matter more than having one compact tree you can read line by line.

To make the split logic stick, take a small labeled dataset with two classes, test two possible first splits, and compute the weighted entropy or weighted Gini for each. Solving one small case by hand is often the fastest way to internalize the whole procedure.