Decision tree explained: 15 nodes you can read, 8,524 leaves that memorize

A decision tree with no depth limit got every day it learned from right: AUC 1.000 over 74,145 rows. On 2016-2026 days, which it had not seen, it dropped to 0.689, worse than a logistic regression. The same algorithm, stopped at the seventh question, reached 0.862.

This is the fifth part of the series, with the same NASA POWER daily data from seven Chilean cities I used for Random Forest, linear regression, K-Means and logistic regression. The question is the usual one: will it rain at least 1 mm tomorrow? I trained on 1984-2012, chose every parameter on a validation period and evaluated on 2016-2026, inside a container with 8 CPUs.

What is a decision tree?

A decision tree is a chain of questions. The first question splits all days into two groups; each group gets its own question, and so on until a leaf. The leaf returns the share of rainy days that ended there during training, which is used as the probability.

Each question compares a single variable with a threshold: “did it rain 0.44 mm or less today?”. To choose it, the algorithm tries each variable with the candidate cut points taken from its observed values and keeps the split that leaves the two purest groups, measured with the Gini index. That is the idea behind CART, the method Breiman, Friedman, Olshen and Stone published in 1984, and what scikit-learn implements.

At depth 3 the tree has 15 nodes and fits on a page. Its first question is today’s rain. If it rained little, the second one uses latitude to separate the cities from Temuco southward from those from Concepción northward; if it rained, it asks how much. The last questions look at rain again or at the north-south component of wind direction. With those three questions it reached AUC 0.831 in 2016-2026.

Reading it also shows its limit. The fourth day in the example, in Punta Arenas, had 0.92 mm and a wind that sent it to a leaf with 53% rain. It did not rain. Eight leaves are eight possible probabilities, and every day in a leaf gets the same one.

The deeper it asks, the more it memorizes

Nothing forces the tree to stop at three questions. If I let it go on, it splits until every leaf holds days of a single class.

With each extra level, AUC on the training days rises. On validation it rises up to depth 7 and then falls. I chose the depth with that curve, without looking at 2016-2026: the depth-7 tree has 127 leaves and gave AUC 0.862 in 2016-2026.

With no limit, the tree reached depth 31 and 8,524 leaves, under nine training days per leaf on average. Many leaves are pure, so they return probabilities of 0 or 1. When they are wrong, they are wrong with full certainty, and it shows in the log loss: 3.11 against 0.36 for the depth-7 tree.

Capping depth is not the only brake. I tried two more, also chosen on validation.

Requiring at least 100 days per leaf left 478 leaves and AUC 0.863 in 2016-2026. Cost-complexity pruning, which cuts branches that add little (starting from a tree with at least 5 days per leaf), left 167 leaves and AUC 0.861. All three ended in the same place.

The purity criterion mattered little. At depth 8, Gini gave AUC 0.8615 and entropy 0.8655, with the same variable at the root.

The root holds; the predictions move

Trees are known as unstable: change the data a little and the whole tree changes. I measured it in two different ways, because a changing structure is not the same as changing answers.

I resampled the data 50 times by city-year blocks, so neighboring days would not be spread out as if they were independent. The first question was today’s rain in all 50 cases; the threshold moved between 0.40 and 0.71 mm. Training one tree per consecutive block of up to three years (the last one, 2011-2012, has two) gave the same: always today’s rain, with thresholds from 0.39 to 0.97 mm.

The predictions did move. The 50 depth-7 trees did not agree on the rain or no-rain label on 34.8% of 2016-2026 days; the depth-3 ones, on 17.6%. With a different measure, not comparable with those, two trees with no limit trained on random halves of the rows disagreed on 24.1% of days. A random forest exists precisely to average those differences out, as the Random Forest post showed.

Which variables matter? I permuted families of variables together on the validation period, so one correlated variable would not hide another. Shuffling today’s and yesterday’s rain lowered the depth-7 tree’s AUC by 0.133; season and latitude, by 0.059; wind, by 0.056; pressure, by 0.029. Humidity and dew point together barely moved it (0.002): this tree’s predictions hardly depended on them. That describes this tree, not what causes rain.

A regression tree predicts in steps

A tree can also predict numbers. Each leaf returns the mean of its days, so the prediction is a staircase.

To predict tomorrow’s max temperature from today’s, a depth-2 tree has 4 steps and misses by 2.50 °C on average in 2016-2026; at depth 4, 16 steps and 1.89 °C; at depth 8, 252 steps and 1.85 °C. Beyond the 39.5 °C of the warmest training day, the prediction is flat: a tree does not extrapolate. In the Random Forest post I measured the same ceiling on a forest.

How well it calibrates and what it costs

The depth-7 tree gives 118 distinct probabilities. It predicted 23.4% rain on average and it rained on 20.3% of 2016-2026 days. Except in the lowest band, it predicted more rain than fell: on the days it gave 50 to 60%, it rained on 44%.

The tree landed between logistic regression and the forest on almost everything: it ranked better than logistic regression and worse than the forest, trained 7 times slower than logistic regression and 67 times faster than the forest. Its advantage is different: it can be read, and it can be rewritten.

Where a decision tree lives in a real system

|--- precip <= 0.44
|   |--- lat <= -37.79
|   |   |--- wind_cos <= -0.09
|   |   |   |--- class: 0.0
|   |   |--- wind_cos >  -0.09
|   |   |   |--- class: 0.0
|   |--- lat >  -37.79
|   |   |--- precip <= 0.03
|   |   |   |--- class: 0.0
|   |   |--- precip >  0.03
|   |   |   |--- class: 0.0
|--- precip >  0.44
|   |--- precip <= 3.53
|   |   |--- wind_cos <= -0.15
|   |   |   |--- class: 0.0
|   |   |--- wind_cos >  -0.15
|   |   |   |--- class: 1.0
|   |--- precip >  3.53
|   |   |--- wind_cos <= 0.76
|   |   |   |--- class: 1.0
|   |   |--- wind_cos >  0.76
|   |   |   |--- class: 1.0

• Exported rules. A small tree translates to if/else or a SQL CASE and runs where there are no machine learning libraries. Rewritten in plain Python, the depth-3 tree scored one day in 0.0044 ms, against 0.25 ms through scikit-learn.
• Explainable triage. When someone has to justify a decision, the path through the tree is the justification: three questions, three answers.
• Building block of other models. Random forests and boosting combine many trees; understanding one means understanding the piece they repeat.
• Exploration. A depth-2 or depth-3 tree quickly shows which variable separates best, before training something bigger.

When I would choose it: when the rule must be readable or exportable, or as a first look at the data. When I would not: when fine-grained probabilities or the stability of each prediction matter, where a 200-tree forest did better here, or when extrapolation is needed.

Sources

• Breiman, L., Friedman, J. H., Olshen, R. A. and Stone, C. J. (1984). Classification and Regression Trees. Wadsworth. Reissued by Routledge, 2017. DOI 10.1201/9781315139470.
• Quinlan, J. R. (1986). “Induction of decision trees”. Machine Learning, 1(1), 81–106. DOI 10.1007/BF00116251.
• scikit-learn, decision trees, DecisionTreeClassifier and cost-complexity pruning.
• NASA POWER, Daily API and data sources.