Statistical FAQs

How do I determine when to stop splitting my tree to avoid overfitting?

JMP does not use a stopping rule. Currently, the platform is purely interactive. However there are several different methods that can help.

The use of p-values for splitting in the Partition node has been implemented in JMP, Version 5.1. When Maximize Significance is chosen, the candidate report includes a new column titled LogWorth. Details are documented in a technical report (Monte Carlo Calibration of Distributions of Partition Statistics).
K fold crossvalidation can be used. This method randomly assigns all the (nonexluded) rows to one of k groups and then calculates crossvalidation statistics.
Hold out samples can be used. The training/validation/test set approach is explained by Bishop (1995, p. 372). The training data set is used to create the tree. The performance of the tree is compared by using an independent validation data set and the tree with the smallest error is selected. The tree should then be confirmed by measuring its performance on a third independent set of data called a test set.

FAQ # 2098
Last Updated: 2005 June 16

919.677.8008 (US)