Make Validation Column Utility

Validation is the process of using part of a data set to estimate model parameters and using the other part to assess the predictive ability of a model. With complex data, this can reduce the risk of model overfitting.

A validation column partitions the data into two or three parts.

•	The training set is used to estimate the model parameters.

•	The validation set is used to help choose a model with good predictive ability.

•	The testing set checks the model’s predictive ability after a model has been chosen.

A validation column can be used as a validation method in the Fit Model platform.

Example of the Make Validation Column Utility

The Lipid Data.jmp data table contains blood measurements, physical measurements, and questionnaire data from 95 subjects at a California hospital. You are interested in using a validation column as a way of validation during future analyses.

1.	Select Help > Sample Data Library and open Lipid Data.jmp.

2.	Select Analyze > Distribution.

3.	Assign Gender to the Y, Columns role. Click OK.

Distribution of Gender in Lipid Data.jmp

Distribution of Gender in Lipid Data.jmp illustrates the distribution of Gender in the data set. Notice that there is not an equal proportion of males and females represented. Because there is a scarcity of females within the data, you want to be sure to balance the genders across the validation and training sets.

4.	Select Cols > Modeling Utilities > Make Validation Column.

5.	Click Stratified Random.

6.	Select Gender as the column used for validation holdback.

Click OK.

A Validation column is added to the data table. You can explore the distribution of the validation and training sets by creating a Mosaic Plot.

8.	Select Analyze > Fit Y by X.

9.	Assign Validation to Y, Response, and Gender to the X, Factor.

10.

Click OK.

Distribution of Gender across Validation and Training Sets

Distribution of Gender across Validation and Training Sets illustrates the distribution of Gender across each of the validation and training sets. Note that about 75% of both females and males are in the training set and about 25% of both females and males are in the validation set.

Launch the Make Validation Column Utility

You can launch the Make Validation Column utility in two ways:

•	Select Cols > Modeling Utilities > Make Validation Column.

•	Click Validation in a platform launch window. Note the following:

‒	The platform must support a Validation column.

‒	No columns must be selected in the Select Columns list.

‒	In the data table, a Validation column is added. In the launch window, the new column is assigned to the Validation role.

Note: Launching the Make Validation Column utility through a platform launch window differs from launching it through the Cols menu. The Purely Random and Stratified Random methods are not available. The default settings use 70% of the data as the training set and 30% of the data as the validation set.

Make Validation Column Report

Enter the percentage or number of rows that you would like to include in each training, validation, and testing set. The default uses 75% of the data as the training set and 25% of the data as the validation set.

Make Validation Column also enables you to name the new validation column. The default is Validation.

There are two different methods available to create the different sets.

Purely Random

Partitions the data into sets based on the allocations entered. For example, if the default values are entered, each row has a probability of 0.75 to be included in the training set and 0.25 probability of being included in the validation set. This option is useful for when you have lots of data with a reasonable distribution.

Stratified Random

Partitions the data into sets based on balancing values from specific columns. Using the Stratified Random option adds a column Notes property to the new Validation column in the data table. The Notes indicate how the data were stratified to generate the Validation column. Use this option when you want equal representation of values from a column in each of the training, validation, and testing sets.