Model Tuning
What is model tuning?
Model tuning is finding the values of model hyperparameters that give you the best model performance. Examples of hyperparameters include the number of nodes in a neural network model or the learning rate in a gradient boosted model.
Software, like JMP, will often give good starting values for hyperparameters. For some problems, finding optimal values can lead to greatly improved predictions.
Design of experiments for model tuning
To find the best values of the hyperparameters, you can run the model using different values of the hyperparameters, selected using a designed experiment. It is common to use a space filling design for the hyperparameters rather than an optimal design or classic factorial-type design. A space filling design is often used in simulation experiments or computer experiments where either no random noise is expected or the purpose is to explore the factor space. Common space filling designs used to tune predictive models are the Latin hypercube design or the fast flexible filling design.
How do I tune a boosted random forest model?
Tuning a random forest model involves finding the best parameter values used for building a tree. These parameters will be problem-specific. Possible parameters include:
- Number of layers used in boosting
- Splits per tree
- Learning rate
- Minimum size split
- Row sampling rate
- Column sampling rate
Let’s create a tuning table for a boosted random forest model using the Percent Recovered response in the Recovery data (using the file found in our introduction to predictive modeling lesson). In this simple example, we’ll vary the number of layers, splits per tree, and learning rate. We choose to make 30 runs.
The R-Square and RASE on the validation data are displayed. The best fit model has four splits per tree, a learning rate of 0.048, and 402 layers. R-Square on the validation set is 0.827. This run gives the best performance according to R-Square.
The beauty of designing an experiment is that we can fit a model to these 30 runs and use the model to predict settings of the hyperparameters to give best performance. Let’s fit a Gaussian process model to the experimental data.
Next, let’s run a boosted tree using the predicted best values of the hyperparameters.
How do I tune a neural network model?
Tuning a neural model involves finding the best values of the model hyperparameters to optimize fit statistics on the validation set. Possible hyperparameters include:
- Number of hidden layers.
- Number of nodes.
- Activation functions.
- Number of models in boosting.
- Learning rate in boosting.
- Number of tours, which controls how many times the algorithm selects starting values for the coefficients.
Let’s create a tuning table for a boosted neural model using the Percent Recovered response in the Recovery data. In this simple example, we’ll vary the number of TanH nodes, number of models in boosting, and the boosting learning rate. We make 30 runs.
You’ll notice that the best run has a validation R-square of 0.879. We can fit a model to these data using Validation R2 as a response then use the model to determine the best settings of the boosted neural hyperparameters.
Next, let’s run a boosted neural network using the predicted best values of the hyperparameters.
