Nonlinear Regression

What is the difference between ordinary regression and nonlinear regression?

The general linear model (Y = XB + E) is called linear because it is linear in its parameters. Nonlinear models are not linear in their parameters. For example, $(y = \beta_0 + \beta_1 x_1 + \beta_2 x_2 + \beta_{12}x_{12} + \beta_{11}x_1^2 + \beta_{22}x_2^2$ is a linear model even though it contains interactions and squared terms, because it is a linear function of the $\beta$ parameters. The model $y = x_1^{\beta_1} + x_2^{\beta_2}$ is a nonlinear model because it is not linear in $\beta_1$ and $\beta_2$. The model $y = e^{\beta_0 + \beta_1 x}$ is nonlinear but can be made linear with a log transformation of the response: $\log(y) = \beta_0 + \beta_1 x$.

To estimate the parameters of nonlinear models, you can use nonlinear least squares regression. In ordinary least squares regression, you find values of the $\beta$ parameters that minimize$\sum\limits_{i=1}^{n} e_i^2 = \sum\limits_{i=1}^{n} (y_i - (\beta_0 + \beta_1 x_i)^2$). The solution is easily found using calculus. In nonlinear least squares regression, you still want to find the model parameters that minimize the sum of squared errors, but in general, you can’t use calculus to solve the minimization problem; you have to use numeric methods instead.

For more information on nonlinear regression, see Fitting Nonlinear Regression Models in JMP - JMP User Community.

Example of nonlinear regression

As an example, consider a problem in civil engineering. When designing a building or other structure, engineers need to account for the forces caused by wind. A key input for estimating these forces is wind speed, which typically increases with height. Near the ground, terrain features such as buildings, trees, and surface roughness affect the wind profile, so the relationship between wind speed and height is nonlinear. Civil engineers use a formula to describe this relationship with parameters that depend on the terrain.

The formula is $U_z = \frac{u_*}{\kappa}\log\left(\frac{z-d}{z_0}\right)$, where

$U_z$ is the mean wind speed (response variable)

$z$ is height above ground (m) (regression variable)

$u_*$ is the friction velocity (m/s)

$k$ is Von Karman’s constant = 0.41

$d$ is zero plane displacement (m) (2/3-3/4 average height of obstacles)

$z_0$ is surface roughness (m).

We can collect data on height above terrain and wind speed and then fit the nonlinear model $U_z = \frac{u_*}{\kappa}\log\left(\frac{z-d}{z_0}\right) = \theta_1 \log(\theta_2 z + \theta_3)$. The model parameters can be collapsed into just three parameters that are used in statistical software. That is, $\theta_1 = \frac{u_*}{\kappa}$, $\theta_2 = \frac{1}{z_0}$, and $\theta_3 =$ -$d/z_0$.

The data are given in the following table.

The solution is given in the figure below. The estimates of the parameters are $\hat{\theta}_1 = 1.65$, $\hat{\theta}_2 = 5.63$, $\hat{\theta}_3 =$ -$142.21$. This means that $\hat{u}_* = \hat{\theta}_1 \kappa = 1.65 \times 0.41 = 0.68$, $\hat{z}_0 = \frac{1}{\hat{\theta}_2} = \frac{1}{5.63} = 0.18$, and $\hat{d} =$ - $\hat{\theta}_2 \hat{\theta}_3 =$ - $5.63 \times $ -$142.21 = 800.64$.

What are some nonlinear least squares algorithms?

Since calculus can’t be used to solve the minimization problem in general, iterative algorithms have been developed. The algorithms used in nonlinear least squares include Gauss-Newton, Quasi-Newton SR-1, and Quasi-Newton BFGS. Each algorithm needs a set of starting values, which then are iteratively updated until some convergence criterion is met.

Gauss-Newton requires calculation of the matrix of partial derivatives of the model with respect to the parameters. However, it does not calculate second partial derivatives; it ignores them instead. Convergence is not guaranteed, the algorithm is sensitive to starting values, and it is not guaranteed to find the global minimum. However, the algorithm is fast and if the initial guess is close to truth, it is likely to converge quickly.

Quasi Newton SR-1 also does not calculate the matrix of second partial derivatives, but it approximates them based on derivatives calculated at two points. With this approximation, convergence is not guaranteed. It also can be slower than other algorithms.

Quasi-Newton BFGS also does not calculate the matrix of second partial derivatives, but it will converge. It is more computationally intensive per iteration, even though it might use fewer iterations overall.

Tips for successful nonlinear modeling

For nonlinear models, convergence often depends on practical modeling choices as much as on the algorithm itself. If a model does not converge or if the solution seems unreasonable, try more than one optimization method or use methods in combination, such as starting with Gauss-Newton and then refining the solution with a Quasi-Newton method. Pay close attention to starting values: use plots, sliders, and parameter profilers to find plausible values, or use a space-filling design to explore a wide range of possible starting values and choose the ones with the smallest sum of squared errors. It can also help to examine the derivatives and simplify the model expression where possible, because cleaner expressions are often easier for numerical optimization methods to evaluate reliably.