17.1 Introduction to Non-Linear Regression

Non-linear Regression in Linear Modeling

Non-linear regression models relationships between variables that don't follow a straight line. It's the tool you reach for when linear models fall short, such as modeling population growth, drug concentration over time, or enzyme activity. These models can capture curves, asymptotes, and rates of change that shift depending on where you are in the data.

Fitting non-linear models is more involved than fitting linear ones. You need to choose the right function, supply good starting values for the algorithm, and watch out for multiple local optima that can trap your optimizer. Interpreting results takes more care, and you typically need more data. But when the underlying relationship truly is non-linear, these models give you far more accurate and meaningful results.

Non-linear Regression

Definition and Characteristics

Non-linear regression is a statistical technique for modeling the relationship between a dependent variable and one or more independent variables when that relationship is not linear in the parameters. That distinction matters: a model like $y = \beta_0 + \beta_1 x^2$ is still linear regression because the parameters ($\beta_0$, $\beta_1$) enter the equation linearly. A model like $y = \beta_0 e^{\beta_1 x}$ is genuinely non-linear because $\beta_1$ sits inside the exponent and can't be isolated with simple algebra.

• Parameters are estimated by minimizing a loss function, typically the sum of squared residuals, using iterative optimization algorithms
  • Common algorithms include Gauss-Newton and Levenberg-Marquardt. Both work by repeatedly updating parameter guesses until the residuals stop improving (convergence).
• Non-linear models can take many functional forms depending on the relationship between the variables:
  • Exponential ($y = ae^{bx}$): growth or decay processes
  • Logarithmic ($y = a + b\ln(x)$): diminishing-return relationships
  • Power ($y = ax^b$): allometric scaling
  • Sigmoidal/Logistic ($y = \frac{L}{1 + e^{-k(x - x_0)}}$): S-shaped growth with an upper bound
• The choice of function should be driven by domain knowledge, theoretical reasoning, or clear empirical patterns in the data, not just curve-fitting convenience.

Appropriate Situations for Non-linear Regression

Non-linear regression is appropriate when the relationship between variables can't be adequately described by a straight line. A good rule of thumb: if the rate of change in your dependent variable itself changes depending on the level of the independent variable, you likely need a non-linear model.

• Saturation and carrying capacity: Population growth slows as the population approaches the environment's carrying capacity. A logistic model captures this; a linear model would predict unlimited growth.
• Exponential growth or decay: Radioactive decay, compound interest, and early-stage epidemic spread all follow exponential patterns.
• Asymptotic behavior: Drug concentration in the bloodstream rises quickly after administration, then levels off at a plateau.

Common applications across fields:

• Biology: Logistic growth models for populations
• Pharmacokinetics: Modeling drug concentration over time
• Biochemistry: The Michaelis-Menten equation for enzyme kinetics ($v = \frac{V_{max}[S]}{K_m + [S]}$)
• Economics: Cobb-Douglas production functions modeling diminishing returns

Linear Regression vs. Non-linear Relationships

Limitations of Linear Regression

Linear regression assumes a constant rate of change in the dependent variable for each unit change in the independent variable. When the true relationship is curved, this assumption breaks down in several ways:

• Biased parameter estimates: Fitting a line to curved data systematically over-predicts in some regions and under-predicts in others. The resulting coefficients don't reflect the actual relationship.
• Missed structural features: Linear models can't capture curvature, asymptotes, or inflection points. For example, the effect of fertilizer on crop yield may be strong at low doses but diminish at high doses. A linear model would average this out and miss the pattern entirely.
• Dangerous extrapolation: Extending a linear fit beyond the observed data range can produce nonsensical predictions. A linear model of drug concentration might predict negative concentrations at later time points, which is physically impossible.

Residual plots are your best diagnostic here. If you fit a linear model and the residuals show a clear curved pattern rather than random scatter, that's strong evidence you need a non-linear approach.

Challenges of Non-linear Modeling

Model Specification and Parameter Estimation

Unlike linear regression, where the functional form (a line) is given, non-linear regression requires you to specify the function before fitting. This introduces several difficulties:

1. Choosing the right function: You must decide a priori whether the relationship is exponential, logistic, power-law, or something else. Picking the wrong form leads to model misspecification, and the data alone won't always make the correct choice obvious.
2. Starting values matter: Iterative algorithms need initial guesses for each parameter. Poor starting values can cause the algorithm to converge slowly, fail to converge at all, or settle on a bad solution. Good strategies include using domain knowledge, plotting the data first, or fitting a simpler approximation to get ballpark estimates.
3. Multiple local optima: The loss surface for non-linear models can have several valleys (local minima). The algorithm may find a local minimum rather than the global minimum, giving you parameter estimates that aren't truly optimal. Running the optimization from multiple different starting values helps you check whether you've found the best solution.

Interpretation and Sample Size Requirements

• Parameter interpretation is more complex than in linear regression. In a linear model, each coefficient has a fixed meaning (the slope is the same everywhere). In a non-linear model, the effect of a predictor changes across the data range. For instance, the slope of a logistic growth curve is steepest near the inflection point and nearly flat near the asymptotes.
• Non-linear models generally require larger sample sizes than comparable linear models. You need enough data points to pin down the curvature, locate asymptotes, and estimate all parameters with reasonable precision. Sparse data in critical regions of the curve (near inflection points or asymptotes) can make estimates unstable.