TDM 30200 Project 10 - Regularization with Lasso and Ridge

In the previous question, you prepared the Spotify dataset and created training and testing splits. We now turn to fitting linear regression models and interpreting their outputs. The goal of this section is to help you recall how linear regression estimates coefficients and how model fit is evaluated using residuals. Recall that linear regression estimates coefficients using the least squares approach, which selects coefficient values that minimize the residual sum of squares (SSR), defined as

$\text{SSR} = \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$.

Let’s recall how in multiple linear regression, we model the relationship between a response variable $Y$ and a set of predictors $X_1, X_2, \dots, X_p$ using the linear model:

$Y = \beta_0 + \beta_1 X_1 + \cdots + \beta_p X_p + \varepsilon$.

This model is typically fit using least squares. However, sometimes only a subset of the available predictors may be truly associated with the response, and including irrelevant variables adds unnecessary complexity to the model. Feature selection methods aim to address these issues by identifying a subset of predictors that are most relevant to the response, resulting in models that are simpler, more interpretable, and potentially better at generalizing to new data.

Subset Selection Methods

Subset selection methods aim to identify a smaller set of predictors that are most strongly related to the response. Rather than fitting a single full model with all predictors, these methods search over models of varying sizes to balance model fit, complexity, and interpretability.

Forward Stepwise Selection

Forward stepwise selection begins with the null model, which contains no predictors and predicts the sample mean of $Y$. Predictors are then added to the model one at a time. At each step, the predictor that provides the greatest improvement according to a specified criterion is included, conditional on the predictors already in the model. This procedure continues until no remaining predictor improves the model sufficiently, or until a stopping rule is met.

Different criteria can be used to guide forward selection. In earlier work, we explored information-criterion-based approaches such as AIC, which balance model fit and complexity and are motivated by predictive performance.

Alternatively, forward selection can be based on p-values. In this approach, a predictor is added only if its associated p-value is below a chosen significance level (for example, $\alpha = 0.05$), conditional on the predictors already in the model. This emphasizes inferential reasoning, as the p-value measures the strength of evidence against the null hypothesis that a coefficient is equal to zero.

Backward Stepwise Selection

Backward stepwise selection takes the opposite approach. The procedure begins with the full model, which includes all $p$ predictors. At each step, the predictor whose removal leads to the smallest decrease in model performance is removed, according to a specified criterion. As with forward selection, the criterion may be based on an information criterion such as AIC or on p-values. In a p-value-based approach, the predictor with the largest p-value (above a chosen threshold) is removed at each step. Backward stepwise selection continues until all remaining predictors satisfy the stopping rule. Unlike forward selection, backward selection requires that $n > p$, since the full least squares model must be fit at the start.

Stepwise Selection

Stepwise selection is a hybrid approach that combines elements of both forward and backward selection. The procedure typically begins like forward selection by adding predictors one at a time. However, after each new predictor is added, the method checks whether any previously included predictors should be removed based on the chosen criterion. This allows predictors to enter and leave the model dynamically as the selection process proceeds. While stepwise selection can be more flexible than purely forward or backward procedures, it remains a greedy algorithm and does not guarantee identification of the model with the lowest test error.

Ridge regression closely mirrors ordinary least squares, with one important modification: when estimating the coefficients, we add a penalty that discourages large coefficient values. Rather than focusing solely on fitting the data as closely as possible, Ridge regression balances model fit with coefficient size.

Ridge regression estimates the coefficients by minimizing the following objective function:

$\sum_{i=1}^{n} (y_i - \hat{y}_i)^2 + \lambda \sum_{j=1}^{p} \beta_j^2$.

The first term, $\sum_{i=1}^{n} (y_i - \hat{y}_i)^2$, is the sum of squared residuals (SSR) and measures how closely the model’s predictions match the observed data. In the Spotify popularity dataset, this corresponds to how far the predicted popularity values are from the actual popularity scores based on features such as danceability, energy, loudness, and tempo. The second term, $\lambda \sum_{j=1}^{p} \beta_j^2$, is the Ridge penalty. The tuning parameter $\lambda \ge 0$ determines how strongly we penalize large slope coefficients.

When $\lambda = 0$, the penalty term drops out entirely and the objective reduces to $\sum_{i=1}^{n} (y_i - \hat{y}_i)^2$, which is exactly the ordinary least squares criterion. In this case, Ridge regression yields the same coefficient estimates as OLS.

As $\lambda$ increases, large coefficients become increasingly costly in the objective function. To keep the overall objective small, the model is forced to reduce the magnitude of the slope coefficients, shrinking them toward zero. This helps stabilize the model and making it less sensitive to noise in the training data, which helps reduce overfitting.

In a multivariable Spotify popularity model, Ridge regression shrinks all feature coefficients toward zero, but not necessarily by the same amount, depending on their relationship with the response. This occurs because the coefficients must balance two competing goals in the objective function: reducing the sum of squared residuals and keeping the penalty term small. Features that are strongly associated with popularity, such as danceability or energy, contribute substantially to reducing the SSR, so shrinking their coefficients too much would noticeably worsen the model fit. As a result, these coefficients tend to remain relatively larger, while weaker predictors, such as disc number or track number, can be shrunk much more with little impact on the SSR.

Because Ridge regression applies a penalty directly to the size of the coefficients, the scale of the predictor variables becomes important. Features measured on larger numeric scales can lead to larger coefficients, which may be penalized more heavily than features on smaller scales. To ensure that the penalty is applied fairly across all predictors, it is common practice to standardize the features before fitting a Ridge regression model.

Lasso regression is closely related to Ridge regression, but it introduces an important difference in how the model penalizes large coefficients. While both methods apply regularization to reduce overfitting, they do so in different ways, which leads to meaningful differences in model behavior and interpretability.

One key distinction is that Lasso regression has the ability to exclude variables entirely from the model. This means that some predictors can be removed from the final equation, resulting in a simpler and more interpretable model. In contrast, Ridge regression shrinks coefficients toward zero but does not eliminate predictors.

This difference arises from the form of the penalty used by each method. Ridge regression penalizes squared coefficients, while Lasso regression penalizes the absolute value of the coefficients. Instead of minimizing a penalty involving squared slopes, Lasso regression estimates coefficients by minimizing the following objective function:

$\sum_{i=1}^{n} (y_i - \hat{y}_i)^2 + \lambda \sum_{j=1}^{p} |\beta_j|$.

The first term above is the sum of squared residuals, which measures how well the model fits the data, and the second term penalizes large coefficient values using absolute values rather than squares. The tuning parameter $\lambda \ge 0$ controls the strength of this penalty and is typically chosen using cross-validation.

The most important practical difference between Ridge and Lasso regression is how coefficients behave as $\lambda$ increases. Ridge regression can shrink coefficients very close to zero, but they generally remain nonzero. Lasso regression, on the other hand, can force some coefficients to become exactly zero when $\lambda$ is sufficiently large.

As a result, increasing $\lambda$ in a Lasso model can cause some predictors to drop out of the model entirely. This makes Lasso particularly useful in settings where the dataset contains many predictors that contribute little to explaining the response. Because Lasso can remove insignificant variables from the equation, it can be more effective than Ridge regression at reducing variance in models that contain a large number of weak or irrelevant predictors. Like other regularization methods, this comes at the cost of introducing some bias, but the overall bias–variance tradeoff often leads to improved predictive performance.