Weighted Least Squares Regression in R

Weighted Least Squares (WLS) regression is an extension of the ordinary least squares (OLS) regression that allows for the possibility of heteroscedasticity, or non-constant variance, in the errors. In WLS, observations can be weighted differently, making it a useful tool when the variance of the residuals is not constant across levels of an independent variable.

In this comprehensive guide, we’ll explore the fundamentals of WLS, the rationale behind it, and demonstrate how to implement it in R.

Table of Contents

1. Basics of Weighted Least Squares
2. Rationale Behind WLS
3. Implementing WLS in R
4. Assessing the Model
5. Advantages and Limitations
6. Conclusion

1. Basics of Weighted Least Squares

Ordinary least squares minimizes the sum of the squared residuals to estimate the coefficients. In contrast, weighted least squares minimizes the sum of the weighted squared residuals. The WLS regression equation is:

Here, each observation has a weight, and these weights determine how much influence each observation has on the estimated coefficients.

2. Rationale Behind WLS

Heteroscedasticity is a situation where the variance of the residuals is not constant across levels of an independent variable. This violation of the constant variance assumption of OLS can lead to inefficient and biased parameter estimates. WLS is one method to tackle heteroscedasticity by giving less weight to observations with larger errors.

3. Implementing WLS in R

Step 1: Diagnosing Heteroscedasticity

Before implementing WLS, it’s essential to diagnose heteroscedasticity. A residual vs. fitted plot can help in this.

model.ols <- lm(mpg ~ wt + hp, data=mtcars)
plot(model.ols$fitted.values, residuals(model.ols), main="Residual vs. Fitted Values")
abline(h=0, col="red")

Step 2: Deciding on Weights

A common method to determine weights when dealing with heteroscedasticity is to use the inverse of the squared fitted values from an initial OLS regression.

weights <- 1 / fitted(model.ols)^2

Step 3: Implementing WLS in R

Now, with the weights determined, we can perform WLS using the lm() function:

model.wls <- lm(mpg ~ wt + hp, data=mtcars, weights=weights)
summary(model.wls)

4. Assessing the Model

Once the WLS regression is performed, it’s essential to check if the heteroscedasticity issue has been addressed:

plot(model.wls$fitted.values, residuals(model.wls), main="WLS: Residual vs. Fitted Values")
abline(h=0, col="red")

If the plot shows a more random scatter (without a pattern), it suggests that the WLS has tackled the heteroscedasticity issue.

5. Advantages and Limitations

Advantages:

• Addresses Heteroscedasticity: WLS is designed to handle non-constant variance in the residuals.
• Flexibility: Observations can be given different weights based on domain knowledge.

Limitations:

• Accuracy of Weights: The efficiency of WLS depends on the accuracy of the weights. Incorrect weights can lead to inefficient estimates.
• Not a Universal Solution: WLS tackles heteroscedasticity but doesn’t address other violations of OLS assumptions, like non-linearity or autocorrelation.

6. Conclusion

Weighted Least Squares regression provides a powerful tool for addressing heteroscedasticity in linear regression models. By incorporating weights into the regression, WLS ensures that each observation contributes to the model in a manner proportional to its reliability. As with all statistical methods, understanding the underlying assumptions and being vigilant about model diagnostics is vital. R’s extensive statistical toolbox, including functions like lm(), makes it relatively straightforward to implement and evaluate WLS models.