Topics

• ANOVA for Multiple Linear Regression

• Nested F Test

• R2 vs. Adj. R2

• AIC & BIC

Restaurant tips

What affects the amount customers tip at a restaurant?

• Response:

  • Tip: amount of the tip

• Predictors:

  • Party: number of people in the party
  • Meal: time of day (Lunch, Dinner, Late Night)
  • Age: age category of person paying the bill (Yadult, Middle, SenCit)

Response Variable

Predictor Variables

Response vs. Predictors

Restaurant tips: model

ANOVA test for MLR

Using the ANOVA table, we can test whether any variable in the model is a significant predictor of the response. We conduct this test using the following hypotheses:

• The statistic for this test is the F test statistic in the ANOVA table

• We calculate the p-value using an F distribution with p and (n−p−1) degrees of freedom

Tips: ANOVA Test

Model df: 3

Model SS: 1188.636 + 38.028 = 1226.664

Model MS: 1226.664/ 3 = 408.888

FStat: 408.888 / 4.160 = 98.2903846

P-value: P(F > 98.2903846) ≈0

• calculated using an F distribution with 3 and 165 degrees of freedom

Tips: ANOVA Test

The data provide sufficient evidence to conclude that at least one coefficient is non-zero, i.e. at least one predictor in the model is significant.

Testing subset of coefficients

• Sometimes we want to test whether a subset of coefficients are all equal to 0

• This is often the case when we want test

  • whether a categorical variable with k levels is a significant predictor of the response
  • whether the interaction between a categorical and quantitative variable is significant

• To do so, we will use the Nested (Partial) F Test

Nested (Partial) F Test

• Suppose we have a full and reduced model:

Full:y=β0+β1x1+⋯+βqxq+βq+1xq+1+…βpxpReduced:y=β0+β1x1+⋯+βqxq

• We want to test whether any of the variables xq+1,xq+2,…,xp are significant predictors. To do so, we will test the hypothesis:

Nested F Test

• The test statistic for this test is

F=(SSEreduced−SSEfull)/# predictors testedSSEfull/(n−pfull−1)

• Calculate the p-value using the F distribution with df1 = # predictors tested and df2 = (n−pfull−1)

Is Meal a significant predictor of tips?

Tips: Nested F test

H0:βlatenight=βlunch=0Ha: at least one βj is not equal to 0

reduced <- lm(Tip ~ Party + Age, data = tips)

full <- lm(Tip ~ Party + Age + Meal, data = tips)

#Nested F test in R
anova(reduced, full)

Tips: Nested F test

F Stat: (686.444−622.979)/2622.979/(169−5−1)=8.303

P-value: P(F > 8.303) = 0.0003

• calculated using an F distribution with 2 and 163 degrees of freedom

The data provide sufficient evidence to conclude that at least one coefficient associated with Meal is not zero. Therefore, Meal is a significant predictor of Tips.

Model with Meal

Including interactions

Does the effect of Party differ based on the Meal time?

Nested F test for interactions

Let's use a Nested F test to determine if Party*Meal is statistically significant.

reduced <- lm(Tip ~ Party + Age + Meal, data = tips)

full <- lm(Tip ~ Party + Age + Meal + Meal * Party, 
           data = tips)

Final model for now

We conclude that the effect of Party does not differ based Meal. Therefore, we will use the original model that only included main effects.

R2

Recall: R2 is the proportion of the variation in the response variable explained by the regression model

R2 will always increase as we add more variables to the model

• If we add enough variables, we can always achieve R2=100%

If we only use R2 to choose a best fit model, we will be prone to choose the model with the most predictor variables

Adjusted R2

Adjusted R²: measure that includes a penalty for unnecessary predictor variables

Similar to R2, it is a measure of the amount of variation in the response that is explained by the regression model

Differs from R2 by using the mean squares rather than sums of squares and therefore adjusting for the number of predictor variables

R2 and Adjusted R2

R2=SSModelSSTotal=1−SSErrorSSTotal

Using R2 and Adj.R2

Adj.R2 can be used as a quick assessment to compare the fit of multiple models; however, it should not be the only assessment!

Use R2 when describing the relationship between the response and predictor variables

Tips: Comparing models

Let's compare two models:

model1 <- lm(Tip ~ Party + Age + Meal, data = tips)
glance(model1) %>% select(r.squared, adj.r.squared)

## # A tibble: 1 x 2
##   r.squared adj.r.squared
##       <dbl>         <dbl>
## 1     0.674         0.664

model2 <- lm(Tip ~ Party + Age + Meal + Day, data = tips)
glance(model2) %>% select(r.squared, adj.r.squared)

## # A tibble: 1 x 2
##   r.squared adj.r.squared
##       <dbl>         <dbl>
## 1     0.683         0.662

AIC & BIC

Akaike's Information Criterion (AIC): AIC=nlog(SSError)−nlog(n)+2(p+1)

Schwarz's Bayesian Information Criterion (BIC) BIC=nlog(SSError)−nlog(n)+log(n)×(p+1)

See the supplemental note on AIC & BIC for derivations.

AIC & BIC

AIC=nlog(SSError)−nlog(n)+2(p+1)BIC=nlog(SSError)−nlog(n)+log(n)×(p+1)

First Term: Decreases as p increases

AIC & BIC

AIC=nlog(SSError)−nlog(n)+2(p+1)BIC=nlog(SSError)−nlog(n)+log(n)×(p+1)

Second Term: Fixed for a given sample size n

AIC & BIC

AIC=nlog(SSError)−nlog(n)+2(p+1)BIC=nlog(SSError)−nlog(n)+log(n)×(p+1)

Third Term: Increases as p increases

Using AIC & BIC

AIC=nlog(SSError)−nlog(n)+2(p+1)BIC=nlog(SSError)−nlog(n)+log(n)×(p+1)

• Choose model with the smaller value of AIC or BIC

• If n≥8, the penalty for BIC is larger than that of AIC, so BIC tends to favor more parsimonious models (i.e. models with fewer terms)

Tips: AIC & BIC

model1 <- lm(Tip ~ Party + Age + Meal, data = tips)
glance(model1) %>% select(AIC, BIC)

## # A tibble: 1 x 2
##     AIC   BIC
##   <dbl> <dbl>
## 1  714.  736.

model2 <- lm(Tip ~ Party + Age + Meal + Day, data = tips)
glance(model2) %>% select(AIC, BIC)

## # A tibble: 1 x 2
##     AIC   BIC
##   <dbl> <dbl>
## 1  720.  757.

Recap

• ANOVA for Multiple Linear Regression

• Nested F Test

• R2 vs. Adj. R2

• AIC & BIC