Simple Linear Regression

# Simple Linear Regression
## Partioning variability
### Prof. Maria Tackett

---

## [Click here for PDF of slides](06-slr-partition-var.pdf)

---

## Topics

- Use analysis of variance to partition variability in the response variable

- Define and calculate `$R^2$`

- Use ANOVA to test the hypothesis

`$$H_0: \beta_1 = 0 \text{ vs }H_a: \beta_1 \neq 0$$`

---

## Cats data

The data set contains the **heart weight** (.term[Hwt]) and **body weight** (.term[Bwt]) for 144 domestic cats.

---

## Distribution of response

---

## The model

<img src="06-slr-partition-var_files/figure-html/unnamed-chunk-6-1.png" style="display: block; margin: auto;" />
---

.alert[
How much of the variation in cats' heart weights can be explained by knowing their body weights?
]

---

## ANOVA

We will use .vocab[Analysis of Variance (ANOVA)] to partition the variation in the response variable `$Y$`.

<br>

---

## Response variable, `$Y$`

---

## Total variation

`$$\large{SS_{Total} = \sum_{i=1}^n(y_i - \bar{y})^2 = (n-1)s_y^2}$$`

---

## Explained variation (Model)

`$$\large{SS_{Model} = \sum_{i = 1}^{n}(\hat{y}_i - \bar{y})^2}$$`

---

## Unexplained variation (Residuals)

`$$\large{SS_{Error} = \sum_{i = 1}^{n}(y_i - \hat{y}_i)^2}$$`

---

`$$\sum_{i=1}^n(y_i - \bar{y})^2 = \sum_{i = 1}^{n}(\hat{y}_i - \bar{y})^2 + \sum_{i = 1}^{n}(y_i - \hat{y}_i)^2$$`
---

`$$\mathbf{\color{blue}{\sum_{i=1}^n(y_i - \bar{y})^2}} = \sum_{i = 1}^{n}(\hat{y}_i - \bar{y})^2 + \sum_{i = 1}^{n}(y_i - \hat{y}_i)^2$$`

---

`$$\sum_{i=1}^n(y_i - \bar{y})^2 = \mathbf{\color{blue}{\sum_{i = 1}^{n}(\hat{y}_i - \bar{y})^2}} + \sum_{i = 1}^{n}(y_i - \hat{y}_i)^2$$`

---

`$$\sum_{i=1}^n(y_i - \bar{y})^2 = \sum_{i = 1}^{n}(\hat{y}_i - \bar{y})^2 + \mathbf{\color{blue}{\sum_{i = 1}^{n}(y_i - \hat{y}_i)^2}}$$`
---

## `$R^2$`

The .vocab[coefficient of determination], <font class = "vocab">R<sup>2</sup></font>, is the proportion of variation in the response, `$Y$`, that is explained by the regression model

<br>

---

## `$R^2$` for our model

`$$SS_{Error} = 299.533$$`

`$$SS_{Total} = 847.625$$` 
]
]

.pull-right[
.small-box-work[
`$$\begin{aligned}R^2 &= \frac{548.092}{847.625} \\[10pt]
&= \mathbf{0.647}\end{aligned}$$`
]
]

<br>

.vocab[About 64.7% of the variation in the heart weight of cats can be explained by variation in body weight.]

---

## ANOVA table

<table>
 <thead>
  <tr>
   <th style="text-align:left;"> Source </th>
   <th style="text-align:right;"> Df </th>
   <th style="text-align:right;"> Sum Sq </th>
   <th style="text-align:right;"> Mean Sq </th>
   <th style="text-align:right;"> F Stat </th>
   <th style="text-align:right;"> Pr(&gt; F) </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Model </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 548.092 </td>
   <td style="text-align:right;"> 548.092 </td>
   <td style="text-align:right;"> 259.835 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Residuals </td>
   <td style="text-align:right;"> 142 </td>
   <td style="text-align:right;"> 299.533 </td>
   <td style="text-align:right;"> 2.109 </td>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;">  </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Total </td>
   <td style="text-align:right;"> 143 </td>
   <td style="text-align:right;"> 847.625 </td>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;">  </td>
  </tr>
</tbody>
</table>

---

## ANOVA table

<table>
 <thead>
  <tr>
   <th style="text-align:left;"> Source </th>
   <th style="text-align:right;"> Df </th>
   <th style="text-align:right;background-color: #dce5b2 !important;"> Sum Sq </th>
   <th style="text-align:right;"> Mean Sq </th>
   <th style="text-align:right;"> F Stat </th>
   <th style="text-align:right;"> Pr(&gt; F) </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Model </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;background-color: #dce5b2 !important;"> 548.092 </td>
   <td style="text-align:right;"> 548.092 </td>
   <td style="text-align:right;"> 259.835 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Residuals </td>
   <td style="text-align:right;"> 142 </td>
   <td style="text-align:right;background-color: #dce5b2 !important;"> 299.533 </td>
   <td style="text-align:right;"> 2.109 </td>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;">  </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Total </td>
   <td style="text-align:right;"> 143 </td>
   <td style="text-align:right;background-color: #dce5b2 !important;"> 847.625 </td>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;">  </td>
  </tr>
</tbody>
</table>

`$SS_{Total} = 847.625 = 548.092 + 299.533$`

`$SS_{Model} = 548.092$`

`$SS_{Error} = 299.533$`

---

## ANOVA Test

<br>

---

## ANOVA Test

<table>
 <thead>
  <tr>
   <th style="text-align:left;"> Source </th>
   <th style="text-align:right;background-color: #dce5b2 !important;"> Df </th>
   <th style="text-align:right;"> Sum Sq </th>
   <th style="text-align:right;"> Mean Sq </th>
   <th style="text-align:right;"> F Stat </th>
   <th style="text-align:right;"> Pr(&gt; F) </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Model </td>
   <td style="text-align:right;background-color: #dce5b2 !important;"> 1 </td>
   <td style="text-align:right;"> 548.092 </td>
   <td style="text-align:right;"> 548.092 </td>
   <td style="text-align:right;"> 259.835 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Residuals </td>
   <td style="text-align:right;background-color: #dce5b2 !important;"> 142 </td>
   <td style="text-align:right;"> 299.533 </td>
   <td style="text-align:right;"> 2.109 </td>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;">  </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Total </td>
   <td style="text-align:right;background-color: #dce5b2 !important;"> 143 </td>
   <td style="text-align:right;"> 847.625 </td>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;">  </td>
  </tr>
</tbody>
</table>

`$df_{Total} = 144 - 1 = 143$`

`$df_{Model} = 1$`

`$df_{Error} = 143 - 1 = 142$`

---

## ANOVA Test

<table>
 <thead>
  <tr>
   <th style="text-align:left;"> Source </th>
   <th style="text-align:right;"> Df </th>
   <th style="text-align:right;"> Sum Sq </th>
   <th style="text-align:right;background-color: #dce5b2 !important;"> Mean Sq </th>
   <th style="text-align:right;"> F Stat </th>
   <th style="text-align:right;"> Pr(&gt; F) </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Model </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 548.092 </td>
   <td style="text-align:right;background-color: #dce5b2 !important;"> 548.092 </td>
   <td style="text-align:right;"> 259.835 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Residuals </td>
   <td style="text-align:right;"> 142 </td>
   <td style="text-align:right;"> 299.533 </td>
   <td style="text-align:right;background-color: #dce5b2 !important;"> 2.109 </td>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;">  </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Total </td>
   <td style="text-align:right;"> 143 </td>
   <td style="text-align:right;"> 847.625 </td>
   <td style="text-align:right;background-color: #dce5b2 !important;">  </td>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;">  </td>
  </tr>
</tbody>
</table>

`$MS_{Model} = \frac{548.092}{1} = 548.092$`

`$MS_{Error} = \frac{299.533}{142} = 2.109$`

---

## ANOVA Test

<table>
 <thead>
  <tr>
   <th style="text-align:left;"> Source </th>
   <th style="text-align:right;"> Df </th>
   <th style="text-align:right;"> Sum Sq </th>
   <th style="text-align:right;"> Mean Sq </th>
   <th style="text-align:right;background-color: #dce5b2 !important;"> F Stat </th>
   <th style="text-align:right;"> Pr(&gt; F) </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Model </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 548.092 </td>
   <td style="text-align:right;"> 548.092 </td>
   <td style="text-align:right;background-color: #dce5b2 !important;"> 259.835 </td>
   <td style="text-align:right;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Residuals </td>
   <td style="text-align:right;"> 142 </td>
   <td style="text-align:right;"> 299.533 </td>
   <td style="text-align:right;"> 2.109 </td>
   <td style="text-align:right;background-color: #dce5b2 !important;">  </td>
   <td style="text-align:right;">  </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Total </td>
   <td style="text-align:right;"> 143 </td>
   <td style="text-align:right;"> 847.625 </td>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;background-color: #dce5b2 !important;">  </td>
   <td style="text-align:right;">  </td>
  </tr>
</tbody>
</table>

`$F = \frac{MS_{Model}}{MS_{Error}}= \frac{548.092}{2.109} = 259.835$`

---

## F distribution

---

## ANOVA test

<table>
 <thead>
  <tr>
   <th style="text-align:left;"> Source </th>
   <th style="text-align:right;"> Df </th>
   <th style="text-align:right;"> Sum Sq </th>
   <th style="text-align:right;"> Mean Sq </th>
   <th style="text-align:right;"> F Stat </th>
   <th style="text-align:right;background-color: #dce5b2 !important;"> Pr(&gt; F) </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Model </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 548.092 </td>
   <td style="text-align:right;"> 548.092 </td>
   <td style="text-align:right;"> 259.835 </td>
   <td style="text-align:right;background-color: #dce5b2 !important;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Residuals </td>
   <td style="text-align:right;"> 142 </td>
   <td style="text-align:right;"> 299.533 </td>
   <td style="text-align:right;"> 2.109 </td>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;background-color: #dce5b2 !important;">  </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Total </td>
   <td style="text-align:right;"> 143 </td>
   <td style="text-align:right;"> 847.625 </td>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;background-color: #dce5b2 !important;">  </td>
  </tr>
</tbody>
</table>

.vocab[P-value]: Probability of observing a test statistic at least as extreme as *F Stat* given the population slope `$\beta_1$` is 0

The p-value is calculated using an `$F$` distribution with 1 and `$n-2$` degrees of freedom

---

## Calculating p-value

---

## ANOVA

<table>
 <thead>
  <tr>
   <th style="text-align:left;"> Source </th>
   <th style="text-align:right;"> Df </th>
   <th style="text-align:right;"> Sum Sq </th>
   <th style="text-align:right;"> Mean Sq </th>
   <th style="text-align:right;"> F Stat </th>
   <th style="text-align:right;background-color: #dce5b2 !important;"> Pr(&gt; F) </th>
  </tr>
 </thead>
<tbody>
  <tr>
   <td style="text-align:left;"> Model </td>
   <td style="text-align:right;"> 1 </td>
   <td style="text-align:right;"> 548.092 </td>
   <td style="text-align:right;"> 548.092 </td>
   <td style="text-align:right;"> 259.835 </td>
   <td style="text-align:right;background-color: #dce5b2 !important;"> 0 </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Residuals </td>
   <td style="text-align:right;"> 142 </td>
   <td style="text-align:right;"> 299.533 </td>
   <td style="text-align:right;"> 2.109 </td>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;background-color: #dce5b2 !important;">  </td>
  </tr>
  <tr>
   <td style="text-align:left;"> Total </td>
   <td style="text-align:right;"> 143 </td>
   <td style="text-align:right;"> 847.625 </td>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;">  </td>
   <td style="text-align:right;background-color: #dce5b2 !important;">  </td>
  </tr>
</tbody>
</table>

The p-value is very small `$(\approx 0)$`, so we reject `$H_0$`.

The data provide strong evidence that population slope, `$\beta_1$`, is different from 0.

.vocab[The data provide sufficient evidence that there is a linear relationship between a cat's heart weight and body weight.]

---

## Recap

- Used analysis of variance to partition variability in the response variable

- Defined and calculated `$R^2$`

- Used ANOVA to test the hypothesis 
`$$H_0: \beta_1 = 0 \text{ vs }H_a: \beta_1 \neq 0$$`