STAT 17: Analisis of Variance (ANOVA)

Prof. Marcela Alfaro Cordoba

Statistics - UCSC

20 Nov 2025

Continuing from Last Time

Quick Recap:

Two-sample t-tests for comparing two means
Cohen’s d for effect sizes
Two-proportion z-tests
Chi-square tests for independence

Today’s Question: What if we want to compare MORE than two groups?

Today’s Learning Objectives

By the end of this lecture, you will be able to:

Conduct one-way analysis of variance (ANOVA)
Understand the F distribution and its properties
Calculate and interpret F-ratios
Apply facts about F distribution in hypothesis testing
Interpret ANOVA results and make appropriate conclusions

Our New Case Study: Marketing Channels

Scenario: An e-commerce company tests four marketing channels:

Email campaigns
Social media ads
Search engine marketing (SEM)
Affiliate partnerships

Question: Do different marketing channels lead to different average customer acquisition costs (CAC)?

Sample: 100 customers from each channel (400 total)

Why Not Multiple t-Tests?

Why not just do 6 t-tests?

Email vs Social Media
Email vs SEM
Email vs Affiliate
Social Media vs SEM
Social Media vs Affiliate
SEM vs Affiliate

The problem: Multiple comparisons inflate Type I error rate!

With $\alpha = 0.05$ per test: $P(\text{at least one false positive}) = 1 - 0.95^6 = 0.265$

26.5% chance of false positive!

The ANOVA Solution

Analysis of Variance (ANOVA):

Tests all groups simultaneously in one test
Controls overall Type I error rate at $\alpha$
Compares variation between groups to variation within groups

Key Idea: If group means truly differ, variation between groups should be larger than variation within groups

ANOVA Hypotheses

Notation: k groups with means $\mu_1, \mu_2, \ldots, \mu_k$

Hypotheses:

$H_0: \mu_1 = \mu_2 = \cdots = \mu_k$ (all means equal)
$H_a:$ At least one mean differs from the others

Important: $H_a$ does NOT say which means differ or how many differ!

For our example: $H_0: \mu_{Email} = \mu_{Social} = \mu_{SEM} = \mu_{Affiliate}$

ANOVA Assumptions

Requirements for valid ANOVA:

Independence: Observations within and between groups are independent
Normality: Data within each group approximately normally distributed
Equal variances: Population variances equal across groups (homoscedasticity)
Random sampling: Data collected through random sampling

Good news: ANOVA is fairly robust to violations when:

Sample sizes are equal or similar
Sample sizes are large (n > 30 per group)

Understanding Variation

Total variation = Between-group variation + Within-group variation

Total Sum of Squares (SST):

\[SST = \sum_{i=1}^{k} \sum_{j=1}^{n_i} (x_{ij} - \bar{x})^2\]

Measures total variability from grand mean $\bar{x}$

Between-Group Sum of Squares (SSB):

\[SSB = \sum_{i=1}^{k} n_i (\bar{x}_i - \bar{x})^2\]

Measures variability of group means from grand mean

Understanding Variation (cont.)

Within-Group Sum of Squares (SSW):

\[SSW = \sum_{i=1}^{k} \sum_{j=1}^{n_i} (x_{ij} - \bar{x}_i)^2\]

Measures variability within each group

The fundamental equation:

\[SST = SSB + SSW\]

Total variation = Explained variation + Unexplained variation

Mean Squares: Standardizing Variation

We convert sums of squares to mean squares (variances):

Mean Square Between (MSB):

\[MSB = \frac{SSB}{df_B} = \frac{SSB}{k-1}\]

Variance of group means (adjusted by sample size)

Mean Square Within (MSW):

\[MSW = \frac{SSW}{df_W} = \frac{SSW}{N-k}\]

Average variance within groups (pooled)

where N = total sample size, k = number of groups

The F-Ratio

Test statistic:

\[F = \frac{MSB}{MSW} = \frac{\text{Between-group variance}}{\text{Within-group variance}}\]

Interpretation:

$F \approx 1$: Group means similar (variation between ≈ variation within)
$F >> 1$: Group means differ (variation between >> variation within)
Large F provides evidence against $H_0$

Distribution: Under $H_0$, $F \sim F_{df_B, df_W}$

The F Distribution

Properties:

Only positive values (ratio of variances)
Right-skewed
Defined by two degrees of freedom: $df_1$ (numerator) and $df_2$ (denominator)
Mean ≈ 1 when $H_0$ is true
As $df_2 \to \infty$, approaches normal-like shape

Facts about F distribution:

$E(F_{df_1, df_2}) = \frac{df_2}{df_2-2}$ for $df_2 > 2$
Always use upper tail for rejection region

Example: Marketing Channel CAC

Data (Customer Acquisition Cost in $):

Channel	n	Mean	SD
Email	100	$45.20	$12.50
Social	100	$52.80	$14.20
SEM	100	$48.30	$13.10
Affiliate	100	$55.70	$15.30

Grand mean: $\bar{x} = \frac{45.20 + 52.80 + 48.30 + 55.70}{4} = 50.50$

Calculating SSB

\[SSB = \sum_{i=1}^{k} n_i (\bar{x}_i - \bar{x})^2\]

\[SSB = 100(45.20-50.50)^2 + 100(52.80-50.50)^2\] \[+ 100(48.30-50.50)^2 + 100(55.70-50.50)^2\]

\[= 100(28.09) + 100(5.29) + 100(4.84) + 100(27.04)\]

\[= 2,809 + 529 + 484 + 2,704 = 6,526\]

Calculating SSW

For each group: $SS_i = (n_i - 1)s_i^2$

Email: $(100-1)(12.50)^2 = 99 \times 156.25 = 15,468.75$

Social: $(100-1)(14.20)^2 = 99 \times 201.64 = 19,962.36$

SEM: $(100-1)(13.10)^2 = 99 \times 171.61 = 16,989.39$

Affiliate: $(100-1)(15.30)^2 = 99 \times 234.09 = 23,174.91$

\[SSW = 15,468.75 + 19,962.36 + 16,989.39 + 23,174.91 = 75,595.41\]

Calculating Mean Squares

Degrees of freedom:

$df_B = k - 1 = 4 - 1 = 3$
$df_W = N - k = 400 - 4 = 396$

Mean Square Between:

\[MSB = \frac{SSB}{df_B} = \frac{6,526}{3} = 2,175.33\]

Mean Square Within:

\[MSW = \frac{SSW}{df_W} = \frac{75,595.41}{396} = 190.90\]

Calculating F-Statistic

\[F = \frac{MSB}{MSW} = \frac{2,175.33}{190.90} = 11.39\]

Interpretation: Between-group variance is 11.39 times larger than within-group variance!

This suggests the group means are different.

Making a Decision

Critical value: For $\alpha = 0.05$, $df_1 = 3$, $df_2 = 396$

Critical value $\approx 2.63$ (from Google Sheets)

Our test statistic: $F = 11.39 > 2.63$

Decision: Reject $H_0$

Conclusion: There is significant evidence that average customer acquisition costs differ across marketing channels.

But which channels differ? ANOVA doesn’t tell us! Need post-hoc tests (next course).

The ANOVA Table

Standard format for presenting results:

Source	SS	df	MS	F	p-value
Between	6,526	3	2,175.33	11.39	<0.001
Within	75,595	396	190.90
Total	82,121	399

Reading the table:

Large F-value → strong evidence of differences
Small p-value → statistically significant
MSW estimates common variance $\sigma^2$

Google Sheets for ANOVA

Function: =F.TEST(array1, array2)

Only compares two arrays - not ideal for ANOVA!

Better approach: Construct your own table in Google Sheets

Google Sheets: Manual ANOVA

For k groups with equal sample sizes n:

// Grand mean
=AVERAGE(all_data_range)

// SSB
=n*SUMSQ(group_means_range - grand_mean)

// SSW
=(n-1)*SUMSQ(group1_SD, group2_SD, ...)

// MSB
=SSB/(k-1)

// MSW  
=SSW/(n*k-k)

// F-statistic
=MSB/MSW

// P-value
=F.DIST.RT(F_stat, k-1, n*k-k)

🧘‍♀️ STRETCH BREAK

Time to move! (5 minutes)

Stand up and stretch 🤸‍♀️
Chat with neighbors about differences of proportions 💬
Grab some water 💧

Welcome Back!

What we’ve covered:

ANOVA framework and assumptions
Sum of squares decomposition
F-ratio calculation
Making decisions with F distribution

Now: F distribution and practical considerations

F Distribution Applications

The F distribution appears in many contexts:

ANOVA: Testing equality of multiple means
Regression: Testing overall model significance
Variance comparison: Testing $H_0: \sigma_1^2 = \sigma_2^2$
Nested models: Comparing model fit

Always: $F = \frac{\text{Variance explained by model}}{\text{Variance not explained (error)}}$

Facts About F Distribution

Key relationships:

If $X \sim \chi^2_{df_1}$ and $Y \sim \chi^2_{df_2}$, then: \[F = \frac{X/df_1}{Y/df_2} \sim F_{df_1, df_2}\]
If $t \sim t_{df}$, then $t^2 \sim F_{1,df}$
As $df_2 \to \infty$: $df_1 \times F_{df_1,df_2} \to \chi^2_{df_1}$
$F_{df_1,df_2,\alpha}$ (critical value) depends on BOTH degrees of freedom

Relationship to t-Test

Special case: When k = 2 (two groups), ANOVA = two-sample t-test

Mathematical relationship:

\[F_{1,df} = t_{df}^2\]

The F-test with 1 numerator df is equivalent to a squared t-test!

Example: Two-sample t-test gives $t = 2.5$ with df = 58

ANOVA would give $F = 2.5^2 = 6.25$ with $df_1 = 1$, $df_2 = 58$

Both give identical p-values!

Effect Size in ANOVA

Statistical significance doesn’t tell magnitude!

Eta-squared ($\eta^2$): Proportion of total variance explained

\[\eta^2 = \frac{SSB}{SST}\]

Ranges from 0 to 1 (like R-squared in regression)

Cohen’s f: Standardized effect size for ANOVA

\[f = \sqrt{\frac{\eta^2}{1-\eta^2}}\]

Interpretation: Small (0.10), Medium (0.25), Large (0.40)

Effect Size: Our Example

Marketing channel data:

\[\eta^2 = \frac{SSB}{SST} = \frac{6,526}{82,121} = 0.0795\]

Interpretation: About 7.95% of total variation in CAC is explained by marketing channel.

Cohen’s f:

\[f = \sqrt{\frac{0.0795}{1-0.0795}} = \sqrt{0.0864} = 0.294\]

Medium effect size - practically meaningful difference!

What If We Reject H₀?

ANOVA tells us: “At least one mean differs”

ANOVA does NOT tell us:

Which specific groups differ
How many groups differ
Direction of differences

Solution: Post-hoc tests

Tukey’s HSD (Honestly Significant Difference)
Bonferroni correction
Scheffé’s method
Dunnett’s test (compare all to control)

Covered in more advanced courses!

Practical Example: Post-Hoc Interpretation

Our marketing channels (ordered by mean CAC):

Email: $45.20
SEM: $48.30
Social Media: $52.80
Affiliate: $55.70

With post-hoc tests, might find:

Email significantly cheaper than Social Media and Affiliate
SEM significantly cheaper than Affiliate
Social Media not significantly different from SEM

Business decision: Prioritize Email and SEM channels for cost efficiency

Common Mistakes to Avoid

Using multiple t-tests instead of ANOVA → inflates Type I error
Ignoring assumptions → check normality, equal variances, independence
Confusing significance with importance → always report effect sizes
Stopping at ANOVA → need post-hoc tests to know which groups differ
Wrong df → $df_B = k-1$, $df_W = N-k$ (not n-k!)
Causation claims → ANOVA shows association, not causation

When ANOVA Assumptions Fail

Normality violated:

Transform data (log, sqrt)
Use Kruskal-Wallis test (non-parametric alternative)

Unequal variances:

Welch’s ANOVA (doesn’t assume equal variances)
Transform to stabilize variance

Small samples:

Check assumptions carefully
Consider non-parametric alternatives
Use bootstrapping methods

ANOVA in Business Context

Common applications:

Marketing: Compare campaign performance, channel effectiveness
Operations: Test process improvements, quality control
Finance: Compare investment strategies, portfolio performance
HR: Evaluate training programs, compensation equity
Retail: Test store layouts, pricing strategies
Product: Compare product variants, features

Connecting the Pieces

This week’s journey:

Lecture 1: Two-sample tests (t-tests, proportion tests) amd Chi-square test.

Lecture 2: ANOVA for multiple groups

Unifying theme: Compare groups, control error rates, report effect sizes, make informed decisions

Looking Ahead

Next unit: Regression

Linear relationships between variables
Prediction and explanation
Another use of F distribution!
Building on everything we’ve learned

This builds on:

Hypothesis testing framework
Variance decomposition (SSB, SSW, SST)
F distribution understanding

Quick Knowledge Check ✅

Rate your confidence (1-5) on Ed Discussion:

Setting up ANOVA hypotheses ⭐⭐⭐⭐⭐
Computing F-statistic ⭐⭐⭐⭐⭐
Interpreting ANOVA results ⭐⭐⭐⭐⭐
Understanding F distribution ⭐⭐⭐⭐⭐
Determining effect sizes ⭐⭐⭐⭐⭐

Post on Ed Discussion!

Need help? Office hours are your friend!

Key Formulas Reference

Sums of Squares:

\[SSB = \sum_{i=1}^{k} n_i (\bar{x}_i - \bar{x})^2\]

\[SSW = \sum_{i=1}^{k} (n_i-1)s_i^2\]

\[SST = SSB + SSW\]

Mean Squares:

\[MSB = \frac{SSB}{k-1}, \quad MSW = \frac{SSW}{N-k}\]

Key Formulas Reference

Test Statistic:

\[F = \frac{MSB}{MSW} \sim F_{k-1, N-k}\]

Effect Size:

\[\eta^2 = \frac{SSB}{SST}, \quad f = \sqrt{\frac{\eta^2}{1-\eta^2}}\]

Thank you! 📊✨

Questions? I have office hours right after class today!

Next up: ANOVA and Linear Regression

Remember:

Post Think-Pair-Share on Ed Discussion and Poll Everywhere
Rate your confidence
ANOVA helps us compare many groups efficiently while controlling error rates!

Keep practicing!

STAT 17: Analisis of Variance (ANOVA)

Continuing from Last Time

Today’s Learning Objectives

Our New Case Study: Marketing Channels

Why Not Multiple t-Tests?

The ANOVA Solution

ANOVA Hypotheses

ANOVA Assumptions

Understanding Variation

Understanding Variation (cont.)

Mean Squares: Standardizing Variation

The F-Ratio

The F Distribution

Example: Marketing Channel CAC

Calculating SSB

Calculating SSW

Calculating Mean Squares

Calculating F-Statistic

Making a Decision

The ANOVA Table

THINK-PAIR-SHARE 1 (7 minutes)

Poll Everywhere Time!

Share your answers in Poll Everywhere!

Google Sheets for ANOVA

Google Sheets: Manual ANOVA

🧘‍♀️ STRETCH BREAK

Time to move! (5 minutes)

Welcome Back!

F Distribution Applications

Facts About F Distribution

Relationship to t-Test

Effect Size in ANOVA

Effect Size: Our Example

THINK-PAIR-SHARE 2 (7 minutes)

Poll Everywhere Time!

Share your answers in Poll Everywhere!

What If We Reject H₀?

Practical Example: Post-Hoc Interpretation

Common Mistakes to Avoid

When ANOVA Assumptions Fail

ANOVA in Business Context

Connecting the Pieces

Looking Ahead

Quick Knowledge Check ✅

Key Formulas Reference

Key Formulas Reference

Thank you! 📊✨