🗓️ Week 03
Exploratory Data Analysis

STAT 204 – Introduction to Statistical Data Analysis

01 Oct 2025

This week

  • Categorical Data + Chi Square Test
  • Exploratory Data Analysis
  • Graphical Summaries (base + ggplot2)

Categorical Data Analysis

Working with Categorical Variables

Categorical data represents groups or categories rather than numerical measurements.

Examples:

  • Survey responses (Yes/No, Agree/Disagree)

  • Demographics (Gender, Race, Education Level)

  • Classifications (Disease status, Product type)

Today’s goals:

  1. Create and visualize categorical data

  2. Test if observed frequencies match expected frequencies (goodness of fit)

  3. Test if two categorical variables are independent (contingency tables)

Creating Categorical Data in R

# Simple vector of categories
coins = c("H", "T", "H", "H", "T", "T", "H", "T", "H", "H")
table(coins)
coins
H T 
6 4 
barplot(table(coins))

Basic functions:

  • table() – counts frequencies

  • barplot() – visualizes frequencies

  • prop.table() – converts counts to proportions

Part 1: Chi-Square Goodness of Fit Test

Question: Are Events Equally Likely?

Example: Do car crashes occur on different days with the same frequency?

Day Mon Tue Wed Thu Fri Sat Sun
Fatalities 20 20 22 22 29 36 31

Hypothesis:

  • \(H_0\): Crashes are equally likely on all days (uniform distribution)

  • \(H_A\): Some days have more crashes than others

Performing the Test

accidents = c(20, 20, 22, 22, 29, 36, 31)
sum(accidents)  # Total: 180
[1] 180
# Expected counts under null hypothesis (equal distribution)
expected_accidents = rep(180/7, 7)
expected_accidents
[1] 25.71429 25.71429 25.71429 25.71429 25.71429 25.71429 25.71429
# Chi-square test
chisq.test(accidents)

    Chi-squared test for given probabilities

data:  accidents
X-squared = 9.2333, df = 6, p-value = 0.1609

Interpretation: very small p-value indicates evidence to reject \(H_0\). We will talk about p-values later in the quarter :).

Visualizing Observed vs Expected

Code
d_matrix <- cbind(accidents, expected_accidents)
rownames(d_matrix) <- c("Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun")
colnames(d_matrix) <- c("Observed", "Expected")
weekday_colors_pastel <- c(
  Mon = "#B3CDE3",  # Light blue
  Tue = "#CCEBC5",  # Light green
  Wed = "#DECBE4",  # Light purple
  Thu = "#FED9A6",  # Light orange
  Fri = "#FFE6CC",  # Light peach
  Sat = "#FBB4AE",  # Light coral
  Sun = "#E5D8BD"   # Light tan
)
barplot(d_matrix, beside=TRUE,
        col = weekday_colors_pastel,
        ylab = "Number of Fatalities")

Weekend days show the largest deviations from expected values

Activity 11: Goodness of Fit Practice

Partner work (10 minutes)

A die is rolled 60 times with the following results:

Face 1 2 3 4 5 6
Count 8 12 9 11 13 7

Your tasks:

  1. Create a vector with the observed counts
  2. What are the expected counts if the die is fair?
  3. Perform a chi-square goodness of fit test
  4. Create a barplot comparing observed vs. expected
  5. Interpret the results: Is the die fair?
# Starter code:
die_rolls = c(8, 12, 9, 11, 13, 7)
# Your code here...

Discuss: What p-value would convince you the die is unfair?

Part 2: Contingency Tables

Testing Independence Between Two Variables

Question: Is there an association between two categorical variables?

Example: Using the built-in UCBAdmissions dataset

  • Admission decision (Admitted/Rejected)

  • Gender (Male/Female)

The UCBAdmissions Dataset

# Load the data
data(UCBAdmissions)
UCBAdmissions
, , Dept = A

          Gender
Admit      Male Female
  Admitted  512     89
  Rejected  313     19

, , Dept = B

          Gender
Admit      Male Female
  Admitted  353     17
  Rejected  207      8

, , Dept = C

          Gender
Admit      Male Female
  Admitted  120    202
  Rejected  205    391

, , Dept = D

          Gender
Admit      Male Female
  Admitted  138    131
  Rejected  279    244

, , Dept = E

          Gender
Admit      Male Female
  Admitted   53     94
  Rejected  138    299

, , Dept = F

          Gender
Admit      Male Female
  Admitted   22     24
  Rejected  351    317
# Create a 2x2 table (collapsing across departments)
admit_data = margin.table(UCBAdmissions, c(1, 2))
admit_data
          Gender
Admit      Male Female
  Admitted 1198    557
  Rejected 1493   1278

This shows admissions by gender at UC Berkeley (famous for Simpson’s Paradox!)

Visualizing Contingency Tables

# Proportions by row (within each admission status)
prop.table(admit_data, margin = 1)
          Gender
Admit           Male    Female
  Admitted 0.6826211 0.3173789
  Rejected 0.5387947 0.4612053
# Mosaic plot
mosaicplot(admit_data, color = c("lightblue", "salmon"),
           main = "UC Berkeley Admissions by Gender",
           xlab = "Admission Status", ylab = "Gender")

Reading mosaic plots:

  • Width = proportion of total in that category

  • Height = conditional proportion

Chi-Square Test of Independence

Hypotheses: - \(H_0\): Admission and Gender are independent - \(H_A\): There is an association between Admission and Gender

Test statistic: \[X^2 = \sum_{\text{all cells}} \frac{(\text{observed} - \text{expected})^2}{\text{expected}}\]

Under \(H_0\), this follows \(\chi^2_{(r-1)(c-1)}\) distribution

Performing the Test

chi_result = chisq.test(admit_data)
chi_result

    Pearson's Chi-squared test with Yates' continuity correction

data:  admit_data
X-squared = 91.61, df = 1, p-value < 2.2e-16
# Expected counts under independence
chi_result$expected
          Gender
Admit          Male    Female
  Admitted 1043.461  711.5389
  Rejected 1647.539 1123.4611
# Residuals (standardized differences)
chi_result$residuals
          Gender
Admit           Male    Female
  Admitted  4.784093 -5.793466
  Rejected -3.807325  4.610614

Interpretation: Statistically significant association between gender and admission

Understanding Residuals

Residuals show which cells differ most from independence:

\[\text{Residual} = \frac{\text{observed} - \text{expected}}{\sqrt{\text{expected}}}\]

chi_result$residuals
          Gender
Admit           Male    Female
  Admitted  4.784093 -5.793466
  Rejected -3.807325  4.610614

Large positive residual: More than expected under independence
Large negative residual: Fewer than expected under independence

Activity 12: Titanic Survival Analysis

Partner work (12 minutes)

Use the built-in Titanic dataset to examine survival by class:

data(Titanic)
# Create a 2x2 table: Survived vs Class (just 1st and 3rd class)
titanic_data = margin.table(Titanic, c(4, 1))
titanic_data
        Class
Survived 1st 2nd 3rd Crew
     No  122 167 528  673
     Yes 203 118 178  212

Your tasks:

  1. Calculate the proportion who survived in each class
  2. Create a mosaic plot with shading: mosaicplot(..., shade = TRUE)
  3. Perform a chi-square test of independence
  4. Examine the residuals to see which cells are most unusual
  5. Interpret: Was survival independent of passenger class?

Discuss: What do the residuals tell you about survival patterns?

Mosaic Plots with Shading

Shading enhances mosaic plots by showing residuals:

mosaicplot(admit_data, shade = TRUE,
           main = "UC Berkeley Admissions",
           xlab = "Admission Status", ylab = "Gender")

Color coding:

  • Blue shades: More than expected (positive residuals)

  • Red shades: Fewer than expected (negative residuals)

  • Darker colors = larger residuals

Detailed Example: HairEyeColor Dataset

Three-Way Contingency Table

data(HairEyeColor)
HairEyeColor
, , Sex = Male

       Eye
Hair    Brown Blue Hazel Green
  Black    32   11    10     3
  Brown    53   50    25    15
  Red      10   10     7     7
  Blond     3   30     5     8

, , Sex = Female

       Eye
Hair    Brown Blue Hazel Green
  Black    36    9     5     2
  Brown    66   34    29    14
  Red      16    7     7     7
  Blond     4   64     5     8
# Collapse across Sex to get Hair vs Eye
hair_eye = margin.table(HairEyeColor, c(1, 2))
hair_eye
       Eye
Hair    Brown Blue Hazel Green
  Black    68   20    15     5
  Brown   119   84    54    29
  Red      26   17    14    14
  Blond     7   94    10    16

Analyzing Hair and Eye Color

# Chi-square test
chisq.test(hair_eye)

    Pearson's Chi-squared test

data:  hair_eye
X-squared = 138.29, df = 9, p-value < 2.2e-16
# Visualize
mosaicplot(hair_eye, shade = TRUE,
           main = "Hair Color vs Eye Color",
           las = 1)

Observation: Strong association – certain combinations (e.g., Blond/Blue) are more common than expected

Activity 13: Complete Categorical Analysis

Partner work (15 minutes)

Use the HairEyeColor dataset to answer:

Does the relationship between hair and eye color differ by sex?

# Separate tables for males and females
male_data = HairEyeColor[,,"Male"]
female_data = HairEyeColor[,,"Female"]

Your tasks:

  1. Create mosaic plots for males and females separately
  2. Perform chi-square tests for each
  3. Compare the residuals – which combinations are most unusual in each sex?
  4. Calculate the proportions: prop.table(male_data, margin = 1)

Discussion questions: - Are the associations similar for males and females? - Which hair/eye combinations show the strongest associations? - How would you report these findings?

Advanced: Manual Chi-Square Calculation

Self-study guide

Understanding the math behind chisq.test():

# Example with admit_data
T = admit_data

# Step 1: Calculate expected frequencies
row_totals = rowSums(T)
col_totals = colSums(T)
grand_total = sum(T)

Expected = outer(row_totals, col_totals) / grand_total
Expected
             Male    Female
Admitted 1043.461  711.5389
Rejected 1647.539 1123.4611
# Step 2: Calculate chi-square statistic
chi_stat = sum((T - Expected)^2 / Expected)
chi_stat
[1] 92.20528
# Step 3: Calculate p-value
df = (nrow(T) - 1) * (ncol(T) - 1)
p_value = 1 - pchisq(chi_stat, df)
p_value
[1] 0

Key Assumptions for Chi-Square Tests

Self-study guide

When chi-square tests are valid:

  1. Independence: Observations must be independent
  2. Sample size:
    • Each expected count should be ≥ 5
    • If some expected counts < 5, use Fisher’s exact test instead
  3. Random sampling: Data should come from random samples

Warning signs: - Small expected counts → Fisher’s exact test - Non-independent observations → Different methods needed

# Fisher's exact test for small samples
fisher.test(matrix(c(3, 1, 1, 5), nrow = 2))

    Fisher's Exact Test for Count Data

data:  matrix(c(3, 1, 1, 5), nrow = 2)
p-value = 0.1905
alternative hypothesis: true odds ratio is not equal to 1
95 percent confidence interval:
   0.40161 930.24692
sample estimates:
odds ratio 
  10.29391 

Summary: Categorical Data Analysis

Test Type Question R Function Key Output
Goodness of Fit Do observed frequencies match expected? chisq.test(x) p-value
Independence Are two variables associated? chisq.test(table) p-value, residuals
Visualization See patterns mosaicplot(..., shade=T) Color-coded residuals

Workflow:

  1. Create table with table() or use built-in dataset

  2. Visualize with barplot() or mosaicplot()

  3. Test with chisq.test()

  4. Examine residuals to understand patterns

  5. Report findings with context

Practical Tips

Good practices:

✓ Always visualize before testing
✓ Check expected counts (should be ≥ 5)
✓ Examine residuals to understand where associations exist
✓ Report both statistical and practical significance

Common mistakes:

✗ Ignoring small expected counts
✗ Testing without looking at the data first
✗ Only reporting p-values without interpretation
✗ Treating ordinal categories as nominal

Exploratory Data Analysis (EDA)

What is EDA?

John Tukey and other statisticians devised a collection of methods for exploratory data analysis.

Key distinction:

  • Confirmatory analysis → drawing inferential conclusions, hypothesis testing
  • Exploratory methods → discovering patterns, few distributional assumptions

Today’s goal: Learn to explore data systematically before formal analysis

The Four R’s of EDA

Tukey’s framework centers on four themes:

  1. Revelation – graphical displays, discovering patterns
  2. Resistance – methods insensitive to extreme observations (outliers)
  3. Residuals – focus on deviations from fitted models
  4. Reexpression – transforming data to reveal hidden patterns

We’ll explore each through practical examples!

Part 1: Revelation & Resistance

Case Study: College Data

Dataset: 1995 data about colleges from U.S. News and World Report

library(ISLR)
head(College)
                             Private Apps Accept Enroll Top10perc Top25perc
Abilene Christian University     Yes 1660   1232    721        23        52
Adelphi University               Yes 2186   1924    512        16        29
Adrian College                   Yes 1428   1097    336        22        50
Agnes Scott College              Yes  417    349    137        60        89
Alaska Pacific University        Yes  193    146     55        16        44
Albertson College                Yes  587    479    158        38        62
                             F.Undergrad P.Undergrad Outstate Room.Board Books
Abilene Christian University        2885         537     7440       3300   450
Adelphi University                  2683        1227    12280       6450   750
Adrian College                      1036          99    11250       3750   400
Agnes Scott College                  510          63    12960       5450   450
Alaska Pacific University            249         869     7560       4120   800
Albertson College                    678          41    13500       3335   500
                             Personal PhD Terminal S.F.Ratio perc.alumni Expend
Abilene Christian University     2200  70       78      18.1          12   7041
Adelphi University               1500  29       30      12.2          16  10527
Adrian College                   1165  53       66      12.9          30   8735
Agnes Scott College               875  92       97       7.7          37  19016
Alaska Pacific University        1500  76       72      11.9           2  10922
Albertson College                 675  67       73       9.4          11   9727
                             Grad.Rate
Abilene Christian University        60
Adelphi University                  56
Adrian College                      54
Agnes Scott College                 59
Alaska Pacific University           15
Albertson College                   55

Question: What patterns can we discover about graduation rates?

Revelation: Initial Visualization

Let’s start with a simple view of graduation rates:

stripchart(College$Grad.Rate, method="stack", pch=19,
           xlab="Graduation Rate")

What do you notice?

  • Distribution shape

  • Potential outliers

  • Range of values

Revelation: Comparing Groups

Does private vs. public status matter?

stripchart(College$Grad.Rate ~ College$Private, method="stack", pch=19,
           xlab="Graduation Rate",
           ylab="Private College")

Observation: Private colleges appear to have higher graduation rates overall

Resistance: Boxplots

Boxplots are resistant – they’re not overly influenced by outliers

b.output <- boxplot(Grad.Rate ~ Private, data=College,
                   horizontal = TRUE,
                   ylab="Private", xlab="Graduation Rate")

The boxplot clearly shows the median and quartiles for each group

Activity 7: Understanding Boxplot Output

Partner work (10 minutes)

The boxplot object stores useful information:

b.output$stats
     [,1] [,2]
[1,]   24   24
[2,]   46   58
[3,]   55   69
[4,]   65   81
[5,]   93  100
## 2nd row is the 1st quartiles
## 3rd row is the medians
## 4th row is the 3rd quartiles

Your tasks:

  1. Extract and examine the outliers: b.output$out and b.output$group
  2. Which group (private or public) has outliers?
  3. Calculate the IQR (Interquartile Range) for each group manually
  4. Create a similar boxplot comparing Outstate (out-of-state tuition) by Private status

Discuss: Why might outliers exist in graduation rates? Are they errors or legitimate cases?

Extracting Outliers

Self-study guide

# Outlier values
b.output$out
 [1]  98 100  10  95  15  18 118  21  22  21  15  21
# Which group they belong to (1 = No, 2 = Yes for Private)
b.output$group
 [1] 1 1 1 1 2 2 2 2 2 2 2 2

Boxplots identify outliers as values beyond 1.5 × IQR from the quartiles

Part 2: Residuals & Resistant Lines

Relationships Between Variables

Question: How does student quality (top 25% of HS class) relate to graduation rate?

plot(College$Top25perc, College$Grad.Rate, 
     xlab="Percent of new students from top 25% of class", 
     ylab="Graduation Rate")

Clear positive relationship – but how do we quantify it?

Resistant Lines: Tukey’s Method

Tukey’s resistant line is robust to outliers:

  • Divides data into three regions (left, middle, right)
  • Computes “resistant” summary points for each region
  • Fits a line through these summary points
fit = line(College$Top25perc, College$Grad.Rate)
coef(fit)
[1] 41.3658537  0.4390244

Interpretation: For every 1% increase in top 25% composition, graduation rate increases by ~0.43%

Comparing Regular vs. Resistant Regression

plot(College$Top25perc, College$Grad.Rate, 
     xlab="Percent of new students from top 25% of class", 
     ylab="Graduation Rate")
abline(coef(fit), col='red', lwd=2)  # Resistant line
abline(lm(Grad.Rate~Top25perc, data=College), col='blue', lwd=2)  # OLS
legend("bottomright", legend=c("Resistant Line", "OLS"), 
       col=c("red", "blue"), lwd=2)

Notice how similar they are when outliers aren’t extreme!

Focusing on Residuals

Residuals show what the model misses:

plot(College$Top25perc, fit$residuals, xlab="Top 25%",
     ylab="Residual", pch=19)
abline(h=0, col="red", lty=2)

What to look for:

  • Random scatter → good model

  • Patterns → model is missing something

  • Outliers → unusual cases to investigate

Activity 8: Exploring Relationships

Partner work (10 minutes)

Using the College dataset:

  1. Create a scatterplot of Accept (applications accepted) vs Apps (applications received)
  2. Fit both a resistant line (line()) and regular regression (lm())
  3. Plot both lines on your scatterplot (use different colors)
  4. Create a residual plot for the resistant line
# Starter code:
plot(College$Apps, College$Accept)
# Your code here...

Discuss: - Are there outliers? - Do the two lines differ substantially? - What does the residual plot tell you?

Part 3: Reexpression (Transformations)

When Linear Models Don’t Fit

Example: BGSU enrollment from 1955 to 1970

bgsu = read.csv('data/bgsu.csv')
attach(bgsu)
plot(Year, Enrollment, pch=19)

Problem: The relationship is clearly not linear – it curves upward

Attempting a Linear Fit

attach(bgsu)
fit = lm(Enrollment ~ Year)
plot(Year, Enrollment, pch=19)
abline(fit, col="blue", lwd=2)
plot(Year, fit$residuals, xlab="Year", ylab="Residuals", pch=19)
abline(h=0, col="red", lty=2)

Red flag: Residuals show a clear pattern (not random!) → poor model fit

Reexpression: Log Transformation

Exponential growth model: \[\text{Enrollment} = a \times \exp(b \times \text{Year})\]

Taking the log of both sides: \[\log(\text{Enrollment}) = \log(a) + b \times \text{Year}\]

Now we have a linear relationship in log-transformed data!

Fitting the Transformed Model

bgsu$log.Enrollment = log(bgsu$Enrollment)
attach(bgsu)
plot(Year, log.Enrollment, ylab="Log(Enrollment)", pch=19)
fit2 = lm(log.Enrollment ~ Year)
abline(fit2, col="blue", lwd=2)
plot(Year, fit2$residuals, ylab="Residuals", pch=19)
abline(h=0, col="red", lty=2)

Much better! Residuals now appear random with no pattern

Interpreting Log-Transformed Models

fit2$coef
  (Intercept)          Year 
-153.25703366    0.08268126 

Interpretation: - Intercept: log(enrollment) when Year = 0 (not meaningful here) - Slope (0.063): Each year, log(enrollment) increases by 0.063 - This means enrollment grows by approximately 6.3% per year (exponential growth)

Activity 9: Transformation Practice

Partner work (10 minutes)

The dataset below shows population growth:

# Simulated data
years <- 1960:1980
population <- c(50, 55, 62, 70, 80, 93, 109, 128, 152, 182, 
                220, 268, 328, 404, 500, 623, 781, 984, 1247, 1589, 2034)
pop_data <- data.frame(Year = years, Population = population)

Your tasks:

  1. Plot Population vs Year – what pattern do you see?
  2. Fit a linear model and examine residuals
  3. Create a log(Population) variable and plot it vs Year
  4. Fit a linear model to the log-transformed data
  5. Compare the two residual plots

Discuss: Which model fits better? What does this suggest about population growth?

Common Transformations

Self-study guide

Different relationships need different transformations:

Pattern Transformation When to Use
Exponential growth log(y) Data growing multiplicatively
Power relationship log(x) and log(y) Both variables span large ranges
Square root sqrt(y) Count data with increasing variance
Inverse 1/x or 1/y Asymptotic relationships

Rule of thumb: If residuals show patterns, try transforming!

Summary: The Four R’s in Action

Theme What We Did Key Tool
Revelation Discovered patterns with graphics Stripcharts, scatterplots
Resistance Used outlier-robust methods Boxplots, resistant lines
Residuals Checked model fit Residual plots
Reexpression Transformed to reveal patterns Log transformation

Key principle: EDA is iterative – plot, model, check residuals, transform if needed, repeat!

EDA Workflow

1. PLOT the data (revelation)
   ↓
2. CHECK for outliers (resistance)
   ↓
3. FIT a model
   ↓
4. EXAMINE residuals
   ↓
5. TRANSFORM if needed (reexpression)
   ↓
6. REPEAT until satisfied

Remember: EDA comes before formal hypothesis testing!

Activity 10: Complete EDA Challenge

Partner work (15 minutes)

Using the mtcars dataset, perform a complete EDA:

Goal: Understand the relationship between hp (horsepower) and mpg (fuel efficiency)

  1. Revelation: Create an appropriate visualization
  2. Resistance: Identify any outliers (use boxplots or examine extreme values)
  3. Fit models: Try both line() and lm()
  4. Residuals: Create and interpret residual plots
  5. Reexpression: If residuals show patterns, try log(mpg) or log(hp)
# Your code here:
data(mtcars)

Present: Be ready to share your findings – which model works best?

A More Modern Approach to EDA

  • Question-and-Answer Workflow: Use domain knowledge to guide explorations by asking specific questions related to your domain problem
  • Exploratory vs. Explanatory:
    • Exploratory: Preliminary visualizations produced quickly to uncover patterns
    • Explanatory: Polished, presentation-quality figures for external audiences
  • Choose Appropriate Visualizations: Match visualization type to variable types (numeric, categorical, time-based)

Source: Barter, R. L., & Yu, B. (2024). Veridical Data Science, Chapter 5. MIT Press. Available at https://vdsbook.com/05-data_viz

Common Techniques:

  • Histograms and boxplots for single variables
  • Scatterplots for two numeric variables
  • Line plots for trends over time
  • Bar charts for categorical comparisons

Source: Veridic Data Science

Describing a Single Variable:

Central Tendency:

  • Mean: Average value; sensitive to outliers

  • Median: Middle value; robust to outliers

Spread:

  • Variance/Standard Deviation: Measure of variability around the mean

  • Interquartile Range (IQR): Difference between 75th and 25th percentiles

Describing Relationships Between Variables:

Covariance & Correlation:

  • Covariance: Measures how two variables vary together (scale-dependent)

  • Correlation: Standardized measure between -1 and 1

Key Principle: Always ensure comparability when making comparisons (e.g., use rates per capita rather than raw counts when populations differ)

PCS Framework for Trustworthy EDA

Before presenting EDA results, evaluate them using the PCS Framework:

Predictability

  • Verify findings using external data sources
  • Check if results appear in other studies or literature
  • Test if findings hold across different time periods

Computability

  • Document all code and computational steps
  • Ensure reproducibility of analyses

PCS Framework for Trustworthy EDA

Before presenting EDA results, evaluate them using the PCS Framework:

Stability

Assess robustness to perturbations:

  • Data perturbations: Add noise to test if conclusions hold

  • Cleaning judgment calls: Try alternative imputation methods

  • Visualization choices: Test different chart types, filtering decisions, color schemes

Bottom Line: Your EDA findings should be stable across reasonable alternative judgment calls and resistant to plausible data variations.

Resources for EDA

Books:

  • Exploratory Data Analysis by John Tukey (classic!)

  • R for Data Science by Wickham & Grolemund (modern approach)

  • Veridic Data Science by Bin Yu & Rebecca L. Barter (modern approach)

R Functions to Remember:

  • Visualization: plot(), boxplot(), hist(), stripchart()

  • Resistant methods: median(), IQR(), line()

  • Transformations: log(), sqrt(), exp()

  • Residuals: residuals() or fit$residuals

Practice: EDA skills improve with experience – explore many datasets!

Graphical Summaries

Overview: Two Approaches to Graphics in R

Today we’ll cover:

  1. Base R Graphics (brief overview - reference material provided)
  2. ggplot2 (our main focus - modern, powerful approach)

In-class focus: We’ll spend most time on ggplot2 as it’s more intuitive and widely used in modern data science.

Base R Graphics: Quick Introduction

Basic Plotting with Base R

hitting.data = read.table("data/batting.history.txt", header=TRUE, sep="\t")
attach(hitting.data)
plot(Year, HR)

Key function: plot() creates basic visualizations

We’ll see a few examples, but detailed customization is in the reference slides.

Activity 4: Base R Plotting Practice

Partner work (10 minutes)

Using the mtcars dataset:

  1. Create a scatterplot of mpg vs wt (weight)
  2. Add axis labels: “Weight (1000 lbs)” and “Miles per Gallon”
  3. Add a title: “Fuel Efficiency vs Car Weight”
  4. Change the plotting symbol to filled circles (pch = 19)

Bonus: Add a lowess smoothing line in red

# Starter code:
data(mtcars)
plot(mtcars$wt, mtcars$mpg)

Discuss: What relationship do you observe?

Base R Graphics: Reference Material

⚠️ Self-Study Guide - Not Covered in Detail in Class

The following slides contain useful reference material for customizing base R graphics. Review these on your own time.

Reference: Adjusting Plot Attributes

Self-study guide

Parameters in the function plot:

plot(Year, HR, xlab = "Season",
     ylab = "Avg HR Hit Per Team Per Game",
     main = "Home Run Hitting in the MLB Across Seasons",
     sub = " (a) ")

Reference: Changing Plot Type and Symbols

Self-study guide

plot(Year, HR, xlab = "Season", type = "b",
     ylab = "Avg HR Hit Per Team Per Game",
     main = "Home Run Hitting in the MLB Across Seasons")

Common type values: - "p" for points (default) - "l" for lines - "b" for both - "h" for histogram-like vertical lines

Reference: Adding Layers

Self-study guide

plot(Year, HR, xlab = "Season", pch = 19, cex = 0.9,
     ylab = "Avg HR Hit Per Team Per Game",
     main = "Home Run Hitting in the MLB Across Seasons")
lines(lowess(Year, HR), lwd = 2)

Reference: Multiple Plots with Lattice

Self-study guide

Mileage depends on the number of cylinders:

library(lattice)
xyplot(mpg ~ wt | cyl, data = mtcars,
       xlab = "Weight", ylab = "Mileage",
       pch = 19, cex = 1.5)

Reference: Density Plots with Lattice

Self-study guide

densityplot(~ wt, groups = cyl, data = mtcars, 
            auto.key = list(space = "top"))

Reference: Low-Level Functions

Self-study guide

Creating custom plots from scratch:

plot.new()
plot.window(xlim = c(-1.5,1.5), ylim = c(-1.5,1.5), pty = "s")
theta = seq(0, 2*pi, length = 200)
x = cos(theta); y = sin(theta)
lines(x, y)
text(0, 0, "center")

Reference: Multiple Figures

Self-study guide

par(mfrow = c(2,1))
fit = lowess(Year, HR, f = 1/12)
Residual = HR - fit$y
plot(Year, HR, xlab = "Season",
     ylab = "Avg HR Hit Per Team Per Game")
plot(Year, Residual, xlab = "Season", ylab = "Residuals")

Reference: Customization Options

Self-study guide

Common parameters:

  • xlim and ylim – axis ranges
  • xaxt="n" and yaxt="n" – suppress axes
  • pch – plotting symbols (1-25)
  • cex – size of symbols
  • lwd – line width
  • col – colors
  • lty – line types (1-6)

Reference: Colors and Text

Self-study guide

# View available colors
colors()
  [1] "white"                "aliceblue"            "antiquewhite"        
  [4] "antiquewhite1"        "antiquewhite2"        "antiquewhite3"       
  [7] "antiquewhite4"        "aquamarine"           "aquamarine1"         
 [10] "aquamarine2"          "aquamarine3"          "aquamarine4"         
 [13] "azure"                "azure1"               "azure2"              
 [16] "azure3"               "azure4"               "beige"               
 [19] "bisque"               "bisque1"              "bisque2"             
 [22] "bisque3"              "bisque4"              "black"               
 [25] "blanchedalmond"       "blue"                 "blue1"               
 [28] "blue2"                "blue3"                "blue4"               
 [31] "blueviolet"           "brown"                "brown1"              
 [34] "brown2"               "brown3"               "brown4"              
 [37] "burlywood"            "burlywood1"           "burlywood2"          
 [40] "burlywood3"           "burlywood4"           "cadetblue"           
 [43] "cadetblue1"           "cadetblue2"           "cadetblue3"          
 [46] "cadetblue4"           "chartreuse"           "chartreuse1"         
 [49] "chartreuse2"          "chartreuse3"          "chartreuse4"         
 [52] "chocolate"            "chocolate1"           "chocolate2"          
 [55] "chocolate3"           "chocolate4"           "coral"               
 [58] "coral1"               "coral2"               "coral3"              
 [61] "coral4"               "cornflowerblue"       "cornsilk"            
 [64] "cornsilk1"            "cornsilk2"            "cornsilk3"           
 [67] "cornsilk4"            "cyan"                 "cyan1"               
 [70] "cyan2"                "cyan3"                "cyan4"               
 [73] "darkblue"             "darkcyan"             "darkgoldenrod"       
 [76] "darkgoldenrod1"       "darkgoldenrod2"       "darkgoldenrod3"      
 [79] "darkgoldenrod4"       "darkgray"             "darkgreen"           
 [82] "darkgrey"             "darkkhaki"            "darkmagenta"         
 [85] "darkolivegreen"       "darkolivegreen1"      "darkolivegreen2"     
 [88] "darkolivegreen3"      "darkolivegreen4"      "darkorange"          
 [91] "darkorange1"          "darkorange2"          "darkorange3"         
 [94] "darkorange4"          "darkorchid"           "darkorchid1"         
 [97] "darkorchid2"          "darkorchid3"          "darkorchid4"         
[100] "darkred"              "darksalmon"           "darkseagreen"        
[103] "darkseagreen1"        "darkseagreen2"        "darkseagreen3"       
[106] "darkseagreen4"        "darkslateblue"        "darkslategray"       
[109] "darkslategray1"       "darkslategray2"       "darkslategray3"      
[112] "darkslategray4"       "darkslategrey"        "darkturquoise"       
[115] "darkviolet"           "deeppink"             "deeppink1"           
[118] "deeppink2"            "deeppink3"            "deeppink4"           
[121] "deepskyblue"          "deepskyblue1"         "deepskyblue2"        
[124] "deepskyblue3"         "deepskyblue4"         "dimgray"             
[127] "dimgrey"              "dodgerblue"           "dodgerblue1"         
[130] "dodgerblue2"          "dodgerblue3"          "dodgerblue4"         
[133] "firebrick"            "firebrick1"           "firebrick2"          
[136] "firebrick3"           "firebrick4"           "floralwhite"         
[139] "forestgreen"          "gainsboro"            "ghostwhite"          
[142] "gold"                 "gold1"                "gold2"               
[145] "gold3"                "gold4"                "goldenrod"           
[148] "goldenrod1"           "goldenrod2"           "goldenrod3"          
[151] "goldenrod4"           "gray"                 "gray0"               
[154] "gray1"                "gray2"                "gray3"               
[157] "gray4"                "gray5"                "gray6"               
[160] "gray7"                "gray8"                "gray9"               
[163] "gray10"               "gray11"               "gray12"              
[166] "gray13"               "gray14"               "gray15"              
[169] "gray16"               "gray17"               "gray18"              
[172] "gray19"               "gray20"               "gray21"              
[175] "gray22"               "gray23"               "gray24"              
[178] "gray25"               "gray26"               "gray27"              
[181] "gray28"               "gray29"               "gray30"              
[184] "gray31"               "gray32"               "gray33"              
[187] "gray34"               "gray35"               "gray36"              
[190] "gray37"               "gray38"               "gray39"              
[193] "gray40"               "gray41"               "gray42"              
[196] "gray43"               "gray44"               "gray45"              
[199] "gray46"               "gray47"               "gray48"              
[202] "gray49"               "gray50"               "gray51"              
[205] "gray52"               "gray53"               "gray54"              
[208] "gray55"               "gray56"               "gray57"              
[211] "gray58"               "gray59"               "gray60"              
[214] "gray61"               "gray62"               "gray63"              
[217] "gray64"               "gray65"               "gray66"              
[220] "gray67"               "gray68"               "gray69"              
[223] "gray70"               "gray71"               "gray72"              
[226] "gray73"               "gray74"               "gray75"              
[229] "gray76"               "gray77"               "gray78"              
[232] "gray79"               "gray80"               "gray81"              
[235] "gray82"               "gray83"               "gray84"              
[238] "gray85"               "gray86"               "gray87"              
[241] "gray88"               "gray89"               "gray90"              
[244] "gray91"               "gray92"               "gray93"              
[247] "gray94"               "gray95"               "gray96"              
[250] "gray97"               "gray98"               "gray99"              
[253] "gray100"              "green"                "green1"              
[256] "green2"               "green3"               "green4"              
[259] "greenyellow"          "grey"                 "grey0"               
[262] "grey1"                "grey2"                "grey3"               
[265] "grey4"                "grey5"                "grey6"               
[268] "grey7"                "grey8"                "grey9"               
[271] "grey10"               "grey11"               "grey12"              
[274] "grey13"               "grey14"               "grey15"              
[277] "grey16"               "grey17"               "grey18"              
[280] "grey19"               "grey20"               "grey21"              
[283] "grey22"               "grey23"               "grey24"              
[286] "grey25"               "grey26"               "grey27"              
[289] "grey28"               "grey29"               "grey30"              
[292] "grey31"               "grey32"               "grey33"              
[295] "grey34"               "grey35"               "grey36"              
[298] "grey37"               "grey38"               "grey39"              
[301] "grey40"               "grey41"               "grey42"              
[304] "grey43"               "grey44"               "grey45"              
[307] "grey46"               "grey47"               "grey48"              
[310] "grey49"               "grey50"               "grey51"              
[313] "grey52"               "grey53"               "grey54"              
[316] "grey55"               "grey56"               "grey57"              
[319] "grey58"               "grey59"               "grey60"              
[322] "grey61"               "grey62"               "grey63"              
[325] "grey64"               "grey65"               "grey66"              
[328] "grey67"               "grey68"               "grey69"              
[331] "grey70"               "grey71"               "grey72"              
[334] "grey73"               "grey74"               "grey75"              
[337] "grey76"               "grey77"               "grey78"              
[340] "grey79"               "grey80"               "grey81"              
[343] "grey82"               "grey83"               "grey84"              
[346] "grey85"               "grey86"               "grey87"              
[349] "grey88"               "grey89"               "grey90"              
[352] "grey91"               "grey92"               "grey93"              
[355] "grey94"               "grey95"               "grey96"              
[358] "grey97"               "grey98"               "grey99"              
[361] "grey100"              "honeydew"             "honeydew1"           
[364] "honeydew2"            "honeydew3"            "honeydew4"           
[367] "hotpink"              "hotpink1"             "hotpink2"            
[370] "hotpink3"             "hotpink4"             "indianred"           
[373] "indianred1"           "indianred2"           "indianred3"          
[376] "indianred4"           "ivory"                "ivory1"              
[379] "ivory2"               "ivory3"               "ivory4"              
[382] "khaki"                "khaki1"               "khaki2"              
[385] "khaki3"               "khaki4"               "lavender"            
[388] "lavenderblush"        "lavenderblush1"       "lavenderblush2"      
[391] "lavenderblush3"       "lavenderblush4"       "lawngreen"           
[394] "lemonchiffon"         "lemonchiffon1"        "lemonchiffon2"       
[397] "lemonchiffon3"        "lemonchiffon4"        "lightblue"           
[400] "lightblue1"           "lightblue2"           "lightblue3"          
[403] "lightblue4"           "lightcoral"           "lightcyan"           
[406] "lightcyan1"           "lightcyan2"           "lightcyan3"          
[409] "lightcyan4"           "lightgoldenrod"       "lightgoldenrod1"     
[412] "lightgoldenrod2"      "lightgoldenrod3"      "lightgoldenrod4"     
[415] "lightgoldenrodyellow" "lightgray"            "lightgreen"          
[418] "lightgrey"            "lightpink"            "lightpink1"          
[421] "lightpink2"           "lightpink3"           "lightpink4"          
[424] "lightsalmon"          "lightsalmon1"         "lightsalmon2"        
[427] "lightsalmon3"         "lightsalmon4"         "lightseagreen"       
[430] "lightskyblue"         "lightskyblue1"        "lightskyblue2"       
[433] "lightskyblue3"        "lightskyblue4"        "lightslateblue"      
[436] "lightslategray"       "lightslategrey"       "lightsteelblue"      
[439] "lightsteelblue1"      "lightsteelblue2"      "lightsteelblue3"     
[442] "lightsteelblue4"      "lightyellow"          "lightyellow1"        
[445] "lightyellow2"         "lightyellow3"         "lightyellow4"        
[448] "limegreen"            "linen"                "magenta"             
[451] "magenta1"             "magenta2"             "magenta3"            
[454] "magenta4"             "maroon"               "maroon1"             
[457] "maroon2"              "maroon3"              "maroon4"             
[460] "mediumaquamarine"     "mediumblue"           "mediumorchid"        
[463] "mediumorchid1"        "mediumorchid2"        "mediumorchid3"       
[466] "mediumorchid4"        "mediumpurple"         "mediumpurple1"       
[469] "mediumpurple2"        "mediumpurple3"        "mediumpurple4"       
[472] "mediumseagreen"       "mediumslateblue"      "mediumspringgreen"   
[475] "mediumturquoise"      "mediumvioletred"      "midnightblue"        
[478] "mintcream"            "mistyrose"            "mistyrose1"          
[481] "mistyrose2"           "mistyrose3"           "mistyrose4"          
[484] "moccasin"             "navajowhite"          "navajowhite1"        
[487] "navajowhite2"         "navajowhite3"         "navajowhite4"        
[490] "navy"                 "navyblue"             "oldlace"             
[493] "olivedrab"            "olivedrab1"           "olivedrab2"          
[496] "olivedrab3"           "olivedrab4"           "orange"              
[499] "orange1"              "orange2"              "orange3"             
[502] "orange4"              "orangered"            "orangered1"          
[505] "orangered2"           "orangered3"           "orangered4"          
[508] "orchid"               "orchid1"              "orchid2"             
[511] "orchid3"              "orchid4"              "palegoldenrod"       
[514] "palegreen"            "palegreen1"           "palegreen2"          
[517] "palegreen3"           "palegreen4"           "paleturquoise"       
[520] "paleturquoise1"       "paleturquoise2"       "paleturquoise3"      
[523] "paleturquoise4"       "palevioletred"        "palevioletred1"      
[526] "palevioletred2"       "palevioletred3"       "palevioletred4"      
[529] "papayawhip"           "peachpuff"            "peachpuff1"          
[532] "peachpuff2"           "peachpuff3"           "peachpuff4"          
[535] "peru"                 "pink"                 "pink1"               
[538] "pink2"                "pink3"                "pink4"               
[541] "plum"                 "plum1"                "plum2"               
[544] "plum3"                "plum4"                "powderblue"          
[547] "purple"               "purple1"              "purple2"             
[550] "purple3"              "purple4"              "red"                 
[553] "red1"                 "red2"                 "red3"                
[556] "red4"                 "rosybrown"            "rosybrown1"          
[559] "rosybrown2"           "rosybrown3"           "rosybrown4"          
[562] "royalblue"            "royalblue1"           "royalblue2"          
[565] "royalblue3"           "royalblue4"           "saddlebrown"         
[568] "salmon"               "salmon1"              "salmon2"             
[571] "salmon3"              "salmon4"              "sandybrown"          
[574] "seagreen"             "seagreen1"            "seagreen2"           
[577] "seagreen3"            "seagreen4"            "seashell"            
[580] "seashell1"            "seashell2"            "seashell3"           
[583] "seashell4"            "sienna"               "sienna1"             
[586] "sienna2"              "sienna3"              "sienna4"             
[589] "skyblue"              "skyblue1"             "skyblue2"            
[592] "skyblue3"             "skyblue4"             "slateblue"           
[595] "slateblue1"           "slateblue2"           "slateblue3"          
[598] "slateblue4"           "slategray"            "slategray1"          
[601] "slategray2"           "slategray3"           "slategray4"          
[604] "slategrey"            "snow"                 "snow1"               
[607] "snow2"                "snow3"                "snow4"               
[610] "springgreen"          "springgreen1"         "springgreen2"        
[613] "springgreen3"         "springgreen4"         "steelblue"           
[616] "steelblue1"           "steelblue2"           "steelblue3"          
[619] "steelblue4"           "tan"                  "tan1"                
[622] "tan2"                 "tan3"                 "tan4"                
[625] "thistle"              "thistle1"             "thistle2"            
[628] "thistle3"             "thistle4"             "tomato"              
[631] "tomato1"              "tomato2"              "tomato3"             
[634] "tomato4"              "turquoise"            "turquoise1"          
[637] "turquoise2"           "turquoise3"           "turquoise4"          
[640] "violet"               "violetred"            "violetred1"          
[643] "violetred2"           "violetred3"           "violetred4"          
[646] "wheat"                "wheat1"               "wheat2"              
[649] "wheat3"               "wheat4"               "whitesmoke"          
[652] "yellow"               "yellow1"              "yellow2"             
[655] "yellow3"              "yellow4"              "yellowgreen"         
# Adding text
plot(0, 0, type = "n", xlim = c(-1,1), ylim = c(-1,1))
text(0, 0.5, "Example text", col = "blue", cex = 1.5)

Tip: Use RColorBrewer package for colorblind-safe palettes

Reference: Exporting Graphs

Self-study guide

### Save as PDF
pdf(file = "plot.pdf")
plot(mtcars$wt, mtcars$mpg)
dev.off()
quartz_off_screen 
                2 
### Other formats: png(), jpeg(), etc.

ggplot2: Modern Data Visualization

Why ggplot2?

Advantages over base R:

  • Consistent syntax across plot types
  • Layered approach builds plots incrementally
  • Automatic legends and color schemes
  • Better defaults for publication-quality graphics
  • Easier faceting (multiple subplots)

Philosophy: Grammar of Graphics - build plots by combining data, aesthetics, and geometric objects

The ggplot2 Framework

Three essential components:

  1. Data – your dataset
  2. Aesthetics (aes()) – map variables to visual properties
  3. Geoms (geom_*()) – how to display the data
library(ggplot2)

library(readr)
pums <- read_csv('data/psam_p41.csv')
pumsReduced <- pums %>% 
  select(PUMA, SALARY=WAGP, WEIGHT=PWGTP, AGE=AGEP, RACE=RAC1P, HICOV, NATIVITY)

pumsFinal <- pumsReduced %>% select(-NATIVITY)
str(pumsFinal)
tibble [42,080 × 6] (S3: tbl_df/tbl/data.frame)
 $ PUMA  : chr [1:42080] "00705" "01301" "00200" "01321" ...
 $ SALARY: num [1:42080] 12000 5700 0 0 0 0 30000 0 0 1800 ...
 $ WEIGHT: num [1:42080] 5 68 81 36 66 34 144 88 50 48 ...
 $ AGE   : num [1:42080] 18 19 37 20 18 85 32 63 37 19 ...
 $ RACE  : num [1:42080] 1 1 1 1 6 1 1 1 3 9 ...
 $ HICOV : num [1:42080] 1 1 1 1 1 1 1 1 2 1 ...
ggplot(data = pumsFinal, aes(x = RACE)) +
  geom_bar()

Building Plots Layer by Layer

# Start with data and aesthetics
ggplot(pumsFinal, aes(x = SALARY))

# Add a geometric layer
ggplot(pumsFinal, aes(x = SALARY)) +
  geom_density()

# Add labels
ggplot(pumsFinal, aes(x = SALARY)) +
  geom_density() +
  labs(title = "Distribution of Salaries",
       x = "Annual Salary",
       y = "Density")

Common Geoms: One Variable

Categorical variable:

ggplot(pumsFinal, aes(x = RACE)) +
  geom_bar()

Continuous variable:

# Histogram
ggplot(pumsFinal, aes(x = SALARY)) +
  geom_histogram(bins = 30)

# Density plot
ggplot(pumsFinal, aes(x = SALARY)) +
  geom_density()

Common Geoms: Two Variables

Scatterplot:

ggplot(mtcars, aes(x = wt, y = mpg)) +
  geom_point()

Boxplot by group:

ggplot(pumsFinal, aes(x = RACE, y = SALARY)) +
  geom_boxplot()

Bar plot (summary statistic):

ggplot(pumsFinal, aes(x = RACE, y = SALARY)) +
  geom_bar(stat = "summary", fun = "mean")

Activity 5: Basic ggplot2 Practice

Partner work (10 minutes)

Using the palmerpenguins::penguins dataset:

  1. Create a histogram of bill_length_mm
  2. Create a scatterplot of bill_length_mm vs bill_depth_mm
  3. Add appropriate axis labels and a title
  4. Change the point color to “steelblue”
# Load the data
library(palmerpenguins)
data(penguins)

# Your code here:

Discuss: What patterns do you notice in bill dimensions?

Understanding Aesthetics (aes())

Inside aes() - maps data to visuals:

ggplot(pumsFinal, aes(x = RACE, y = SALARY, fill = RACE)) +
  geom_boxplot()

Outside aes() - fixed values:

ggplot(pumsFinal, aes(x = RACE, y = SALARY)) +
  geom_boxplot(fill = "lightblue")

Critical distinction: Use aes() when mapping to variables, set directly when using constants

Customizing Axes and Labels

ggplot(pumsFinal, aes(x = RACE, y = SALARY)) +
  geom_bar(stat = "summary", fun = "mean") +
  labs(
    title = "Average Salary by Race",
    subtitle = "Based on census data",
    x = "Race",
    y = "Average Salary ($)",
    caption = "Source: PUMS data"
  ) +
  theme_minimal()

Faceting: Multiple Subplots

By one variable:

ggplot(penguins, aes(x = bill_length_mm, y = bill_depth_mm)) +
  geom_point() +
  facet_wrap(~ species)

By two variables:

ggplot(penguins, aes(x = bill_length_mm, y = bill_depth_mm)) +
  geom_point() +
  facet_grid(species ~ island)

Activity 6: Advanced ggplot2

Partner work (10 minutes)

Using penguins data, create a visualization that:

  1. Shows the relationship between flipper length and body mass
  2. Colors points by species
  3. Uses facet_wrap() to create separate panels by island
  4. Adds a smooth trend line with geom_smooth()
  5. Uses an appropriate theme (try theme_bw() or theme_minimal())
# Your code here:

Discuss: - How do the relationships differ across islands? - Which species tends to be largest?

Comparing Base R vs ggplot2

Same plot, different approaches:

Base R:

plot(penguins$bill_length_mm, penguins$bill_depth_mm,
     col = as.numeric(penguins$species),
     xlab = "Bill Length (mm)", ylab = "Bill Depth (mm)")
legend("topright", legend = levels(penguins$species),
       col = 1:3, pch = 1)

ggplot2:

ggplot(penguins, aes(x = bill_length_mm, y = bill_depth_mm, 
                     color = species)) +
  geom_point() +
  labs(x = "Bill Length (mm)", y = "Bill Depth (mm)")

Summary: Graphical Summaries

Key takeaways:

  • Base R graphics are quick and simple but require more code for customization (see reference slides for details)
  • ggplot2 uses a layered grammar:
    • Data + Aesthetics + Geoms
    • Easy to customize and extend
    • Better for complex visualizations

Recommendation: Use ggplot2 for most tasks, especially for publication-quality graphics

Resources for Graphics

General - Data Visualization Catalogue - Veridic Data Science

ggplot2: - ggplot2 documentation - R Graphics Cookbook - ggplot2 cheat sheet

Base R: - See reference slides (gray background) - ?plot, ?par for help

Practice: The best way to learn is by creating many plots!