Exploring Data Frames

Week 2 · Tuesday, Oct 6 · Lecture

Dr. Marcela Alfaro Córdoba

Today’s plan

Time What
1:30 Recap: two sentences from Thursday
1:40 Project folders and relative paths
1:55 Loading real data, and the first look
2:15 Reading str() line by line
2:30 Adding and removing rows and columns
2:50 A loading script you can source()

Learning goals: load a CSV you didn’t make, say what’s in it, and change its shape on purpose.

Type along. Still no AI. Your DS printouts are due Thursday.

Recap: the two sentences

  1. Each column of a data frame is a vector.
  2. Every element of a vector has the same type.

⏱ 4 min

On paper, no laptop. What does each line print?

d <- data.frame(id = 1:3, tag = c("a", "b", "c"))
class(d[1])
class(d[[1]])
length(d)
typeof(d)

"data.frame" · "integer" · 2 (columns!) · "list"

Where your data lives

One folder per project

stat266a/
├── data/          <- raw data. NEVER edited by hand.
├── scripts/       <- your .R files
├── figures/       <- plots you save
├── week01/
└── week02/
  • Open the project with File → Open Folder…, every time
  • Your R session then starts in stat266a/, which is what makes the next slide work

Important

data/ is read-only. If you fix something, you fix it in code, so the fix is written down and repeatable. Never in Excel.

Absolute vs. relative paths

# Absolute: works on exactly one computer, forever
read.csv("/Users/marcela/Documents/stat266a/data/gapminder_data.csv")

# Relative: works on any computer that has the project folder
read.csv("data/gapminder_data.csv")
getwd()          # where am I?
list.files()     # what's here?
list.files("data")

Warning

Error in file(file, "rt") : cannot open the connection plus No such file or directory almost always means your working directory is wrong, not that the file is missing. Run getwd() first.

Load the gapminder data

Life expectancy, population and GDP per person, by country and year.

gapminder <- read.csv("data/gapminder_data.csv")

Don’t have the file? Download it once, in code:

download.file(
  "https://swcarpentry.github.io/r-novice-gapminder/data/gapminder_data.csv",
  destfile = "data/gapminder_data.csv")

Downloading in your script rather than clicking a link means the next person — including you in March — can rebuild the whole thing from the script alone.

The first look

Six functions, every single time

str(gapminder)       # size, column types, first values
summary(gapminder)   # per-column summaries
dim(gapminder)       #> [1] 1704    6      rows, columns
nrow(gapminder); ncol(gapminder)
colnames(gapminder)
head(gapminder)      # first 6 rows; head(gapminder, 10) for 10
tail(gapminder, 3)   # last 3

And two that make more sense now than they used to:

typeof(gapminder)    #> [1] "list"    a data frame IS a list
length(gapminder)    #> [1] 6        ...of 6 columns

str(), line by line

str(gapminder)
#> 'data.frame':    1704 obs. of  6 variables:
#>  $ country  : chr  "Afghanistan" "Afghanistan" "Afghanistan" ...
#>  $ year     : int  1952 1957 1962 1967 1972 ...
#>  $ pop      : num  8425333 9240934 10267083 ...
#>  $ continent: chr  "Asia" "Asia" "Asia" "Asia" ...
#>  $ lifeExp  : num  28.8 30.3 32 34 36.1 ...
#>  $ gdpPercap: num  779 821 853 836 740 ...
  • Line 1: the class, then obs. = rows, variables = columns
  • Every line after: one column — $ name : type first values ...
  • chr character · num double · int integer · logi logical

1704 = 142 countries × 12 years. Knowing the shape should be is how you notice when it isn’t.

Factors in str() output

Read the same file, asking for text columns as factors:

gapminder <- read.csv("data/gapminder_data.csv", stringsAsFactors = TRUE)
str(gapminder)
#>  $ country  : Factor w/ 142 levels "Afghanistan",..: 1 1 1 1 1 ...
#>  $ continent: Factor w/ 5 levels "Africa","Americas",..: 3 3 3 3 3 ...
  • Factor w/ 142 levels tells you there are exactly 142 distinct countries, for free
  • The numbers are the codes, pointing at the levels
  • nlevels(gapminder$country), levels(gapminder$continent)

Note

A quick count of categories is what factors are good for. chr is friendlier for most other things. Neither is “the right one”.

What summary() adds

summary(gapminder$lifeExp)
#>    Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
#>   23.60   48.20   60.71   59.47   70.85   82.60
summary(gapminder$continent)
#>   Africa Americas     Asia   Europe  Oceania
#>      624      300      396      360       24

Three things to look for on a first pass:

  1. Impossible values. A Min. of -99 in a column of ages is a missing-data code, not an age.
  2. NA's, which summary() reports as an extra line only when they exist.
  3. Counts that surprise you. Oceania has 24 rows: 2 countries × 12 years.

Don’t trust the top of a file

head() shows the tidiest part of most datasets. Problems live in the middle and at the end.

tail(gapminder, 15)
gapminder[sample(nrow(gapminder), 5), ]     # 5 random rows
# Other cheap sanity checks
range(gapminder$year)            #> [1] 1952 2007
sum(is.na(gapminder$lifeExp))    #> [1] 0        count missing
length(unique(gapminder$country))
table(gapminder$year)            # same number of rows per year?

Tip

View(gapminder) opens a spreadsheet-like viewer. Look, sort, filter — but every change goes in the script, never in the viewer.

Your turn: look beyond the top

⏱ 7 min

  1. Store the last 8 rows of gapminder in bottom
  2. Store 5 randomly chosen rows (no row twice) in spot_check
  3. Run spot_check twice. Why does it change? Then run it again after set.seed(266) twice. What changed?
  4. Using table(), check that every year has the same number of rows
bottom <- tail(gapminder, 8)
spot_check <- gapminder[sample(nrow(gapminder), 5), ]

set.seed() makes random choices reproducible: same seed, same rows, on any computer.

Changing shape

New columns from old ones

gapminder$pop_millions <- gapminder$pop / 1e6
gapminder$gdp_total <- gapminder$gdpPercap * gapminder$pop
ncol(gapminder)      #> [1] 8

Or bind a vector on as a column:

long_lived <- gapminder$lifeExp > 70
gapminder2 <- cbind(gapminder, long_lived)
table(long_lived)
#> long_lived
#> FALSE  TRUE
#>  1217   487

$<- changes the data frame in place. cbind() returns a new one, so you have to assign it to something.

cbind() and its one rule

age <- c(2, 3, 5, 12)
cbind(gapminder, age)
#> Error in data.frame(..., check.names = FALSE) :
#>   arguments imply differing number of rows: 1704, 4

The column must be as long as the data frame has rows:

nrow(gapminder)    #> [1] 1704
length(age)        #> [1] 4

Warning

If the length divides evenly into nrow(), R sometimes recycles it instead of erroring. You get a full column of nonsense, quietly. Always compare length() to nrow() yourself.

New rows with rbind()

extra <- data.frame(country = "Chile", year = 2012, pop = 17.4e6,
                    continent = "Americas", lifeExp = 79.6,
                    gdpPercap = 15300, pop_millions = 17.4,
                    gdp_total = 15300 * 17.4e6)
gapminder3 <- rbind(gapminder, extra)
nrow(gapminder3)     #> [1] 1705
tail(gapminder3, 1)

rbind() needs the same columns, by name. Miss one and you get:

#> Error in rbind(...) : numbers of columns of arguments do not match

A list() works as a row too: rbind(d, list("Chile", 2012, ...)). In that case the order has to match, which is easier to get wrong.

A trap with factors

If country is a factor, "Chile" must be one of its levels:

gm_f <- read.csv("data/gapminder_data.csv", stringsAsFactors = TRUE)
nlevels(gm_f$country)          #> [1] 142

Modern R (4.1+) adds the new level for you. Older code, older R, or a stricter function may instead give you NA and a warning:

#> Warning: invalid factor level, NA generated

Tip

After any rbind(), check: tail(d, 1) and str(d). You’re looking for an NA that appeared without being asked for.

Removing rows and columns

gapminder[-3, ]          # everything except row 3
gapminder[-(1:12), ]     # drop the first 12 rows
gapminder[, -6]          # everything except column 6
head(gapminder[, -c(7, 8)])

Counting columns is fragile. Name them instead:

drop <- names(gapminder) %in% c("pop_millions", "gdp_total")
head(gapminder[, !drop])

%in% gives a logical vector — one TRUE/FALSE per column. ! flips it. This pattern still reads correctly six months later, and after someone adds a column.

Duplicating rows, by accident

d <- data.frame(id = 1:2, tag = c("a", "b"))
d <- rbind(d, d)
d
#>   id tag
#> 1  1   a
#> 2  2   b
#> 3  1   a
#> 4  2   b

rbind() doesn’t check whether it’s stacking the same data twice. Run the same chunk twice and your data silently doubles.

nrow(d)                  #> [1] 4
sum(duplicated(d))       #> [1] 2
d[!duplicated(d), ]

Important

This is the strongest argument for a loading script: it always starts from the file, so re-running it can’t accumulate.

Missing values

df <- data.frame(x = c(1, NA, 3), y = c("a", "b", NA))
sum(is.na(df))              #> [1] 2
colSums(is.na(df))          #> x y
                            #> 1 1
mean(df$x)                  #> [1] NA
mean(df$x, na.rm = TRUE)    #> [1] 2
na.omit(df)                 # drops ROW 2 and ROW 3

Warning

na.omit() drops the whole row if any column is missing. On a wide dataset that can quietly throw out most of your data. Always compare nrow() before and after, and say in your report how many rows you lost.

Your turn: build a data frame

⏱ 10 min

In week02/trip.R:

  1. With data.frame(), create trip_log with one row and three columns: city (text), country (text), nights (a number)
  2. With rbind(), add two more stops
  3. With cbind(), add a logical column rained (one value per stop)
  4. Run str(trip_log). Are the types what you expected?
  5. Add a column nights_per_city that is nights divided by sum(nights)
  6. Drop the row with the fewest nights, without typing its row number — find it with which.min()

Step 6 hint: trip_log[-which.min(trip_log$nights), ]. Code that finds the row beats code that hard-codes 3.

Your turn: read str() like a pro

⏱ 6 min

#> 'data.frame':    88 obs. of  6 variables:
#>  $ country  : Factor w/ 22 levels "Argentina","Australia",..: 3 3 3 3 1 ...
#>  $ continent: Factor w/ 5 levels "Africa","Americas",..: 2 2 2 2 2 ...
#>  $ year     : int  1967 1987 2002 2007 1967 ...
#>  $ lifeExp  : num  57.6 65.2 71 72.4 65.6 ...
#>  $ pop      : int  8835000 11626000 12300000 12420476 ...
#>  $ gdpPercap: num  2961 3448 3630 4172 8053 ...

Using only that output, no other functions: how many rows? how many columns? how many distinct countries? what type is year? what is the first country, by name?

88 · 6 · 22 · int · the 3rd level alphabetically — you can’t name it from this output alone, and noticing that is the point.

Work you can re-run

Put loading in a script

scripts/load-gapminder.R:

# Load and prepare the gapminder data.
# Run with: source("scripts/load-gapminder.R")

if (!file.exists("data/gapminder_data.csv")) {
  download.file(
    "https://swcarpentry.github.io/r-novice-gapminder/data/gapminder_data.csv",
    destfile = "data/gapminder_data.csv")
}

gapminder <- read.csv("data/gapminder_data.csv")
gapminder$pop_millions <- gapminder$pop / 1e6

str(gapminder)

source() runs a whole script

source("scripts/load-gapminder.R")

Everything the script creates appears in your session. In Positron, the Source button at the top of the editor does the same for the open file.

Tip

The restart test. Session → Restart R, then source("scripts/load-gapminder.R"). If gapminder is back exactly as it was, your analysis lives in your script, not in your session’s memory. If it isn’t, something you need was only ever typed in the console.

Do this at the end of every work session. It takes ten seconds and it is the whole idea of reproducibility.

Your turn: your own loading script

⏱ 10 min

  1. Create scripts/load-gapminder.R with the code from two slides ago
  2. Inspect it from the console without opening it: readLines("scripts/load-gapminder.R")
  3. Restart R (Session → Restart R), then source() it
  4. Confirm: nrow(gapminder) and ncol(gapminder)
  5. Add one more prepared column to the script, restart, source again

Step 3 is the one that matters. A script that only works in an R session you’ve been typing in for two hours isn’t a script yet.

Key points

  • Open the project folder; use relative paths like data/file.csv
  • read.csv() reads a CSV; stringsAsFactors = TRUE makes text columns factors
  • str() gives size, column types and first values in one line each
  • Check tail() and a random sample, not just head()
  • cbind() adds columns (length must equal nrow()), rbind() adds rows (columns must match)
  • Negative indices and %in% remove; na.omit() drops whole rows
  • source() re-runs a saved script, so your data can always be rebuilt

Before you go

Thursday, Oct 8: bring your two printed DS PDFs (Lessons 3 and 4) and a pencil. We start with the discussion check.

  • Be ready to explain any line of your own solution
  • Be ready to predict what a line prints, without running it

Tonight or tomorrow: Lesson 5: Exploring Data Frames on the Intro to R site, to lock in today’s material. Not graded this week, still no AI.

Project: the proposal presentation is Tuesday, Oct 20. By this Thursday you should have a topic and a candidate data source. Come to office hours and talk it through with me.

Next week: subsetting data — pulling out exactly the rows and columns you want.