Author

Homework 2: Data Import, Cleaning, and Exploration in R

Working with the PalmTraits 1.0 Dataset

Overview

In this assignment, you will work with the PalmTraits 1.0 dataset, a comprehensive global database containing functional traits for over 2,500 palm species worldwide. This real-world dataset requires careful cleaning and manipulation before analysis. You’ll import the data, apply data cleaning techniques using base R functions, learn to use dplyr for data reshaping, and perform exploratory data analysis through descriptive statistics and visualizations.

Learning Objectives

By completing this assignment, you will be able to:

Import data from various file formats (CSV, Excel, text files) into R using appropriate functions
Perform data cleaning and manipulation using base R and dplyr functions
Calculate descriptive statistics (mean, median, variance, quantiles) and interpret their meaning in context
Create and interpret histograms, boxplots, scatterplots, and summary tables for exploratory data analysis

About the Dataset

The PalmTraits 1.0 database contains functional trait data for palm species (family Arecaceae) from around the world. Palms are ecologically important in tropical and subtropical ecosystems, and this database captures traits related to:

Growth forms (climbing, erect, acaulescent)
Stem characteristics (height, diameter, clustering)
Armature (presence of spines)
Leaf characteristics (blade length, number of leaves)
Fruit characteristics (length, width, color, shape)

Citation: Kissling, W.D., Balslev, H., Baker, W.J., Dransfield, J., Göldel, B., Lim, J.Y., Onstein, R.E., & Svenning, J.-C. (2019). PalmTraits 1.0, a species-level functional trait database of palms worldwide. Scientific Data, 6, 178. https://www.nature.com/articles/s41597-019-0189-0

Setup

Downloading the Dataset

Before starting your assignment, you need to download the PalmTraits dataset:

Go to: https://datadryad.org/stash/dataset/doi:10.5061/dryad.ts45225
Click on “Download Dataset” to get the full dataset (it’s free and open access)
Extract the downloaded files
You will need the file: PalmTraits_1.0.txt (this is a tab-delimited text file, under the Sep 09, 2020 tab)

Place this file in a data/ folder within your project directory for easy access and reproducibility.

Create Your Quarto Document

Create a new Quarto document (.qmd file) for this assignment. Begin with the setup chunk:

library(dplyr)      # for data reshaping (used in one section only)

# Set working directory if needed
# setwd("path/to/your/project")

Part 1: Data Import (20 points)

1.1 Import the PalmTraits Dataset

Import the main PalmTraits dataset, which is a tab-delimited text file.

# Import the tab-delimited file
# palm_data <- read.delim(file="data/PalmTraits_10.txt", 
#                         header = TRUE,
#                         sep = "\t",
#                         stringsAsFactors = FALSE)

Instructions: - Import the PalmTraits file using read.delim() or read.table() - Display the first 10 rows using head(palm_data, 10) - Check the structure using str(palm_data) - Display the dimensions using dim(palm_data) - Show the column names using names(palm_data)

1.2 Explore Key Variables

Examine some key trait variables in the dataset to understand what you’re working with.

# Look at a summary of the entire dataset
# Write your code here

# Examine specific continuous variables
# Write your code here

# Examine categorical variables
# Write your code here

Instructions: - Run summary() on the entire dataset - Identify at least 3 continuous (numeric) variables related to palm morphology - Identify at least 3 binary/categorical variables related to growth form or armature - Write 3-4 sentences describing the range and distribution of these variables based on the summary statistics

Part 2: Data Cleaning and Manipulation with Base R (35 points)

This section focuses on cleaning the PalmTraits dataset. Real-world ecological data often has missing values, inconsistencies, and variables that need transformation before analysis.

2.1 Identify Data Quality Issues (5 points)

Before cleaning, systematically examine your data for quality issues.

Instructions: - Count total missing values: sum(is.na(palm_data)) - Count missing values per column: colSums(is.na(palm_data)) - Identify columns with the most missing data - Check for duplicate rows: sum(duplicated(palm_data)) - Check the data types: str(palm_data) - Look for potential outliers in numeric variables using summary() and examining min/max values

2.2 Create a Working Subset (3 points)

To make the analysis more manageable and focused, create a subset of the PalmTraits data containing key variables of interest.

Instructions: Select the following variables (or similar ones if column names differ slightly): - Species name - At least 3 growth form variables (e.g., Climbing, Acaulescent, Erect) - Maximum stem height (MaxStemHeight_m) - Maximum blade length (MaxBladeLength_m)
- Average fruit length (AverageFruitLength_cm) - Average fruit width (AverageFruitWidth_cm) - Stem armature (StemArmed) - Any other variables you find interesting

# Create subset with key variables
# palm_subset <- palm_data[, c(list the variables of interest in here)]

Report: How many variables and observations are in your subset?

2.3 Handle Missing Values (8 points)

Address missing values in your subset using appropriate strategies.

Instructions:

Count missing values in your subset:

#colSums(is.na(palm_subset))
# # Calculate percentage missing for each variable
# Write your code here

Decide on a strategy for each variable with missing values. For this dataset:
- Binary variables (Climbing, Acaulescent, Erect, StemArmed): If missing, you might want to keep them as NA or convert to 0 (absent) based on ecological knowledge
- Continuous variables (stem height, blade length, fruit dimensions): Consider removing rows where KEY variables are missing, or document the missingness
Implement your strategy. For example:

# Option 1: Keep only complete cases for key continuous variables
#palm_clean <- palm_subset[complete.cases(palm_subset[, c("MaxStemHeight_m",                                       #"AverageFruitLength_cm")]), ]

# Option 2: Create a flag for missing data
#palm_subset$missing_fruit <- is.na(palm_subset$AverageFruitLength_cm)

2.4 Fix Data Types (5 points)

Ensure all columns have the correct data type for analysis.

Instructions: - Binary trait variables (Climbing, Acaulescent, Erect, StemArmed) should be factors or kept as numeric (0/1/2) - Continuous measurements should be numeric - Species names should be character type

# Check current types
#str(palm_clean)

# Convert binary variables to factors if needed

# Ensure numeric variables are numeric

# Show structure after conversions
# str(palm_clean)

Show the structure before and after conversions. Explain (2-3 sentences) why these type conversions are important for subsequent analyses.

2.5 Handle Outliers and Create Data Quality Flags (6 points)

Identify and decide how to handle potential outliers in morphological measurements.

Instructions:

For MaxStemHeight_m, identify potential outliers:

# Calculate quartiles and IQR
# Q1 <- quantile( )
# Q3 <- quantile( )
# IQR_value <- Q3 - Q1
# 
# lower_bound <- Q1 - 1.5 * IQR_value
# upper_bound <- Q3 + 1.5 * IQR_value
# 
# # Identify outliers
# outliers <- palm_clean$MaxStemHeight_m < lower_bound | 
#             palm_clean$MaxStemHeight_m > upper_bound
# sum(outliers, na.rm = TRUE)
# 
# # Look at the outlier values
# palm_clean[outliers & !is.na(outliers), c("SpecName", "MaxStemHeight_m")]

Research whether these are true outliers or biologically realistic extreme values:
- Note: Some palm species like Ceroxylon quindiuense can exceed 60m in height
- Climbing palms like Calamus species can exceed 100m in length
Make a decision: Keep, remove, or flag these values

2.6 Create New Derived Variables (8 points)

Create new variables that will be useful for analysis and interpretation.

Instructions:

Create a size category variable based on maximum stem height:

# Create size categories
# palm_clean$size_category <- ifelse(palm_clean$MaxStemHeight_m <= 5, "Small",
#                             ifelse(palm_clean$MaxStemHeight_m <= 15, "Medium",
#                             ifelse(palm_clean$MaxStemHeight_m <= 30, "Tall", "Very Tall")))
# 
# # Convert to factor with ordered levels
# palm_clean$size_category <- factor(palm_clean$size_category,
#                                     levels = c("Small", "Medium", "Tall", "Very Tall"))
# 
# table(palm_clean$size_category)

Create a fruit aspect ratio (length/width) to characterize fruit shape:

# Write your code here

Create a growth form summary variable that combines information:

# Write your code here

Part 3: Data Reshaping with dplyr (15 points)

Now that your data is clean, you’ll learn to reshape and summarize it using the dplyr package. The dplyr package provides powerful functions for data manipulation and uses the pipe operator %>% to chain operations together.

3.1 Understanding the Pipe Operator `%>%` (3 points)

The pipe operator %>% (read as “then”) takes the output from one function and passes it as the first argument to the next function. This makes code more readable by allowing you to read operations from left to right, top to bottom.

Syntax comparison:

# Without pipe - nested functions (hard to read):
# head(summary(palm_clean))

# With pipe - sequential operations (easy to read):
# palm_clean %>% summary() %>% head()

# Another example without pipe:
# subset(palm_clean, MaxStemHeight_m > 20)

# Same operation with pipe:
# palm_clean %>% subset(MaxStemHeight_m > 20)

The key principle: x %>% f(y) is equivalent to f(x, y). The pipe takes what’s on the left and inserts it as the first argument of the function on the right.

Your task: Write three examples using the pipe operator with your palm data: 1. Filter the data to show only climbing palms 2. Select three columns and display the first 5 rows 3. Calculate the mean stem height (without using dplyr summarize yet)

Show your code and briefly explain what each pipe operation does.

3.2 Using group_by() and summarize() for Aggregation (12 points)

The combination of group_by() and summarize() allows you to calculate summary statistics for different groups in your data. This is one of the most powerful operations in ecological data analysis.

Step-by-step explanation:

group_by() splits your data into groups based on one or more categorical variables
summarize() (or summarise()) calculates summary statistics for each group
The pipe %>% connects these operations in a readable sequence

Basic syntax:

# Template:
# your_data %>%
#   group_by(grouping_variable) %>%
#   summarize(
#     new_column_name = function(column_name)
#   )

Example with explanation:

# Let's say you want to calculate average stem height by size category:

# palm_clean %>%                           # Start with your clean data, THEN
#   group_by(size_category) %>%            # Group it by size category, THEN
#   summarize(                             # Calculate summaries:
#     avg_height = mean(MaxStemHeight_m, na.rm = TRUE),    # average height
#     sd_height = sd(MaxStemHeight_m, na.rm = TRUE),       # standard deviation
#     count = n()                                          # number of species
#   )

# The n() function counts the number of rows in each group

Your task - Create a comprehensive grouped summary:

Using your cleaned palm dataset, create a grouped summary that addresses the following requirements:

Choose a grouping variable: Use either size_category, primary_growth_form, or StemArmed (armed vs unarmed)
Calculate AT LEAST FIVE summary statistics for each group. You must include:
- A measure of central tendency for stem height: mean() or median()
- A measure of spread for stem height: var() or sd()
- The number of observations in each group: n()
- A measure of central tendency for fruit length: mean() or median()
- One additional summary of your choice (e.g., minimum, maximum, or summary of another variable like blade length)
Store this result in a new object called palm_summary

# Your code here - follow this template:
# palm_summary <- palm_clean %>%
#   group_by(___your_grouping_variable___) %>%
#   summarize(
#     mean_height = mean(MaxStemHeight_m, na.rm = TRUE),
#     var_height = var(MaxStemHeight_m, na.rm = TRUE),
#     count = n(),
#     median_fruit = median(AverageFruitLength_cm, na.rm = TRUE),
#     ___ = ___
#   )
# 
# # Display your summary table
# print(palm_summary)

Bonus challenge (optional, 2 extra points): Try grouping by TWO categorical variables to see how traits vary across combinations:

# add your code here

Important note: Remember to include na.rm = TRUE in functions like mean(), var(), sd(), median() to handle any remaining missing values.

Part 4: Descriptive Statistics with Base R (15 points)

Calculate and interpret descriptive statistics for your cleaned dataset using base R functions.

4.1 Comprehensive Statistics for Key Variables (8 points)

For THREE numeric variables in your palm dataset (e.g., MaxStemHeight_m, MaxBladeLength_m, AverageFruitLength_cm), calculate comprehensive statistics.

Instructions: For each variable, calculate:

# For MaxStemHeight_m (repeat for other variables):

# Mean
# mean(palm_clean$MaxStemHeight_m, na.rm = TRUE)

# Median
# median(palm_clean$MaxStemHeight_m, na.rm = TRUE)

# Variance
# var(palm_clean$MaxStemHeight_m, na.rm = TRUE)

# Standard deviation
# sd(palm_clean$MaxStemHeight_m, na.rm = TRUE)

# Range
# range(palm_clean$MaxStemHeight_m, na.rm = TRUE)

# Quantiles (0%, 25%, 50%, 75%, 100%)
# quantile(palm_clean$MaxStemHeight_m, probs = c(0, 0.25, 0.5, 0.75, 1), na.rm = TRUE)

# Coefficient of variation (CV) - a standardized measure of dispersion
# cv <- sd(palm_clean$MaxStemHeight_m, na.rm = TRUE) / 
#       mean(palm_clean$MaxStemHeight_m, na.rm = TRUE) * 100

Create a summary table organizing these results:

# Create a data frame to display results nicely
# add your code here
# print(stats_summary)

4.2 Frequency Tables for Categorical Variables (4 points)

Create frequency tables for categorical growth form and armature variables.

Instructions:

# Frequency table for primary growth form
# table(palm_clean$primary_growth_form)

# Proportions
# prop.table(table(palm_clean$primary_growth_form))

# Two-way table: growth form vs armature
# table(palm_clean$primary_growth_form, palm_clean$StemArmed)

# Two-way proportions (row percentages)
# prop.table(table(palm_clean$primary_growth_form, palm_clean$StemArmed), margin = 1)

4.3 Group Statistics Using Base R (3 points)

Calculate summary statistics by groups using base R functions (compare to your dplyr results from Part 3).

Instructions:

# Using aggregate() function, example:
# aggregate(MaxStemHeight_m ~ primary_growth_form, data = palm_clean, FUN = mean)

# Using tapply() function, example: 
# tapply(palm_clean$MaxStemHeight_m, palm_clean$primary_growth_form, mean, na.rm = TRUE)

Part 5: Exploratory Data Analysis with Visualizations (20 points)

Create visualizations to explore your cleaned palm dataset using base R plotting functions.

5.1 Histogram - Distribution of Stem Heights (5 points)

Create a histogram showing the distribution of maximum stem height.

# Write your code here

5.2 Boxplot - Comparing Stem Heights Across Growth Forms (5 points)

Create boxplots comparing stem height across different growth forms or size categories.

# Write your code here

5.3 Scatterplot - Relationship Between Morphological Traits (5 points)

Create a scatterplot exploring the relationship between two continuous morphological traits.

# Write your code here

5.4 Multi-Panel Display (5 points)

Create a comprehensive multi-panel figure showing multiple aspects of your palm data.

# Set up a 2x2 plotting area
# par(mfrow = c(2, 2), mar = c(4, 4, 2, 1))

# Panel 1: Histogram of stem height

# Panel 2: Boxplot of fruit length by growth form

# Panel 3: Scatterplot of fruit dimensions

# Panel 4: Barplot of growth form frequencies

# Reset plotting area
# par(mfrow = c(1, 1))

Part 6: Summary Table Creation (5 points)

Create a comprehensive summary table that presents your key findings in a clear, organized format.

Instructions: Create a data frame that summarizes key statistics for your main continuous variables, organized by growth form or another grouping variable.

# Write your code here

Part 7: Reflection and Interpretation (10 points)

Write a comprehensive summary (200 words) that synthesizes your entire analysis. Address the following:

7.1 Data Quality and Cleaning Process

What were the main data quality issues you encountered in the PalmTraits dataset?
How did you address these issues? Be specific about your cleaning steps and decisions.
What challenges did you face in deciding how to handle missing data or outliers?
How might your cleaning decisions affect the interpretation of results?

7.2 Key Findings

What are the 3-4 most important findings from your descriptive statistics and visualizations?
What patterns or relationships did you discover about palm morphology and growth forms?
Were there any surprising or unexpected findings?
How do your quantitative findings relate to what is known about palm ecology? (Consider things like growth strategies, herbivore defense, seed dispersal, etc.)

7.3 Methodological Reflection

Compare your experience using base R functions versus dplyr for data manipulation. What did you find easier or harder about each approach?
Which type of visualization (histogram, boxplot, scatterplot) was most informative for understanding your data? Why?
What did you learn about the importance of data cleaning before conducting statistical analyses?

Submission Instructions

What to Submit

Submit a single .qmd (Quarto document) file that contains:

All your code chunks with descriptive labels
Written explanations and interpretations for each section
Output from your code (tables, statistics, and visualizations)
Your comprehensive reflection in Part 7

Important Requirements for Reproducibility

Your .qmd file must be fully reproducible, meaning it should run completely on the instructor’s computer without errors. To ensure this:

Use relative file paths, not absolute paths:
- ✅ Good: read.delim("data/PalmTraits_10.txt")
- ❌ Bad: read.delim("C:/Users/YourName/Documents/data/PalmTraits_10.txt")
Include clear instructions about data files:
- At the top of your document, include a comment block explaining where to download the data
- Mention the exact filename you’re using
- If you’ve created a subset or modified version, include that file with your submission
Load all required packages at the beginning in the setup chunk:
- Only use readxl and dplyr as specified
- Include any additional packages only if explicitly needed
Test your document thoroughly before submitting:
- Click “Render” in RStudio to ensure it runs from start to finish without errors
- Close R and RStudio completely
- Reopen RStudio and try rendering again to verify true reproducibility
- Check that all plots and tables appear correctly in the rendered output
Set code chunk options appropriately:
- Use #| warning: false and #| message: false in the setup chunk
- Use #| echo: true to show your code (default, but be explicit)
- Use #| fig.width: and #| fig.height: to control plot sizes if needed
Document your workflow:
- Include comments in your code explaining complex operations
- Use meaningful variable names
- Organize your code logically within each section

File Naming

Name your file as: LastName_FirstName_HW2.qmd

What to Include with Your Submission

Your .qmd file (required)
The rendered HTML or PDF output (optional but recommended)
A brief README note if you made any non-standard decisions about the data

Note about the data file: Since the PalmTraits dataset is publicly available and can be downloaded from the Dryad link provided, you do NOT need to submit the data file itself. However, your code must include clear instructions on where to obtain it and what filename to use.

Grading Criteria

Your assignment will be evaluated on:

Completeness (25 points): All sections addressed with required components
- Part 1: Data Import (20 points)
- Part 2: Data Cleaning (35 points)
- Part 3: dplyr Reshaping (15 points)
- Part 4: Descriptive Statistics (15 points)
- Part 5: Visualizations (20 points)
- Part 6: Summary Table (5 points)
- Part 7: Reflection (10 points)
Code Quality (20 points):
- Clean, well-commented, reproducible code
- Appropriate use of base R and dplyr functions
- Correct syntax and no errors
- Efficient and logical workflow
Data Cleaning Thoroughness (25 points):
- Systematic identification of data quality issues
- Thoughtful and well-justified handling of missing values and outliers
- Appropriate data type conversions
- Meaningful derived variables
Interpretation and Critical Thinking (20 points):
- Clear, thoughtful interpretation of results in ecological context
- Evidence of biological/ecological knowledge
- Critical reflection on methods and limitations
- Synthesis of findings across analyses
Visualizations (10 points):
- Appropriate plot types for the data
- Well-labeled and professional-looking plots
- Clear and insightful interpretations

Total: 100 points (plus 2 possible bonus points)

Due Date

Friday Oct 17, 2025 at the end of the day

Helpful Resources

About the PalmTraits Dataset

Original paper: Kissling et al. (2019) Scientific Data, DOI: 10.1038/s41597-019-0189-0
Dataset location: https://datadryad.org/stash/dataset/doi:10.5061/dryad.ts45225
Documentation: The Dryad repository includes detailed metadata about all variables

R Documentation

Data import: ?read.delim, ?read.table
Missing values: ?is.na, ?complete.cases
Base R statistics: ?mean, ?median, ?var, ?sd, ?quantile
Base R plotting: ?hist, ?boxplot, ?plot, ?par
dplyr functions: ?dplyr::group_by, ?dplyr::summarize
Pipe operator: ?magrittr::pipe

Getting Help

R built-in help: Type ?function_name in the console
If you get stuck, try breaking down complex operations into smaller steps
Check that your variable names match exactly (R is case-sensitive)
Make sure you’re using the correct dataset object after each cleaning step

Homework 2: Data Import, Cleaning, and Exploration in R

Working with the PalmTraits 1.0 Dataset

Overview

Learning Objectives

About the Dataset

Setup

Downloading the Dataset

Create Your Quarto Document

Part 1: Data Import (20 points)

1.1 Import the PalmTraits Dataset

1.2 Explore Key Variables

Part 2: Data Cleaning and Manipulation with Base R (35 points)

2.1 Identify Data Quality Issues (5 points)

2.2 Create a Working Subset (3 points)

2.3 Handle Missing Values (8 points)

2.4 Fix Data Types (5 points)

2.5 Handle Outliers and Create Data Quality Flags (6 points)

2.6 Create New Derived Variables (8 points)

Part 3: Data Reshaping with dplyr (15 points)

3.1 Understanding the Pipe Operator %>% (3 points)

3.2 Using group_by() and summarize() for Aggregation (12 points)

Part 4: Descriptive Statistics with Base R (15 points)

4.1 Comprehensive Statistics for Key Variables (8 points)

4.2 Frequency Tables for Categorical Variables (4 points)

4.3 Group Statistics Using Base R (3 points)

Part 5: Exploratory Data Analysis with Visualizations (20 points)

5.1 Histogram - Distribution of Stem Heights (5 points)

5.2 Boxplot - Comparing Stem Heights Across Growth Forms (5 points)

5.3 Scatterplot - Relationship Between Morphological Traits (5 points)

5.4 Multi-Panel Display (5 points)

Part 6: Summary Table Creation (5 points)

Part 7: Reflection and Interpretation (10 points)

7.1 Data Quality and Cleaning Process

7.2 Key Findings

7.3 Methodological Reflection

Submission Instructions

What to Submit

Important Requirements for Reproducibility

File Naming

What to Include with Your Submission

Grading Criteria

Due Date

Helpful Resources

About the PalmTraits Dataset

R Documentation

Getting Help

3.1 Understanding the Pipe Operator `%>%` (3 points)