The Role of Regular Expressions in Creating a Tidy Data

In this analysis, let's prepare a tidy data that can be used for later analysis employing regular expressions in R and demonstrate the strength of regular expressions.

Let's check if the package 'downloader' is installed in our computer. If it is not installed, install it.

if(!require(downloader)){
    install.packages('downloader', repos='http://cran.us.r-project.org')
}

library(downloader)         # load the package for use

Load other required packages.

require(downloader)
library(plyr)

One of the most exciting areas in all of data science right now is wearable computing - see for example this article . Companies like Fitbit, Nike, and Jawbone Up are racing to develop the most advanced algorithms to attract new users.

The data we will use in this analysis is collected from accelerometers from the Samsung Galaxy S smartphone. A full description is available here.

setwd("C:/Fish/Ranalysis/Re")  # set working directory
rm(list=ls())  # clear workspace

Download the data as Dataset.zip from here and unzip it.

url<-"https://d396qusza40orc.cloudfront.net/getdata%2Fprojectfiles%2FUCI%20HAR%20Dataset.zip"

download(url, dest="dataset.zip") 

unzip ("dataset.zip")

Let's see the list of files and folders

dir()

1. "dataset.zip"
2. "UCI HAR Dataset"

"UCI HAR Dataset" is the directory created after unzipping Dataset.zip

Change directory to the unzipped folder, which is "UCI HAR Dataset"

setwd("C:/Fish/Ranalysis/Re/UCI HAR Dataset")

Let's see the list of files and folders in the "UCI HAR Dataset" folder.

dir()

1. "activity_labels.txt"
2. "features.txt"
3. "features_info.txt"
4. "merged_and_cleaned_data.txt"
5. "merged_and_cleaned_data2.txt"
6. "README.txt"
7. "test"
8. "tidydata2.txt"
9. "train"

Let's see the contents of the folders test and train.

list.files("./train")

1. "Inertial Signals"
2. "subject_train.txt"
3. "X_train.txt"
4. "y_train.txt"

list.files("./test")

1. "Inertial Signals"
2. "subject_test.txt"
3. "X_test.txt"
4. "y_test.txt"

"README.txt" contains detailed information about the different data sets.

The files to be used in this analysis are shown in the figure below. Files in the Inertial Signals folders are not being used here. From the figure, we see that we will use Activity, Subject and Features as part of descriptive variable names for the final data frame we will create.

Merge the training and the test sets to create one data set

As a first step in creating a tidy data, let's merge the training and test sets.

X_train <- read.table("train/X_train.txt")
X_test <- read.table("test/X_test.txt")

dim(X_train)

1. 7352
2. 561

dim(X_test)

1. 2947
2. 561

X <- rbind(X_train, X_test)

dim(X)

1. 10299
2. 561

y_train <- read.table("train/y_train.txt")
y_test <- read.table("test/y_test.txt")

dim(y_train)

1. 7352
2. 1

dim(y_test)

1. 2947
2. 1

Y <- rbind(y_train, y_test)

dim(Y)

1. 10299
2. 1

subject_train <- read.table("train/subject_train.txt")
subject_test <- read.table("test/subject_test.txt")

dim(subject_train)

1. 7352
2. 1

dim(subject_test)

1. 2947
2. 1

Subject <- rbind(subject_train, subject_test)

dim(Subject)

1. 10299
2. 1

Features <- read.table("features.txt")

dim(Features)

1. 561
2. 2

Let's set names to the variables

names(Subject)<-c("subject")
names(Y)<- c("activity")
names(X)<- Features[ ,2]

Regular expressions

Now, let's extract only the measurements on the mean and standard deviation for each measurement using regular expression functions.

help(grep) shows us the functions we can use for regular expressions in R.

Let's search the indices of the names that contain-mean() and -std(). \\ is escape character.

We are using the function grep to search indices that contain mean() and std() in the Features variable.

indices <- grep("-mean\\(\\)|-std\\(\\)", Features[, 2])
extracted <- X[, indices]

Let's give name to extracted from Feature.

Let's remove "()" using the regular expression function gsub and change the characters to lower case for readability.

names(extracted) <- Features[indices, 2]
names(extracted) <- gsub("\\(|\\)", "", names(extracted))
names(extracted) <- tolower(names(extracted))

dim(extracted)

1. 10299
2. 66

Descriptive activity names

Now, let's use descriptive activity names to name the activities in the data set.

activities <- read.table("activity_labels.txt")

Let's see activities

activities

	V1	V2
1	1	WALKING
2	2	WALKING_UPSTAIRS
3	3	WALKING_DOWNSTAIRS
4	4	SITTING
5	5	STANDING
6	6	LAYING

Let's remove "_" using the regular expression function gsub and change the characters to lower case for readability.

activities[, 2] = gsub("_", "", tolower(as.character(activities[, 2])))
Y[,1] = activities[Y[,1], 2]
names(Y) <- "activity"

Y[1:10,1]   # Just checking that it has been appropriately renamed

1. "standing"
2. "standing"
3. "standing"
4. "standing"
5. "standing"
6. "standing"
7. "standing"
8. "standing"
9. "standing"
10. "standing"

Label the data set with descriptive activity names

Now, let's appropriately label the data set with descriptive activity names.

names(Subject) <- "subject"

Now, let's create a data frame and save it as merged _and cleaned_data.txt

clean <- cbind(Subject,Y,extracted)

write.table(clean, "merged_and_cleaned_data.txt")

dim(clean)

1. 10299
2. 68

From the clean data set, let's create a second, independent tidy data set with the average of each variable for each activity and each subject.

We can use the handy function aggregate for this purpose.

clean2<-aggregate(. ~subject + activity, clean, mean)
clean2<-clean2[order(clean2$subject,clean2$activity),]
write.table(clean2, file = "merged_and_cleaned_data2.txt",row.name=FALSE)

dim(clean2)

1. 180
2. 68

This is a quick overview of the application of regular expressions in R to create a tidy data that can be used for later analysis.