The Data Mine (STAT 19000) Projects Fall 2019

Filter by

Click here for the syllabus.

26 of august

Project 1. Scavenger hunt: getting acquainted with UNIX on Scholar

Topics include:
Getting access to the Scholar cluster.
Setting up the ThinLinc client.
Finding resources on Scholar:
Firefox, Text Editor, LibreOffice,
Document Viewer, and the Terminal.
The /home versus /scratch versus /class filesystems.
(In particular, making sure everyone has access to the /class/datamine directories.)
Getting access to sample data sets
that are available for analysis on Scholar. Etc... posted on Friday, August 16 seminars occur during Monday, August 19 to Thursday, August 22 due by 10:00 A.M. on Monday, August 26

Project 1 Solutions by Tim Park

2 of september

No project due this week

Labor Day occurs on Monday, September 2.
9 of september

Project 2. Mapping with R

Topics include:
Why do we use R?
Examples of mapping and visualization in R,
using ggmap.
Mapping AirBnB data from Los Angeles.
Creating a Google Maps API Key.
Modifying a map to visualize
the AirBnB data from London. posted on Friday, August 23 seminars occur during Monday, August 26 to Thursday, August 29 due by 10:00 A.M. on Monday, September 9 (long time frame because of Labor Day)

Project 2 Solutions by Tim Park

16 of september

Project 3. Getting familiar with R

Topics include:
How do we use R?
Data frames.
Factors and vectors.
Indexing.
Plotting.
Etc... posted on Friday, September 6 seminars occur during Monday, September 9 to Thursday, September 12 due by 10:00 A.M. on Monday, September 16

Project 3 Solutions by Tim Park

23 of september

Project 4. Vectorized operations in R

Topics include:
Thinking in a vectorized way about R.
Functions such as paste,
mean, variance, table, etc.
Formatting dates.
NAs and bad/missing data.
Etc... posted on Friday, September 13 seminars occur during Monday, September 16 to Thursday, September 19 due by 10:00 A.M. on Monday, September 23 Thursday, September 26

Project 4 Solutions by Tim Park

30 of september

Project 5. Getting familiar with the apply suite of functions in R

Topics include:
How does R apply a function to a data set?
Focus on the tapply function as an introduction. Etc... posted on Friday, September 20 seminars occur during Monday, September 23 to Thursday, September 26 due by 10:00 A.M. on Monday, September 30 Thursday, October 3

Project 5 Solutions by Tim Park

7 of october

No project due this week

October Break occurs on Monday-Tuesday, October 7-8.
14 of october

Project 6. More functions in R

Topics include:
Overview of split, merge, subset, and others.
Brief discussion of ggplot/tidyverse.
Etc... posted on Friday, September 27 seminars occur during Monday, September 30 to Thursday, October 3 due by 10:00 A.M. on Monday, October 14 (long time frame because of October Break)

Project 6 Solutions by Tim Park

21 of october

Project 7. Getting familiar with UNIX/bash

Topics include:
Why use UNIX and bash?
Comparisons of the types of
analysis in R and in bash.
Introduction to the bash shell.
Bash functions such as:
cat, head, tail, pwd, cd, ls, cp,
mv, rm, mkdir, rmdir, wc,
diff, grep, cut, man.
Etc... posted on Friday, October 11 seminars occur during Monday, October 14 to Thursday, October 17 due by 10:00 A.M. on Monday, October 21
28 of october

Project 8. Pipes in bash

Topics include:
How do we use bash tools in tandem?
Pipes in bash.
Also, examples with sort and uniq.
Downloading data with curl and wget.
Using sftp and ssh to connect to UNIX servers.
Etc... posted on Friday, October 18 seminars occur during Monday, October 21 to Thursday, October 24 due by 10:00 A.M. on Monday, October 28
4 of november

Project 9. The awk utility

Topics include:
Using awk for one-line data analysis.
Cleaning, searching, filtering, parsing, reshaping, summarizing, etc... posted on Friday, October 25 seminars occur during Monday, October 28 to Thursday, October 31 due by 10:00 A.M. on Monday, November 4
11 of november

Project 10. R versus bash/awk

Topics include:
When is it more appropriate
to use R or bash (and awk)?
The pro's and con's of
each tool, with examples
regarding complexity, speed,
memory, reproducibility.
Etc... posted on Friday, November 1 seminars occur during Monday, November 4 to Thursday, November 7 due by 10:00 A.M. on Monday, November 11
18 of november

Project 11. Pattern matching

Topics include:
Using pattern matching with
R, bash, grep, awk.
Common themes of pattern matching
such as regular expressions.
Etc... posted on Friday, November 8 seminars occur during Monday, November 11 to Thursday, November 14 due by 10:00 A.M. on Monday, November 18
25 of november

Project 12. Fun with pattern matching

Topics include:
Applying the skills of
pattern matching and
regular expressions in a fun
project to end the semester. posted on Friday, November 15 seminars occur during Monday, November 18 to Thursday, November 21 due by 10:00 A.M. on Monday, November 25
2 of December

No project due this week

Thanksgiving Break occurs on Wednesday-Saturday, November 27-30.
9 of december

Optional Projects

Choice of 4 optional projects
to wrap up the semester.
These projects can be used to
replace grades from projects that
were missed during the semester
or to raise the grade of earlier projects.
Remember that students also are
able to drop their lowest 2 project grades. posted on Friday, November 22 seminars occur during Monday, December 2 to Thursday, December 5 due by 10:00 A.M. on Monday, December 9 (long time frame because of Thanksgiving)